How to Achieve the Minimum Viable AI Governance

26 January 2026 - ID G00836167 - 18 min read
By Alys Woodward
Midsize enterprises face significant risks and missed opportunities without effective AI governance. Midsize enterprise CIOs should set a minimum viable governance strategy for AI by focusing on activities in three categories: policies and controls, the AI governance operating model, and oversight systems.

Insights at a Glance

Midsize enterprises (MSEs) face significant risks and missed opportunities without effective AI governance. Establishing minimum viable AI governance enables effective, reliable and repeatable oversight of AI initiatives, supporting innovation while mitigating threats. This approach is critical for organizations with lean teams, balancing the need for control with the flexibility required for evolving AI technologies.
Key Insights:
  • MSEs must avoid four major AI governance pitfalls: overly strict governance rules that stifle innovation from the outset, excessive complexity that reduces adherence, lax oversight that fails to address risks, and inflexibility that prevents adaptation as AI technologies evolve.
  • Minimum viable AI governance should be structured around three core categories: policies and controls, the AI governance operating model, and oversight systems.
  • Effective policies and controls include setting clear, actionable principles (such as equity and bias principles and transparency), addressing AI-specific security vulnerabilities, and ensuring ongoing legal and regulatory compliance.
  • The governance operating model should reflect the organizational structure and integrate with existing IT governance and risk management frameworks.
  • Oversight systems must monitor compliance, coordinate vendor engagement, validate AI outputs, and ensure data governance supports AI initiatives without duplicating broader data governance efforts.
Recommended Actions:
  1. Build an AI governance framework organized under policies and controls, the governance operating model, and oversight systems; tailor activities to the organizational structure and resource constraints.
  2. When setting AI policies and controls, develop expedited AI usage policies, including a preapproved list of public Generative AI (GenAI) tools that meet security standards, to reduce the risk from rapid business adoption.
  3. When defining an AI governance operating model, establish an AI governance or steering committee with cross-functional representation. Meetings should be frequent (e.g., biweekly or weekly) in an AI-first environment. They can be less frequent (e.g., monthly) in more AI-steady environments.
  4. In terms of oversight systems, monitor the use of AI, either by technical tools or through processes and affirmations from employees.

Strategic Planning Assumption

By 2027, AI governance will become a requirement of all sovereign AI laws and regulations worldwide.

Impact Brief

Without governance, AI introduces untenable risks due to its haphazard use by employees. Seventy-two percent of MSEs report evidence or suspicions of employees using prohibited or unauthorized public GenAI tools that do not comply with organizational policy.1 Conversely, organizations in regulated industries such as financial services or government find that employees are reluctant to experiment without reassurance that AI is governed.
AI governance involves a potentially large number of varied activities, stretching MSE CIOs’ resources still further. Only 8% of midsize organizations have comprehensive AI governance according to Gartner’s 2025 Cybersecurity Innovations in AI Risk Management and Use Survey. For midsize organizations and organizations with lean IT teams, it is important to attain the minimum viable AI governance so that AI is governed reliably and repeatably.
Midsize organizations must avoid four pitfalls when designing AI governance practices:
  • Overly strict governance rules that stifle innovation by preventing users from experimenting with tools and exploring how AI could address their needs.
  • Excessive complexity that reduces adherence; users don’t understand how or why to comply with the rules and decide to ignore them.
  • Lax oversight that fails to address risks, allowing users to inadvertently create security vulnerabilities, leak data, or fall out of compliance without realizing it.
  • Inflexibility that prevents adaptation as AI technologies evolve, which will be important due to the early state of the market.

Actions and Cautions

For midsize organizations and organizations with lean IT teams, it is important to attain the minimum viable AI governance so that AI is governed reliably and repeatably.

Actions

  • Build an AI governance framework with activities organized under three categories: policies and controls, the governance operating model, and oversight systems.
  • Set up an AI governance committee or leverage an existing AI steering committee as a capability to create initial usage policies for public GenAI tools and a framework for collaboration across business units, functions and IT. Focus policies and controls on setting general principles, addressing AI-specific security vulnerabilities.
  • Create an AI governance operating model to reflect the structure of your organization (don’t overcentralize in a decentralized organization, for example). Integrate your AI governance with other governance frameworks like IT governance and enterprise risk management. Allow your AI governance to adapt over time to changing conditions such as legislation, organizational policy and technology maturity.
  • Create appropriate oversight systems to ensure compliance and monitor AI-related activities, projects and tools, but don’t move to adopting AI governance tools too early. Cash-strapped MSEs may need to start with human processes.

Caution

If AI governance is overly complex or restrictive, users simply won’t adhere to it.

How to Execute

Gartner inquiries demonstrate that MSEs are divided in how they approach AI: some start from the value, considering use cases and building a priority list, while others start from AI governance, wanting to shut down the possibilities of shadow AI and risky behaviors with public AI tools. This, combined with the complexity and rapid change of AI, makes it difficult for MSEs, in particular, to get started with AI governance.
A key initial starting point should be an AI strategy. AI often starts before there is a defined strategy, with business users experimenting with consumer GenAI tools; however, this citizen development should be incorporated into the AI strategy in the form of “everyday AI.” Without an AI strategy, it’s hard to define the level of strictness or laxity that AI governance needs to support, but everyday AI is an important element of how AI will penetrate the organization. This document assumes the existence of an AI strategy. For more information on creating an AI strategy, please see “AI Leaders: Maturity Guide for AI Strategy.”
Definition: AI governance is the process of creating policies, assigning decision rights and ensuring organizational accountability for the risks and decisions for the application and use of AI techniques.
Figure 1 defines the three categories of activities in a minimum viable AI governance model for MSEs.
Figure 1: Minimum Viable AI Governance for MSEs
Minimum viable AI governance includes policies and controls, an AI governance operating model, and oversight systems. Key areas are security, compliance, team structure, monitoring, vendor coordination, and data governance for effective risk management.

1. Policies and Controls

Under “policies and controls,” we have grouped areas that specify what you are going to do and what rules surround those activities. Principles, AI security, and legal and regulatory compliance are all part of policies and controls. Privacy, sustainable AI and trustworthy AI also come into this category alongside principles.

Set General Principles to Speed Up Decision Making

MSE IT leaders in charge of AI governance should define and promote guiding principles for the development and use of AI. These principles should enable progress based on AI within the framework of existing corporate values and ethics or new ones if AI means they are required.
MSEs often struggle with principles because they almost seem too obvious. When setting principles, the typical pitfalls are to make them either too generic or too tech-specific, so activities build up that are outside the strategy. Principles should apply to multiple scenarios; the idea is to build principles that make decision making easier and faster, not to build specific rules that require analysis to see whether they apply. Ethical AI and responsible AI considerations are part of this principle set. It’s also important to communicate the principles. Large enterprises are generally more aware of the need to communicate their work than MSEs, particularly where MSE IT departments have been asset-centric.
While AI literacy is a work in progress, we recommend you add a sentence to explain why each principle holds, so that business leaders and users understand the AI principles have a genuine need to exist.
Example Principles
Equity and bias: At all stages of the AI life cycle, it is critical that all authorized users watch for and correct or report inappropriate bias in AI training data, prompts and output. This is important so that we continue to take high-quality business decisions, reduce risks and continually improve by enhancing the performance and reliability of AI over time.
Transparency: Whenever AI is used, all authorized users must document and communicate which AI tools were used, how they were used, and what output was produced. This is important to maintain the trust that supports our organization, as people (colleagues and customers) who rely on the data, models, and solutions generated need to understand where AI was used.
For more information, see Toolkit: Define Your Principles to Develop and Use Responsible AI.

Address AI-Specific Security Threats

AI makes security more challenging because it can introduce new vulnerabilities, such as adversarial attacks that manipulate AI models or exploit their decision-making processes. The complexity and opacity of many AI systems make it difficult to detect and respond to threats, increasing the risk of unintended consequences or breaches. AI can also automate and scale cyberattacks, meaning malicious actors can target organizations more efficiently. Gartner’s Technology Adoption Roadmap for Midsize Enterprises for 2026 survey data shows that despite recognizing the misuse of AI and GenAI as a top cybersecurity risk, nearly half of MSEs report being unprepared to manage AI-related cyber risks.
To address these challenges, organizations must implement robust security measures tailored to AI systems, including cybersecurity in AI buying and adoption, evaluating data security for AI, and creating safe environments for AI experimentation to avert data losses as employees are learning. Additionally, they should foster a culture of security awareness, invest in specialized AI security expertise, and collaborate with industry peers to share best practices and threat intelligence.
Example Action
Build an expedited AI usage policy with a list of approved public GenAI tools. Business teams often bypass IT to quickly adopt GenAI. Creating a preapproved list of GenAI tools that meet the organization’s established security standards will mean that the quick actions of business teams will not expose the organization to unnecessary risks.
For more information, see 3 Generative AI Security Red Flags for Midsize Enterprise CIOs.

Collaborate With CxOs for Ongoing Legal and Regulatory Compliance

There is an evolving regulatory and legal landscape surrounding GenAI, which influences the future development and markets of this technology. MSE CIOs or AI leaders need to understand these implications in order to drive positive innovation and position their organizations for long-term success.
To navigate this complex environment, MSE CIOs or AI leaders should identify key areas of legal uncertainty. Evolving areas include the meaning of “high risk,” ownership rights, documentation requirements, liability standards, and the distinction between building and using AI. In some areas, the rulemaking direction is unclear, and geographic differences are emerging. They should also work to strengthen the relationship between IT, security, and legal departments by involving legal teams early to review potential GenAI use before investing in development or prototyping.
Collaboration with legal is critical, but ultimately a collaboration across all business functions will support the drive toward AI value, and the AI governance committee will be the mechanism to initiate that collaboration.
Example Actions
  • Review the terms of service for the AI tools and platforms you buy to appropriately prevent breaches of confidentiality and unauthorized use of your content for your legal environment.
  • Build an enforcement and escalation process for breaches of the code of conduct or the AI usage policy (which describes restrictions from an ethical and legal perspective).
For further information, see 9 Regulatory and Legal Implications of Generative AI.

2. AI Governance Operating Model

The AI governance operating model is the structured framework and set of processes an organization uses to oversee, manage, and ensure the responsible development and deployment of AI systems. It defines the roles, responsibilities, policies, and decision-making mechanisms to align AI initiatives with organizational values, regulatory requirements, and ethical standards. The operating model helps organizations mitigate risks, maintain transparency, and foster trust in their AI projects. MSE CIOs should obtain the authority and/or support from key senior leadership to develop and implement an AI governance operating model.
For information on building a lightweight AI governance structure, see How to Build a Lightweight Organizational Structure for AI Governance.

Reflect the Structure of Your Organization and Teams

CIOs new to AI in centralized organizations often try to apply a centralized model to AI, but this can lead to problems. The democratized nature of AI means that its organization and ownership should reflect the organization, rather than imposing centralized control. Attempts to overcontrol will lead to shadow AI, increased risk and increased costs. Align a decentralized organization with decentralized or federated AI. That said, it is essential for IT to be involved in AI governance, so where IT is centralized in a decentralized organization, AI governance will not align.
AI leadership approaches and structures vary due to the novelty of AI governance, and a range of example structures exist. When creating governance structures, consider your AI scope, AI maturity, industry, regulatory compliance and culture.
The most common AI governance structure for MSEs is an AI governance council. An AI governance council brings together a diverse group of stakeholders to spearhead AI governance efforts. It should have direct working relationships with other governance councils where these exist, such as data governance (or data and analytics governance), IT governance, a risk governance council or a cybersecurity council. For MSEs, the council is often chaired by the CIO or the AI leader where one exists. Sometimes a particular business leader emerges as someone who wants to lead AI, for example, the head of customer service in an organization with a strategic customer engagement center. The council is populated by representatives from the business domains and support functions including legal, procurement and HR. It’s also usual for successful MSEs to have a single council, rather than multiple councils focusing on AI steering, AI value and AI governance separately, which is more usual for large enterprises. The lean structure and flat hierarchies of MSEs give the advantage that AI value and AI risk can be considered by the same committee, which accelerates AI literacy and understanding through the organization.

Integrate AI Governance With Existing IT Governance and Risk Management

The AI governance operating model should connect with other governance bodies and extend existing governance models to AI-specific considerations of trust, transparency and diversity. For example, ERP systems don’t need specific governance to ensure people trust them and use them, whereas AI systems do.
The scope of AI governance should be based on the focus areas for decision making around AI (e.g., strategy, investments, risk, value, performance and resources). MSEs need an executive sponsor (a person or a committee or board) to support AI initiatives and remove barriers. The AI governance body should establish and maintain an AI governance charter as a mandate for effective AI governance. The scope of the governance can expand proportionally as AI adoption progresses through more use cases and departments or functions in the organization.
Example of Mandate and Scope
  • Example: IT governance oversees data security and system reliability, while AI governance focuses on the ethical use of AI algorithms and the transparency of decision-making processes.
  • Example: IT governance and AI governance committees jointly ensure that AI models adhere to established data privacy standards, while AI governance takes additional steps to monitor for algorithmic bias and fairness.
  • Example: IT risk management assesses risks related to infrastructure and cybersecurity, whereas AI governance bodies can concentrate on risks unique to AI, such as model drift, unintended consequences, and explainability. For instance, AI governance can establish protocols for regular auditing of AI models to detect and mitigate risks of discrimination or noncompliance with regulations while IT risk management continues to address broader technology risks.

People, Culture and Literacy

AI literacy is crucial for driving the adoption of AI within organizations because it empowers employees to understand and leverage AI technologies effectively. When staff are knowledgeable about AI, they can identify opportunities for innovation, collaborate confidently with technical teams, and mitigate risks associated with implementation. Additionally, fostering AI literacy helps build trust in AI systems, encourages responsible use, and ensures that organizations can adapt to rapidly evolving digital landscapes. Ultimately, a strong foundation in AI literacy supports successful integration and long-term value from AI initiatives.
Figure 2 outlines a roadmap to set up an AI literacy program.
Figure 2: Roadmap for an AI Literacy Program
A five-step roadmap guides AI literacy program design, focusing on assessing skill gaps for different persona groups. Training is tailored by role, with varying needs in AI foundations, value, engineering, and governance for effective upskilling.

3. Oversight Systems

IT defines processes and activities that monitor compliance, coordinate vendor engagements, test and validate AI outputs and ensure that data is appropriately governed for AI.

Monitor and Ensure Compliance

Although the focus of AI governance should be on influencing rather than constraining, miscompliance with procedures must be discouraged, escalated and ultimately sanctioned.
MSE CIOs are finding it hard enough to find the budget for AI technology, let alone technology to govern AI, so monitoring may initially be done via processes and affirmations from employees rather than by AI governance platforms. AI governance platforms are tools designed to ensure organizations adhere to organization policy, regulations and industry standards across common responsible AI principles. These platforms allow leaders responsible for AI and other technical or business leaders to streamline governance processes organizationwide and serve as a central repository for trust, risk and security controls. The market for AI governance platforms is emerging, and the vast majority of organizations adopting these platforms are large enterprises. Some vendors of AI governance platforms with MSE customers are Holistic and Relyance.
For more information on vendors, see Market Guide for AI Governance Platforms.
Example Monitoring Processes
  • Continuous monitoring and runtime enforcement controls based on corporate policy, regulations, frameworks and standards.
  • Regular performance audits — periodically review AI models for accuracy and reliability, including a human review with business SMEs and IT, and compare the results against benchmarks and expected outcomes. Depending on the predictability of outcomes, these reviews may be partially automated.
  • AI incident reporting and response — establish a formal process for employees to report incidents when AI is inaccurate. Track this process on a technology platform, and set up a human team to investigate.

Coordinate Vendor Engagements

The AI governance committee should initiate a process to coordinate how vendors’ technology is introduced to the organization. The level of participation of the AI governance committee depends on the specific responsible, accountable, consulted and informed (RACI) matrix for the organization. The committee could be responsible, accountable, consulted or informed about which vendors’ technologies are being evaluated or implemented depending on the AI committee charter.
Figure 3 shows a governance flow scenario for vendor coordination in MSEs.
Figure 3: Example Flow for Vendor Coordination in MSEs
Vendor coordination for AI involves marketing requesting BYOAI, central committee providing enablement guidelines, oversight setting customer-facing policies, compliance with regulations, and use of centralized data for grounding, balancing flexibility and control.

Test and Validate AI Outputs

Incorporate a capability within the AI governance process to validate AI models. This will require business knowledge to understand the context, alongside the technical ability, to improve the prompting, model or context based on the results generated.
Figure 4 outlines example testing scenarios for AI models.
Figure 4: Example Scenarios for Testing AI Models
Testing AI models faces challenges like reproducibility, evaluation, bias, user experience, and debugging. These issues can cause inconsistent results, complex bias detection, and difficulty in identifying and fixing errors.

Perform Data Governance to Support AI

Getting started with their AI initiatives is when many MSEs find themselves suddenly highly motivated to start to govern and manage their data. A significant proportion of MSEs know they have neglected data and analytics and lack control of their data environments, but other activities have been more urgent. The rise of GenAI leads MSEs to reconsider getting their data governance in place. However, data governance is a significant initiative on its own and should not be fully covered within AI governance as this may derail it.
If data governance is a neglected area, MSE CIOs should focus on identifying early AI use cases and governing the relevant data for those areas. If data governance is already in place, MSE CIOs should coordinate with data governance when prioritizing and enabling AI initiatives.
See Follow These Five Steps to Make Sure Your Data Is AI-Ready.
Note: This document takes the most important points with relevance and achievability for lean teams in MSEs from this research into AI governance: Reference Guide for AI Governance.

Success Measures

MSEs can measure the success of their AI governance programs in the following ways:
Assess the transparency, fairness, and accountability of their AI systems and processes. Key indicators include compliance with regulatory standards, the effectiveness of risk management frameworks, and the presence of clear policies for ethical AI use.
Widespread AI adoption means that AI governance has provided a foundation for innovation that business users have been able to build upon, an indicator that AI governance has been successful.

Evidence

1 2025 Gartner Cybersecurity Innovations in AI Risk Management and Use Survey. This survey was conducted to understand how organizations are managing the cybersecurity risks of GenAI and AI techniques that support it. The research was conducted online from 21 March through 9 May 2025 among 302 cybersecurity leaders in the North America (n = 181), EMEA (n = 71) and Asia/Pacific (n = 50) regions. Qualifying organizations reported enterprisewide revenue of at least $250 million or equivalent for fiscal 2024 and were senior cybersecurity management involved in activities related to AI cybersecurity risk management within their organization. Disclaimer: The results of this survey do not represent global findings or the market as a whole, but reflect the sentiments of the respondents and companies surveyed.
2025 Gartner Technology Adoption Roadmap for Midsize Enterprises Survey. This survey harnesses the collective wisdom of IT leaders to understand deployment plans, adoption timelines, value and risks posed by 50 technologies across infrastructure, digital workspace, AI, data and analytics, network, security, and storage and database. The survey was conducted through an online panel from June through August 2025 among 411 respondents from North America, EMEA and Asia/Pacific across industries and enterprises with annual revenue from $50 million to $1 billion. Qualified respondents were CxOs, senior IT leaders, their peers and their direct reports. ​The results will allow leaders to challenge vendor hype coming from business partners and the vendors themselves. It will also help prioritize modernization efforts and tie into Gartner BuySmart services. As MSEs push more technical work (including security) to cloud and managed service providers (MSPs), IT refocuses internal investments on automation and the digital workplace.