The concept is sound: a central body that sets AI standards, manages vendor relationships, and prevents the company from running 14 redundant pilots. The execution is usually terrible. This is how to build a CoE that operating teams actually want to work with.
A functional AI Center of Excellence has three jobs: set the standards that prevent legal and reputational risk, provide shared infrastructure that individual teams cannot justify building themselves, and identify the top 10 percent of use cases that deserve company-wide investment. Everything else should be left to operating teams to figure out.
The failure pattern is predictable. The CoE is established with broad authority and unclear accountability. It creates a lengthy approval process for AI use cases. Operating teams route around it. The CoE becomes a bottleneck rather than a resource. Leadership loses confidence. The CoE is quietly dissolved or reorganized.
The root cause is almost always a mandate problem: the CoE was given the authority to approve without the responsibility to deliver. When a team wants to deploy an AI workflow and the CoE can say no but cannot be held accountable for the business outcome of saying no, you get risk-averse gatekeeping.
The fix is to restructure the CoE mandate. The CoE should own standards, shared infrastructure, and strategic investment decisions. It should not own individual use-case approvals. Teams should be able to deploy within the standards without CoE sign-off, and the CoE should be measured on how fast teams can move, not how many approvals it processes.
For companies under 500 employees, a formal CoE is usually the wrong structure. You want an AI Steering Group instead: 4 to 6 people who meet monthly, own the vendor portfolio decisions, and maintain the internal AI standards document. No full-time headcount required.
For companies between 500 and 5,000 employees, a lightweight CoE makes sense: one full-time AI lead, two to three embedded practitioners who spend 20 percent of their time on CoE work, and a rotating seat for operating team representatives. The CoE owns the platform layer (model access, API keys, data governance) and the standards layer (what data can go into which models, output review requirements, disclosure policies).
In 2026, the platform layer is simpler than it used to be. Claude via the Anthropic API, with proper data processing agreements and no training opt-out, covers most enterprise use cases. The CoE does not need to evaluate 30 models. It needs to pick one primary foundation model, one secondary model for specific tasks, and enforce that standard across the organization.
Every AI CoE needs a living standards document. Not a 40-page policy manual nobody reads. A practical 5-page reference that tells teams what they need to know before deploying an AI workflow.
The document should cover four things: data classification (which data categories can go into which models), output review requirements (which output types require human review before external use), disclosure obligations (when do you have to tell a client or customer that AI was involved), and incident reporting (what to do when an AI workflow produces a bad output).
The standards document should be updated quarterly and versioned. When standards change, the CoE owns communicating the change to all teams with active AI workflows. This is not optional. Stale standards are worse than no standards because they create false confidence.
Do not write your AI standards document in response to a crisis. Write it before you have anything to be defensive about. The companies that do it proactively look competent. The companies that do it reactively look like they were hiding something.
The metrics that matter for a CoE are: time from use-case identification to pilot deployment, number of active AI workflows across the company, percentage of workflows that meet the standards document requirements, and operating team satisfaction score from a quarterly survey.
The metric that should not matter: number of approvals processed. If the CoE is measuring itself on approval throughput, it has already adopted the wrong mindset. The goal is to enable fast, safe AI deployment -- not to be a checkpoint.
Book an AI Audit with Bill Colbert. In one session you get a clear diagnosis, a prioritized roadmap, and a plan your team can actually execute. No fluff, no vendor agendas.
Book an AI Audit