Centralized or Federated: Where Should AI Actually Sit?
Both models work. Both fail in predictable, opposite ways — and the failure you get is a choice you are making now.
Erin Moore
Fractional Chief AI Officer
Centralize standards and infrastructure; federate application. That is the answer for almost every company under a few thousand people, and the interesting part is not the conclusion but which specific things belong on which side of the line.
How each model fails
A fully central AI team fails by queue. Everything needs the team, the team becomes a bottleneck, and the business units that cannot wait go around it. Centralization does not prevent shadow AI — a slow central team is one of its most reliable causes. The work still happens; it just happens where nobody can see it.
A fully federated model fails by duplication and drift. Four teams buy four overlapping tools. Nobody can answer what the company spends on AI or which systems touch customer data. Each team re-learns the same lessons at full price, and the first incident reveals that no two groups were operating to the same standard.
You are choosing which of these failure modes you would rather manage. Pretending you can avoid both is how organisations end up with the worst of each.
What must be central
Standards. One acceptable use policy, one vendor review, one incident path. These stop working the moment there are several versions — a standard that varies by department is not a standard.
The vendor and spend picture. Someone has to hold the whole list. Without it you cannot negotiate, cannot see duplication, and cannot answer a customer security questionnaire honestly.
Shared infrastructure. Model access, logging, the evaluation harness. Duplicating these is pure waste, and centralizing them is where most of the real cost saving in an AI programme actually comes from.
The risk register. One view, per AI risk register, or nobody can see concentration — four small dependencies on one vendor is not four small risks.
What must be federated
Use case selection. The people doing the work know which twenty minutes of their day is worth automating. A central team guessing on their behalf produces the impressive pilot nobody adopts, which is the single most common cause behind stalled AI pilots.
Domain evaluation. Whether an output is good enough is a domain judgement. Finance decides for finance. Legal decides for legal. A central team cannot hold those standards and should not try.
Adoption. Behaviour change happens inside teams, led by their own managers. It does not transmit from a centre of excellence.
The hybrid that works
A small central function — often one accountable executive plus a couple of engineers — owns standards, infrastructure, vendors and the risk picture. Named practitioners sit inside business units, doing the actual application work, with a dotted line to the centre. The centre's job is to make the federated work faster and safer, and it is measured on that rather than on shipping projects itself.
The failure version of this is a central team that owns standards and also insists on doing all the delivery. That is centralization wearing a hybrid label, and it produces the queue.
This is the shape covered in more depth in AI org structure, and it is why the reporting line matters: a central function that cannot say no to a business unit is advisory, whatever the chart says. NIST's AI Risk Management Framework puts this under the govern function — roles and accountability defined before capability is scaled, not after.
When to revisit
Centralize harder when incidents rise, spend is unknowable, or standards are visibly diverging. Federate harder when the queue is the complaint, when adoption is flat despite working tools, or when business units have started buying their own way around you.
Most companies should start central-heavy while standards are being set, then push application outward deliberately as maturity grows — the progression described in the AI maturity model.
Frequently asked questions
Should AI be centralized or federated? Both. Centralize standards, vendor management, shared infrastructure and the risk register. Federate use case selection, domain evaluation and adoption. Fully central creates a queue; fully federated creates duplication and drift.
What is the most common mistake? A central team that owns the standards and also insists on doing all the delivery. It becomes a bottleneck, and business units route around it — which is how shadow AI starts.
How big should the central team be? Smaller than most expect. For companies under a few thousand people, one accountable executive plus a small number of engineers is typically enough, because the centre's job is to enable the federated work rather than perform it.
When should we change the model? Centralize further when incidents rise or spend becomes unknowable. Federate further when the queue is the main complaint or adoption is flat despite working tools.
Further reading
Tagged with:
Ready to Automate Your Business?
Let's discuss how AI automation can deliver measurable ROI for your organization in 90 days or sooner.