Getting an AI Pilot Into Production
The pilot worked. That was the easy part. Here is everything still missing before it can run unattended.
Erin Moore
Fractional Chief AI Officer
A working pilot is roughly a third of the way to production. The remaining two-thirds is everything that lets the system run when nobody is watching it: a named owner, monitoring, defined failure behaviour, rollback, a support route, access control and a review cadence. Skipping these is why so many successful pilots become abandoned production systems.
The seven
1. A named production owner. A person, funded, who is accountable for this working in eighteen months. Without this, nothing below gets maintained.
2. Monitoring that reflects quality, not uptime. The system being available tells you nothing about whether the outputs are still good. Track rework rate or sample outputs on a schedule.
3. Defined failure behaviour. What happens when the model is unavailable, slow, or returns something unusable? Falling back to the manual process is a legitimate answer, provided someone has decided it and the team knows.
4. Rollback. The ability to switch it off without a deployment, held by someone who can act immediately — see AI incident response.
5. A support route. Where does a confused user go? If the answer is "the person who built it", you have a single point of failure with a holiday allowance.
6. Access control. Who can use it, who can change its configuration, and who can see the data flowing through it.
7. A review date. Booked, in the calendar, against the baseline you established. Systems without a review date run forever regardless of merit.
The honest sequencing question
Doing all seven properly costs more than the pilot did. That is not an argument against production — it is the actual price, and it belongs in the business case before the pilot starts rather than as a surprise afterwards.
If the seven look disproportionate to the value, that is useful information: the use case may be a genuinely good idea that is not worth operationalising. Deciding that deliberately is a better outcome than a half-supported production system nobody owns.
What changes between pilot and production
Pilots run on cooperative users, clean inputs and someone watching. Production has none of those. The specific failures that appear are edge cases nobody mapped, inputs from systems that changed without notice, and the gradual drift that nobody notices because no one is sampling.
NIST's AI RMF Playbook frames "manage" as continuous rather than a launch activity, which is the right mental model: production is the beginning of the obligation, not the end of the project.
Frequently asked questions
What does an AI pilot need before production? A named funded owner, quality monitoring rather than uptime monitoring, defined failure behaviour, immediate rollback, a support route, access control, and a booked review date against the original baseline.
Why do successful pilots get abandoned? Because nobody was funded to own them after launch. Drift goes unnoticed, edge cases accumulate, and eventually people quietly stop using the system.
How much does productionising cost? Typically more than the pilot. That belongs in the business case up front, not as a surprise after everyone has declared success.
What if production support looks disproportionate? Then the honest answer may be not to productionise. A good idea that is not worth operating is a legitimate conclusion, and cheaper than an unsupported system.
Further reading
Tagged with:
Ready to Automate Your Business?
Let's discuss how AI automation can deliver measurable ROI for your organization in 90 days or sooner.