What to Do When Your AI Gets It Badly Wrong
Every AI system will produce a bad output eventually. The difference between an incident and a crisis is whether anyone decided in advance.
Erin Moore
Fractional Chief AI Officer
An AI incident response plan needs four things written down before you need them: what counts as an incident, who can switch the system off, who talks to affected customers, and how you find out it happened. Most companies have none of these, which turns an ordinary bad output into a crisis measured in days.
Define the incident before it happens
Not every wrong output is an incident. Draw the line explicitly, and draw it around consequence rather than technology:
- Anything that reached a customer and was materially wrong.
- Anything involving money, contracts or a commitment made on your behalf.
- Any exposure of data that should not have left your systems.
- A pattern of wrong outputs, even individually small ones.
That last category is the one people miss, and it is where most real harm accumulates — a routing system quietly mishandling a slice of cases for weeks.
The switch-off decision
Name the person who can disable an AI system without seeking permission, and make sure they can do it technically, not just organisationally. An incident where the accountable person has to file a ticket to stop the bleeding is a much longer incident.
This belongs with whoever owns the AI portfolio — the argument in who owns AI governance. Committees cannot make this call at the necessary speed.
Detection is the weak point
Most plans specify response and skip detection. Ask honestly: if the system started producing wrong outputs today, how would you find out, and how long would it take?
Realistic mechanisms for a growing business are a sampled human review on a schedule, a rising rework rate in the team using it, and a clearly advertised internal route for staff to report bad output without it becoming a formal complaint. None of these are sophisticated; all of them beat waiting for a customer to tell you.
Telling people
Decide in advance who communicates and roughly what gets said. If personal data is involved, obligations may apply — the UK ICO's guidance on AI and data protection is a practical starting point for organisations handling it, and CISA's AI resources cover the security dimension.
The instinct to say nothing while you investigate is usually wrong. A short, accurate note early ages far better than a thorough one late.
Afterwards
Write down what happened, what it cost, and what changed as a result. Add the system to the AI risk register if it was not there, and revisit whether it should run at all — which is a live option, not a failure. NIST's AI Risk Management Framework treats that as part of "manage", not as an exception.
Frequently asked questions
What counts as an AI incident? Anything materially wrong that reached a customer, anything touching money or contracts, any data exposure, and any pattern of small wrong outputs. That last one causes the most cumulative harm and is the most often missed.
Who should be able to switch an AI system off? Whoever owns the AI portfolio, with the technical ability to do it directly. Requiring a ticket or a committee turns a short incident into a long one.
How would we even detect a problem? Sampled human review on a schedule, watching the rework rate of the team using the system, and an easy internal route to flag bad output. Most organisations have no detection at all and find out from a customer.
Should we tell customers? Usually yes, early and briefly. Where personal data is involved there may also be regulatory obligations, so check the applicable guidance rather than deciding on instinct.
Further reading
Tagged with:
Ready to Automate Your Business?
Let's discuss how AI automation can deliver measurable ROI for your organization in 90 days or sooner.