AI Governance in Regulated Industries: What Changes
Regulated firms usually over-build the novel parts and under-build the boring parts. The regulator tends to ask about the boring parts.
Erin Moore
Fractional Chief AI Officer
If you are in a regulated sector, most of what you need for AI governance already exists somewhere in your model risk, vendor risk and records management frameworks. The work is extending them, not inventing a parallel regime — and the firms that build a separate AI governance tower usually end up with two frameworks that disagree.
Nothing here is legal or compliance advice, and obligations differ sharply by sector and jurisdiction. Verify against your own regulator's current guidance and your counsel.
What does not change
Your existing obligations. If a decision had to be explainable, documented, auditable and reviewable before, it still does — the fact that a model produced it changes nothing about the duty. Banking supervisors have had model risk management expectations for well over a decade, and those expectations were deliberately written to be technology-agnostic.
Vendor oversight does not change either. Outsourcing a process to a vendor's AI does not outsource your accountability for it. This is the point most AI procurement conversations skip, which is why the AI vendor security questionnaire matters more here than anywhere else.
What genuinely does change
The output is not deterministic. Traditional model validation assumes the same inputs produce the same outputs. Generative systems do not guarantee that, so validation has to move from "verify the answer" to "characterise the distribution of answers and set tolerances". This is the genuine methodological gap, and it is where your model risk team should be spending their effort.
The audit trail has to include the prompt. Recording that a model was used is not enough. The input, retrieved context, model version and output together are the record. Retrofitting this after an examination request is expensive; building it in is cheap.
Version changes are silent. A hosted model can change beneath you without a release note you will notice. Regulated use demands pinned versions and a validation gate on every change — which is a contract term, not a technical control. Add it at purchase, per AI contract negotiation.
Scope creep is the real exposure. A system validated for drafting internal summaries gets quietly pointed at customer communications. The control is periodic re-attestation of actual use against approved use, not a one-time approval.
Where to anchor the framework
Map to something recognised rather than inventing structure. NIST's AI Risk Management Framework and its playbook give you govern, map, measure and manage functions that map cleanly onto existing three-lines-of-defence models. The OECD's AI Principles are useful for the accountability language. If you operate in or sell into the EU, the AI Act text is authoritative on which tier you fall into, and the UK ICO's AI and data protection guidance is unusually practical on the data protection overlay.
Using an external framework also answers the question every examiner asks — what standard are you holding yourselves to — with something other than a document you wrote about yourselves.
What regulated firms consistently get wrong
They over-invest in the novel and under-invest in the mundane. Elaborate bias testing on a system that drafts internal meeting notes, alongside no inventory of which AI tools are actually in use and no incident path.
Start with the inventory, per how to find shadow AI — you cannot govern systems you cannot list, and "we don't know" is the answer that damages you most in an examination. Then the incident response plan. Then risk-tiered controls, with real scrutiny reserved for systems touching customers, money or eligibility.
Frequently asked questions
Do we need a separate AI governance framework? Usually not. Extend your existing model risk, vendor risk and records management frameworks. A parallel AI tower tends to produce two frameworks that contradict each other.
What genuinely changes with generative AI? Outputs are non-deterministic, so validation shifts from verifying an answer to characterising a distribution and setting tolerances. The audit record must also include the prompt, retrieved context and model version.
What is the biggest exposure? Scope creep — a system approved for one use quietly applied to another. Control it with periodic re-attestation of actual use against approved use, not a one-time sign-off.
Where should a regulated firm start? An inventory of AI systems actually in use, then an incident path, then risk-tiered controls. Firms routinely build sophisticated testing before they can list what they are running.
Further reading
Tagged with:
Ready to Automate Your Business?
Let's discuss how AI automation can deliver measurable ROI for your organization in 90 days or sooner.