E-23 Will Fail at the Validation Cycle, Not the Paperwork

OSFI Guideline E-23 will fail because it validates AI models at a point in time, and AI models do not hold still. The failure will show up first in Canadian banks' incident counts, not their audit findings.

What most people think

The consensus in Canadian financial risk is reasonable and wrong. E-23 is read as a model risk management guideline, a familiar genre. Banks have run model validation for decades: credit scoring, market risk, capital models. The instinct is that AI and machine learning models are more models, so you extend the existing validation function to cover them, document the methodology, get sign-off, and file it.

This is not lazy thinking. It is the strongest version of the case. Validation is a real control. It forces independent challenge, it creates an audit trail, and it catches badly specified models before they touch a customer. Regulators trust it because it has worked. Extending a proven control to a new model class is exactly what a mature risk function should try first. The problem is not the intent. The problem is the cadence.

What the data shows

Our live incident database, which tracks reported AI failures from public news, holds 1,500 stories over the last 180 days. The volume is steady: 463 in the last 45 days against 482 in the 45 before.

What is not steady is the mix. Governance stories rose 27 percent, from 193 to 220. Compliance stories fell 35 percent, from 80 to 45. Privacy rose 22 percent. Model stories went from 1 to 0.

That last number is the one that matters. The MIT AI Risk Repository catalogues a risk class called lack of capability or robustness. In 180 days, our database has zero stories matching it. Overreliance and unsafe use: zero. Environmental harm: zero. Meanwhile, privacy leakage drew 196 stories and multi-agent risks drew 119.

The press covers what is visible and dramatic. It does not cover a model that quietly stopped working the way it was validated to work. That failure mode, the one E-23 exists to catch, is invisible in the news. So a bank cannot benchmark its exposure against the market. It has to find it internally, and the validation file is not built to hold it.

Why this happens

Model validation attests to behaviour at a moment. The validator tests the model, reviews the data, checks the assumptions, signs. That signature is true on the day it is written.

An AI or machine learning model in production is not the same object a month later. It gets retrained on fresh data. The world drifts away from its training distribution. Someone changes a prompt, swaps a tool, updates a retrieval index, or connects a new downstream system. Each of those changes behaviour. None of them triggers a re-validation under an annual cadence.

So you get the worst possible artefact: a model that has drifted into non-compliance, sitting in a file that says compliant. The audit trail is not lying on purpose. It is accurately recording a test that has expired. When the incident happens, the first thing the bank will say is that the model was validated. That sentence will be true and useless.

This is why the failure appears in incident counts before audit findings. Audits check whether the process was followed. The process was followed. Incidents check whether the model behaved. It did not.

The best argument against this

The honest objection is that validation is not meant to be a one-time test. Good model risk practice already includes ongoing monitoring, back-testing, and change management. A well-run bank does not validate in April and go to sleep. It watches the model, and it re-validates on material change. If that is true, E-23 is fine, and the failure I am describing is just bad implementation by bad banks.

That objection is correct about the best banks and irrelevant to the guideline. Monitoring exists, but it is usually operational monitoring: is the model up, are outputs in range, did the pipeline break. It is not behavioural attestation against the validation assumptions. And the sign-off artefact still says validated, with a date. Whatever monitoring sits behind it, the document a regulator or a court reads is the validation report, and that report has an expiry the bank did not write down.

E-23 does not require continuous behavioural monitoring as a condition of the validation. It requires validation. Banks will meet the letter, and the letter is the problem.

What I think happens next

By October 2028, at least one Canadian bank or federally regulated insurer will disclose, in a public filing or a trade-press story, an AI or machine learning model incident that occurred inside a period covered by a passing E-23 validation, and the disclosure will describe the model as validated.

E-23 takes effect on 1 May 2027. That gives roughly eighteen months of live validation cycles before my deadline, which is enough time for a drifted model to produce a reportable event.

This claim is wrong if, by October 2028, no such disclosure appears in our feed or in a named Canadian bank's public filings, or if the first disclosed AI incident at a Canadian bank explicitly post-dates its last validation. Either outcome would mean the validation cycle caught the drift in time, and I would take that as good news.

What to do about it

• Replace annual validation with continuous behavioural monitoring for every AI or machine learning model in E-23 scope, and put the monitoring evidence inside the validation file. The validator should be signing against a live feed, not a snapshot.

• Add a behaviour drift since last validation field to the model inventory. Make the validator sign against it. A blank field is a finding.

• Define what counts as material change for each model, in advance. Retraining, prompt changes, tool changes, and data source swaps should all trigger review, not just a new model version.

• Track your own incident counts by model, not just by category. If your model-related incidents are zero while your governance incidents climb, you are probably not measuring the right thing.

• Read the security side of this. ThreatClaw's piece on why your AI's compliance paperwork might be lying to you is at https://www.threatclaw.ai/blog/your-ais-compliance-paperwork-might-be-lying-to-you, and it makes the same point from the attacker's direction. A governance or risk programme can help you build the monitoring evidence, but the discipline of watching the model between sign-offs is yours to keep.

The guideline is not the enemy. The calendar is.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.

Related reading:

• Why Your AI Projects Need Their Own Rulebook on ThreatClaw
• Your AI's Compliance Paperwork Might Be Lying to You on ThreatClaw