Why Your AI Governance Team Is Fighting the Wrong War

Governance programs defend against narratable incidents like multi-agent failures while ignoring the silent risk of human overreliance. We focus on the wrong risk because the news only reports the narratable ones.

What most people think

Most people believe multi-agent risks are the frontier concern. They assume overreliance is a minor human-factors issue with no real-world incidents. This view comes from looking at the headlines and the incident feeds. It creates a false sense of security about the human side of the equation.

What the data shows

Our live incident database tracks reported AI failures from public news. It shows a massive disparity in what is reported versus what is deployed. In the last 180 days, we catalogued 1497 stories. The data from the MIT AI Risk Repository confirms this gap.

The category getting all the attention is multi-agent risks. In the same 180-day window, there are 118 stories about this specific risk class. In stark contrast, the category of overreliance and unsafe use has 0 stories. This zero count persists even though the deployed base for these systems is enormous.

The severity mix of incidents is also shifting. In the last 45 days, we saw 52 critical incidents versus 72 in the prior period. The number of minor incidents rose from 7 to 19. This suggests the reporting mix is moving toward less narratable events, not more.

Top outlets carry very little of the total load. The top five outlets carry just 7% of the stories from the last 45 days. This fragmentation means a specific risk can exist in the background without ever becoming a headline.

Why this happens

The mechanism is narratability. News values and incident taxonomies both select for events with a clear agent and a clear victim. A multi-agent failure has a named system and a named victim. It fits the template of a story.

An overreliance incident produces only a bad decision. The harm is real, but it is attributed to the human. There is no named agent to blame, so it does not meet the threshold for a story or a formal report. The risk is real, but it is structurally unreportable. Because it never enters the agenda, it never gets a governance program.

The best argument against this

The strongest objection is that overreliance is simply rare. The argument is that we do not see stories about it because it rarely happens. If it were a common problem, the news would cover it.

The answer is that the deployed base is so large that the frequency of incidents does not matter. The risk is present at a scale that should generate noise. The silence is not evidence of rarity; it is evidence of structural invisibility. The gap between the deployed base and the reported incidents proves that the risk is there, even if the stories are not.

What I think happens next

Overreliance will first appear in the incident feed not as its own category but as a contributing factor inside a fraud, safety, or professional-liability story. It will be buried in the details, not the headline. I predict that by September 2027, at least three such stories will name overreliance or AI dependency as a factor before any story is filed directly under overreliance/unsafe use.

This would prove the narratability mechanism is wrong. If by 2027-09 the first overreliance-attributed story in our feed is filed directly under overreliance/unsafe use rather than as a factor in another category, the structural invisibility theory breaks down.

What to do about it

You need to change how you measure risk. Here are three steps you can start this week.

• Add an overreliance check to your incident intake process. For every fraud, safety, or professional-liability event, ask whether a human over-trusted an AI output. Log it as a contributing factor, even if you do not categorize the whole event as an AI failure.

• Run one tabletop exercise in the next quarter. The goal is to simulate a scenario where the AI system behaves correctly and the failure is entirely in human reliance. Most teams have never rehearsed this specific failure mode.

• Audit your response protocols. Security teams have detailed playbooks for system failures, but they often lack a clear chain of command for human error. Look at how your team handles a bad decision made by a human who relied on an AI.

For more on how to prepare for these specific failure modes, check out the team at ThreatClaw. They write about the reality of AI failures in their article "When Your AI Fails, Who Do You Call First?" at threatclaw.ai/blog/when-your-ai-fails-who-do-you-call-first.

The security side of the industry is already thinking about this gap. They focus on the tool and the prompt, which is necessary, but governance must focus on the human. The risk is not in the system. It is in the human who trusts it.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.

Related reading:

• When Your AI Fails, Who Do You Call First? on ThreatClaw
• Your AI Assistant's True Weak Point Is Hidden in Its Own Instructions on ThreatClaw