The Silent Risk: Why We're Not Hearing About Overreliance in Generative AI

Overreliance on generative AI is becoming a serious risk that organizations aren't reporting.

The underreporting of overreliance and unsafe use (0 stories in 180 days) combined with the rise in generative AI incidents (29 stories in last 45 days) suggests organizations are deploying powerful generative AI without proper safety protocols.

What most people think

Most people believe that the increasing media attention on generative AI incidents shows that organizations are actively governing these systems. They assume that when we hear more about AI failures, it means organizations are becoming more transparent and that oversight is improving. This view treats all reporting as equal evidence of better governance.

What the data shows

Our live incident database, which tracks reported AI failures from public news, shows a troubling pattern. In the last 180 days, there have been 1309 stories about AI incidents. In just the last 45 days alone, we've seen 469 stories—more than in any similar period.

When we look at specific risk categories, the numbers tell a clearer story. The 'genai' category jumped from 20 stories in the prior 45 days to 29 in the most recent period—a 9-story increase. This shows generative AI incidents are becoming more frequent.

At the same time, the '5.1 Overreliance and unsafe use' risk class has zero reported stories in the past 180 days. This category includes situations where systems make decisions that humans should verify or where users trust AI outputs without proper safeguards.

Even more concerning, documented attack techniques like LLM Prompt Crafting (22 cases) and Evade AI Model (18 cases) aren't showing up in public reporting despite being real documented threats.

Why this happens

Organizations are rapidly adopting generative AI without fully understanding or implementing specific safeguards against known vulnerabilities. The gap between documented attack techniques and reported incidents suggests many organizations don't know how to detect or report these specific risks.

As MITRE ATLAS documents, techniques like prompt crafting and model evasion are real threats with documented cases. Yet our incident database shows no reporting on overreliance risks. This disconnect indicates organizations may lack the proper monitoring systems or reporting mechanisms to catch these issues.

The speed of generative AI deployment has outpaced the development of specific governance practices for these systems. Teams are implementing powerful models without building in the necessary oversight for how humans interact with and depend on these systems.

The best argument against this

The strongest objection is that organizations simply haven't experienced enough overreliance incidents yet. Perhaps these systems are being deployed carefully, and the risks haven't materialized because proper protocols are in place.

This view overlooks two critical points. First, documented attack techniques exist and have been demonstrated in real cases. Second, the rise in generative AI incidents shows these systems are encountering problems—just not the specific overreliance problems we should expect to see.

If organizations had proper safety protocols, we would expect to see some reporting on overreliance cases. The complete absence of such reporting, alongside increasing generative AI incidents, suggests these protocols aren't in place.

What I think happens next

The first major incident resulting from overreliance on generative AI will occur before organizations begin regularly reporting on this risk class. I predict this will happen by September 2027.

This prediction is based on the current pattern of deployment without specific safeguards. As more organizations integrate generative AI into critical decision-making processes, the likelihood of serious overreliance incidents increases.

This claim would be proven wrong if organizations begin regularly reporting on overreliance and unsafe use risks before a major incident occurs. If we see consistent reporting on these issues before September 2027, it would suggest organizations are becoming more aware and transparent about these risks.

What to do about it

1. Conduct specific vulnerability testing for prompt crafting and evasion techniques. Use documented attack patterns to test your systems before deployment.

2. Implement rate limiting and anomaly detection for generative AI systems. These controls can help identify unusual patterns that might indicate overreliance or manipulation attempts.

3. Develop incident response protocols for overreliance scenarios. Create specific plans for when AI systems make critical decisions that should have human verification.

4. Establish regular reporting mechanisms for overreliance risks. Create internal processes to track and report on situations where systems are being used in ways that create unsafe dependencies.

5. Review your systems against documented attack techniques. Compare your safeguards against known vulnerabilities like prompt crafting and model evasion.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.

Related reading:

• How to Stress-Test Your AI Before Hackers Do on ThreatClaw
• Hackers Read the Government's 'Must Patch' List Too on ThreatClaw