Your AI incident count measures headlines, not harm

The incident count your board sees is a measure of editorial attention, not of harm.

Our live incident database, which tracks reported AI failures from public news, shows that the top five outlets carry just 7% of stories in the last 45 days. That means 93% of what gets counted is derivative coverage. The number on the slide is not a census. It is a measure of how many outlets decided to republish a handful of originals.

What most people think

The standard view is that incident databases are censuses. If the count goes up, more incidents happened. If a category count falls 41%, that category got safer. Boards compare category trends across periods as if the denominator were stable and as if every story were an independent observation. This is a reasonable belief. It is also the belief the data contradicts.

What the data shows

Our live incident database, which tracks reported AI failures from public news, ingested 1500 stories in the last 180 days. In the last 45 days it recorded 450 stories. In the 45 days before that, 493. Total volume moved 9%. Category counts moved far more. Compliance fell from 86 to 45, a 41% drop. Governance rose from 194 to 217, up 23%. Fairness fell from 50 to 35. Privacy rose from 74 to 86. Genai fell from 30 to 22. Security fell from 44 to 37. Safety fell from 14 to 8. Model fell from 1 to 0.

A 9% change in total volume cannot mechanically produce a 41% swing in one category unless the composition of stories changed. And the composition did change. The top five outlets carry 7% of the last 45 days: IAPP at 2%, Law360 at 2%, Yahoo Finance at 1%, Fortune at 1%, Reuters at 1%. The remaining 93% is a long tail of low-authority republications.

The severity mix also moved. The last 45 days had 51 critical, 380 major, and 19 minor stories. The prior 45 days had 74 critical, 412 major, and 7 minor. The minor count nearly tripled while the critical count fell by a third. That is not a story about the world getting safer. It is a story about which stories got picked up.

The catalogue gap is starker. The MIT AI Risk Repository lists subdomains that the news barely reports. Over 180 days, overreliance and unsafe use had 0 stories. Environmental harm had 0. Lack of capability or robustness had 0. Competitive dynamics had 2. Meanwhile, privacy leakage had 188 stories, multi-agent risks 116, fraud and manipulation 108, security vulnerabilities 101, and governance failure 41. These are live policy topics with zero or near-zero coverage. They read as zero because no authoritative original was written, not because nothing happened.

The security side shows the same gap from a different angle. Our sister platform ThreatClaw published 40 articles in 60 days, with 24 in ai-security and recurring tags like data poisoning and prompt injection. MITRE ATLAS lists attack techniques with documented real-world cases that the news window never mentions: LLM Prompt Crafting with 22 cases, Evade AI Model with 18, AI-Enabled Product or Service with 16, AI Agent Tool Invocation with 15, User Harm with 12, Financial Harm with 11. These are not hypotheticals. They are catalogued cases with no matching stories in our 180-day window.

Why this happens

News ingestion is not sampling. It is syndication. A single original report from an authoritative outlet gets picked up by dozens of low-authority sites, each counted as a separate story. A category with one strong original can inflate by dozens of stories in a week. A category with no original reads as zero even when the underlying risk is live.

The mechanism has three parts. First, ingestion counts stories, not events. Second, outlets differ enormously in whether they originate reporting or republish it. Third, the long tail is where most of the volume lives. When 93% of stories come from outside the top five, the count is mostly a measure of how far a small number of originals travelled.

This is why total volume can move 9% while a category moves 41%. The category did not change. The editorial attention did. And because boards are shown category trend lines without an authority weight, resource allocation follows editorial fashion.

The best argument against this

The strongest objection is that authority weighting is itself a judgement call. Who decides which outlets count as authoritative? A weighted count can be gamed by labelling friendly outlets as high-authority and unfriendly ones as low. Raw counts are transparent and reproducible. A board can audit a raw count. A weighted count hides a editorial choice inside a number.

That objection is correct about the risk and wrong about the conclusion. The choice is already being made. Every raw count implicitly weights every outlet equally, which is itself an authority judgement, just an unstated one. The question is not whether to weight. It is whether to weight openly or silently. A published methodology that names the outlets, states the weights, and shows the raw count alongside the weighted one is more auditable than a raw count that pretends to be neutral. The fix is transparency, not abandonment.

What I think happens next

By 2028-03, at least one of the major AI incident trackers, whether the OECD AI Incidents Monitor, the AI Incident Database, or a commercial vendor, will publish an outlet-authority or deduplication-weighted incident count alongside its raw count. The pressure is already visible: the EU AI Act's serious-incident reporting obligations under Article 73 apply from 2027-12-02, and regulators will need a count that means something. If by 2028-03 none of the OECD AI Incidents Monitor, the AI Incident Database, or any named commercial tracker publishes a weighted or deduplicated count, the measurement-validity argument has not landed.

What to do about it

• Add a deduplication step that collapses syndicated coverage to the originating outlet before any trend is reported. If you cannot identify the originator, mark the story as derivative and exclude it from the headline count.
• Show the board the top-five-outlet share next to every incident trend chart. A 41% category move next to a 7% authority share is a different conversation than a 41% move on its own.
• Stop comparing category counts across periods until an authority weight is applied. Report direction and named originals, not percentage changes.
• Publish your weighting methodology. Name the outlets, state the weights, and show the raw count alongside the weighted one so the choice is auditable.
• Check your own risk register against the catalogue gaps. If overreliance, environmental harm, and competitive dynamics are absent from your incident feed, that is a coverage problem, not a clean bill of health.

A governance or risk programme can help here, but the measurement fix is yours to make. The count is not the harm. Treat it that way.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.

Related reading:

• Your AI Needs a Safety Net in Layers on ThreatClaw
• Security Checklists Don't Work for AI. Measure These Five Things Instead. on ThreatClaw