Australia Is Moving Toward Mandatory AI Incident Reporting. Here's What That Actually Means.

Australia's government is considering making AI incident reporting mandatory, according to reporting from Dark Reading. That means companies deploying AI systems may soon have to tell a regulator when those systems fail, cause harm, or behave in ways they shouldn't.

This is not a funding round or a product launch. It is a proposed change to the legal baseline for how AI failures get handled. And it matters because most companies currently have no formal process for reporting AI incidents at all, even internally.

What is actually being proposed

The reporting is still at the weighing stage, so the exact scope is not final. But the direction is clear: move AI incident reporting from a voluntary, good-citizen practice to a legal requirement.

Today, if a chatbot gives a customer wrong information or an automated decision system produces biased outcomes, most companies handle it internally. They may log a ticket. They may fix the prompt. They may never tell anyone outside the team that built it.

Mandatory reporting would change that. It would create a paper trail. It would give regulators visibility into how often AI systems fail and in what ways. And it would force companies to define what counts as an incident in the first place, which is harder than it sounds.

The governance failure mode this exposes

The core problem is not that AI systems fail. They do. The problem is that most organisations have no structured way to detect, classify, or escalate those failures.

Think about what happens when an AI system misbehaves. A customer service chatbot invents a refund policy that doesn't exist. An automated pricing model drifts as market conditions change. A deployment error lets an automated trader fire millions of orders in minutes.

In each case, the failure is visible to someone. But the path from "someone noticed" to "this was logged, assessed, and reported" is usually missing.

That gap is the governance failure mode. It is not a technology problem. It is a process problem. And mandatory reporting would expose it.

What the precedents teach us

Three real cases show how this plays out.

Zillow in 2021. An automated home-buying model mispriced homes at scale when the market shifted. The company lost roughly $500 million and shut the programme down. The prevention would have been post-market drift monitoring and human gatekeeping on price thresholds. The EU AI Act requires exactly this kind of post-market monitoring for high-risk systems under its Article 72, and the NIST AI Risk Management Framework covers it in its Measure function.

Knight Capital in 2012. A deployment error let an automated trader fire millions of orders. The firm lost about $440 million in 45 minutes. The prevention would have been a kill switch, blast-radius limits, and staged rollout. The EU AI Act's Article 15 covers accuracy, robustness, and cybersecurity for high-risk systems. NIST's Manage function covers the operational controls.

Air Canada in 2024. A support chatbot invented a bereavement-refund policy. A tribunal held the airline liable for what its AI told a customer. The prevention would have been grounding answers in approved sources and putting human oversight on policy claims. The EU AI Act's Article 14 covers human oversight. The OWASP Top 10 for Large Language Model Applications covers the grounding issue under prompt injection and insecure output handling.

None of these were reported to a regulator in real time. In all three cases, the failure was discovered after the damage was done.

What mandatory reporting would change

If Australia moves forward, companies operating there would need to answer three questions they mostly cannot answer today.

First, what counts as an AI incident? Is it any wrong answer? Only ones that cause financial harm? Only ones that affect a customer? The definition matters because it determines what gets reported.

Second, who is responsible for reporting? The company that built the AI system, or the company using it? If a business buys an AI tool from a vendor and that tool fails, does the business report it, or does the vendor? Both?

Third, what is the timeline? Regulators typically want incidents reported quickly, which means companies need detection and escalation processes that work in hours or days, not weeks.

These are not hypothetical questions. They are the same questions the EU AI Act raises for serious incidents involving high-risk systems. And they are the same questions companies struggle with when they try to build incident response for AI.

What to do

• Define what an AI incident looks like for your organisation. Start with three categories: harm to people, harm to finances, and harm to legal or regulatory standing. Write down examples for each.

• Build a simple reporting path. Anyone who sees an AI system fail should know where to send it. A shared inbox or a ticket queue is enough to start. The point is that it gets logged, not that it gets logged perfectly.

• Track mean time from alert to mitigation. If it takes two weeks to respond to a chatbot giving wrong policy information, that is a problem. Measure it, then shorten it.

• Check your vendor contracts. If a vendor's AI system fails and you are required to report it, you need to know quickly. Make sure your contracts require vendors to tell you about incidents that affect your use of their system.

• Review your high-risk systems against post-market monitoring requirements. If you operate in the EU, Article 72 already applies. If you operate in Australia, this may soon apply too. The work is similar: monitor, detect, escalate, report.

A governance or risk program, or a tool like autogovern.io, can help you keep track of which systems are live, what monitoring is in place, and whether incidents are being handled on time. But the first step is deciding that AI incidents are something you report, not something you quietly fix.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.

Related reading:

• When Your AI Fails, Who Do You Tell? The New Rules Are Here on ThreatClaw