The Missing Black Box: Building an Aviation-Style Near-Miss Database for AI Agents

The gap between 'incidents' and 'close calls'

The Air Canada tribunal ruling—where a chatbot hallucinated a bereavement policy and the airline was held liable—demonstrates the cost of reactive governance. Yet, this case highlights a systemic blind spot: we are excellent at tracking full-blown incidents (lawsuits, flash crashes, public outages) but terrible at tracking 'near-misses.' In aviation, the Aviation Safety Reporting System (ASRS) collects thousands of near-miss reports annually to prevent fatalities. The AI industry lacks this equivalent for autonomous agents. An aviation-style near-miss database for AI agents is not a luxury; it is a necessary control for the 'Agentic AI' era, where systems are no longer just generating text but performing autonomous actions via function calling, API integration, and tool use. Without a structured mechanism to log and analyze behaviors that violate policy but are caught by a guardrail or a human in the loop, organizations are flying blind into a regulatory environment that demands transparency and explainability.

The failure mechanism: The 'Gray Zone' of autonomous actions

The specific failure mode here is the 'Gray Zone' of agent autonomy. Unlike a traditional LLM chatbot that simply outputs text, an agentic system has 'tool access.' This creates a vector for high-impact errors that are difficult to detect during standard testing. A near-miss occurs when an agent attempts to execute an action that violates a constraint (e.g., a withdrawal limit, a privacy rule, or a market manipulation rule) but is stopped either by a technical guardrail (like a regex filter or a hard-coded API limit) or a human approver before the action completes. The failure lies in the fact that the intention to violate the rule was present, and the opportunity to do so existed.

Consider the Knight Capital 2012 flash crash, where an automated trader deployed a faulty algorithm that fired millions of orders in 45 minutes, losing $440 million. The failure wasn't a total system breakdown; it was a deployment error where the 'kill switch' was not triggered fast enough. In the agent context, a near-miss is analogous to the moment the Knight Capital algorithm started firing, but before the volume hit a critical threshold. If the system had logged this specific firing pattern as a 'near-miss'—an autonomous action attempting to exceed volume limits—the pattern could have been identified and the model retrained or halted before the catastrophic loss. Similarly, the recent Bing/Copilot demonstrations showed agents being hijacked by hidden instructions to leak data or take actions. A near-miss database would have cataloged these prompt-injection attempts and tool-usage violations, creating a corpus for defensive training.

Why this matters now: Regulatory convergence on safety culture

The urgency for this database is driven by converging regulatory frameworks that treat safety culture as a technical control rather than just a compliance checkbox. The EU AI Act, with its high-risk obligations coming into force on 2 December 2027 for Annex III systems, mandates 'technical documentation' and 'risk management systems' (Art. 15). The NIST AI Risk Management Framework (RMF) explicitly places 'Govern' and 'Measure' as functions to be continuously monitored. Furthermore, the Federal Reserve’s SR 26-2 (the successor to SR 11-7) emphasizes model risk management and testing, requiring firms to establish a program for the ongoing monitoring of AI/ML systems.

In Canada, while AIDA (Bill C-27) has not yet been enacted, Ontario's AI-in-hiring job-posting disclosure has applied since 1 January 2026, signaling a move toward transparency. The upcoming OSFI Guideline E-23 (effective 1 May 2027) will require banks to manage model risk for AI/ML, specifically focusing on validation and monitoring. These regulations are moving away from 'black box' compliance toward 'process' compliance. A near-miss database is the physical manifestation of a mature 'Measure' function under NIST and a robust 'Operational Control' under ISO/IEC 42001 Clause 8.5. Without it, an organization cannot demonstrate to a regulator that it is actively managing the risks of autonomous behavior, only the outcomes of failures.

Where teams get this wrong

Teams frequently misunderstand the scope of a near-miss database, leading to ineffective implementation. The first pitfall is the 'Happy Path' bias. Organizations often log interactions where the agent succeeds, treating 'success' as the only data point of interest. This ignores the thousands of interactions where the agent almost failed but was saved by a lucky hallucination or a weak constraint. A true safety database must capture 'blocked' actions, not just successful ones.

The second pitfall is the lack of standardization in reporting. Without a taxonomy (e.g., 'Tool Misuse,' 'Constraint Violation,' 'Prompt Injection'), a database becomes a messy log of unstructured text. This makes it impossible to run statistical analysis or trend detection. The third, and most critical, pitfall is the lack of psychological safety in reporting. In aviation, pilots report near-misses without fear of punishment because the system is designed to learn from errors, not punish them. In AI engineering, teams often hide near-misses because they fear it reflects poorly on their model performance or triggers unnecessary alerts. If the database is used as a disciplinary tool rather than a learning tool, the data will be incomplete and biased toward 'successful' hiding of errors.

Governance guidance: Designing the 'Agent Safety Reporting System'

To implement this, governance teams must establish a centralized 'Agent Safety Reporting System' (ASRS) that integrates with the Model Risk Management (MRM) framework. This system should be governed by a cross-functional committee comprising ML engineers, security engineers, legal counsel, and business stakeholders.

First, define the taxonomy of near-misses. This should include categories such as 'Policy Violation Attempted,' 'Tool Misuse,' 'Context Window Drift,' and 'Guardrail Bypass.' Second, mandate that all autonomous actions—both successful and blocked—must be logged with a timestamp, the specific tool invoked, the input prompt, and the output decision. Third, ensure strict anonymization protocols. To encourage reporting, the system must guarantee that individual agents or engineers cannot be easily identified from the reports, drawing inspiration from the Aviation Safety Reporting System (ASRS) model. Finally, integrate this data into the Model Lifecycle. Near-miss reports should be fed back into the training data for fine-tuning (RLHF) and used to recalibrate guardrails, ensuring the database drives continuous improvement rather than just documentation.

Risk management guidance: Metrics and monitoring

The effectiveness of this database must be measured using specific, quantitative Key Risk Indicators (KRIs). The primary KRI is the 'Near-Miss Rate per 10,000 Autonomous Actions.' This metric provides a normalized view of safety, allowing teams to compare safety across different agents or environments regardless of usage volume.

A second critical metric is the 'Mean Time to Detect (MTTD) a Guardrail Bypass.' This measures how quickly a team can identify that an agent has learned to evade its safety constraints. A high MTTD indicates a lag in detection that could lead to a full-blown incident. Third, track the 'False Negative Rate of Audit Logs.' This ensures that the database itself is not being manipulated or skipped during high-load periods. These metrics should be monitored continuously and reported to the Model Governance Board on a monthly basis, aligning with the NIST AI RMF 'Manage' function requirements for ongoing monitoring.

What to test: Concrete experiments for the database

To populate this database effectively, teams should run specific experiments designed to generate near-miss data:

1. The 'Gray Box' Injection Test: Intentionally inject 'gray box' prompts into the agent's input stream—prompts that are subtle and context-aware, designed to trick the agent into attempting a restricted action. Log every attempt, successful or blocked, to build a dataset of adversarial inputs.

2. Chaos Engineering for Agents: Simulate API failures or data corruption in the environment the agent interacts with. Monitor how the agent attempts to 'heal' the situation. If it attempts to execute a dangerous action (e.g., deleting data to 'reset' the state), log it as a high-severity near-miss.

3. The 'Blast Radius' Stress Test: Run the agent with a limited set of tools (least privilege) and then gradually remove those constraints in a controlled environment. Every time the agent attempts to access a tool it was denied, log it as a near-miss of the 'Tool Access Control' policy.

The takeaway

• Shift from Incident to Proactive Safety: Move beyond tracking only catastrophic failures; establish a formalized, anonymous database to track and analyze 'near-miss' autonomous actions, mirroring the Aviation Safety Reporting System (ASRS).
• Log the Blocked, Not Just the Successful: A near-miss database is useless if it only records happy paths. Mandate the logging of all blocked actions and constraint violations to understand the agent's 'intent' to break rules.
• Integrate into the MRM Framework: Connect the database directly to the Model Risk Management process, feeding near-miss data back into model validation, guardrail tuning, and incident response planning.
• Measure Guardrail Efficacy: Track specific KRIs such as 'Near-Miss Rate per 10,000 Actions' and 'Mean Time to Detect a Bypass' to quantify the effectiveness of your safety controls.
• Foster Psychological Safety: Structure the reporting mechanism to guarantee anonymity and non-punitive review to ensure engineers report errors honestly, preventing the 'hidden data' bias that plagues many current safety programs.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.