AI literacy lessons teach kids that chatbots lie. That is a governance lesson for every customer-facing AI.

What happened

Schools in several countries now run AI literacy lessons where kids are taught to test chatbots: ask about a policy, a historical fact, or a math problem, and then check whether the answer is true. The lesson is that chatbots can be confidently wrong. That is a good lesson for students, and it is also a perfect description of the risk every customer-facing chatbot carries today.

Nothing in these lessons is new. The failure mode they demonstrate is the same one that cost Air Canada money in 2024. A customer asked their support chatbot about bereavement refunds. The chatbot invented a policy that did not exist. Air Canada tried to argue in a tribunal that the chatbot was a separate legal entity. The tribunal rejected that and held the airline liable for what its AI told a customer.

The mechanism behind the error

A chatbot does not know what it does not know. It generates the most plausible next words based on its training, not on your actual policy documents. When a user asks a specific question about a refund, a deadline, or a warranty, the model has no internal truth checker. It produces an answer that sounds right. If the answer is not grounded in an approved source, it is a guess dressed as policy.

This is why the Air Canada case is the canonical example. The chatbot was not malicious and not broken in a technical sense. It was ungrounded. It had no mechanism to say "I do not know" or to fetch the real policy before answering. The company's mistake was treating the chatbot as a finished product rather than as a system that needs constant supervision.

The same mechanism appears in Microsoft's 2016 chatbot incident, where users steered a learning bot into producing hateful content. The bot was designed to learn from conversation, and it learned exactly what it was fed. No output guardrails, no red-teaming before release, and it was pulled within a day. Both cases share a root cause: the AI was released without controls that match its failure modes.

The governance failure pattern

This is a customer-facing chatbot accountability failure. The pattern has three parts. First, the AI can make authoritative claims about things that matter, like refunds or policies. Second, those claims are not verified against a trusted source before the user sees them. Third, when the claim is wrong, the organisation struggles to take responsibility because the AI "said" it, not a human.

The Air Canada tribunal settled the legal question: you are responsible for what your AI tells customers. There is no chatbot liability shield. The governance question is how you prevent the error from happening in the first place, and how you detect it when it does.

The controls that apply

There are two practical control layers. The first is grounding. Before a chatbot answers a policy question, it should retrieve the answer from an approved, curated set of documents, not from its memory. This is retrieval-augmented generation, often called RAG. If the answer is not in the approved source, the bot should say it does not know and route the customer to a human. This directly addresses the Air Canada failure.

The second layer is human oversight and guardrails. This maps to the EU AI Act's transparency and human oversight requirements, which apply to AI systems that interact with people. The law expects that a human can intervene or override the system's output. It also expects that the system's outputs are traceable. For a chatbot, that means logging every answer, the source it used, and the confidence it had. If a customer disputes a statement, you must be able to reconstruct exactly what was said and why.

For teams using the OWASP LLM framework, the relevant controls are about output validation and sensitive information disclosure. You should test your bot with adversarial inputs before release, and you should monitor live conversations for authority breaches, where the bot states a policy that is not in your approved sources.

What this means for your chatbot

If you run a customer-facing AI, the school lesson is your risk register. The kids are testing your bot. So are your customers, your competitors, and eventually a regulator or a judge. The question is not whether your bot will make a false policy claim. It is how quickly you will find out and whether you can prove what happened.

The cost of not doing this is not just a bad review. It is a tribunal finding you liable, a regulator asking why you had no human oversight, and a reputation hit that no AI convenience is worth.

What to do

• Ground your chatbot's answers in an approved document store. If the answer is not in the source, the bot must say it does not know and hand off to a human.
• Log every conversation with enough detail to reconstruct the exact claim, the source used, and the time. Aim for 100% of disputed statements being recoverable from logs.
• Track an authority-breach rate per 1,000 conversations. If the rate is above zero, investigate the root cause and fix the source or the retrieval logic.
• Run red-team tests before every release, including prompts designed to trick the bot into inventing policies or repeating harmful content.
• Make sure a human can override the bot in real time. This is not optional under the EU AI Act's human oversight expectations, and it is the difference between a controlled system and a liability generator.

A governance or risk program can help you set these controls, track the metrics, and run the red-team cycles. But the first step is accepting that your chatbot is not a finished product. It is a system that needs the same supervision as any other employee who talks to customers.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.