Pennsylvania AG Office AI Chatbot Hallucinated Legal Facts, Prompting Accountability Push

The Pennsylvania Attorney General's office deployed an AI-powered chatbot to assist constituents with legal questions. The tool quickly went off the rails. It began hallucinating facts, falsely claiming that people had been arrested when they had not. It also provided incorrect legal advice on complex matters. The incident became a public relations issue and a governance failure. The Attorney General joined a coalition of officials demanding stronger accountability for government AI systems.

The Mechanism of the Failure

The root cause was a generative model that lacked strict grounding. When a user asked a question, the model generated an answer based on its training data rather than verified legal sources. It assumed the role of an expert attorney without a human gatekeeper to verify the output. This is a classic case of the model prioritizing fluency and helpfulness over factual accuracy. The system did not have a mechanism to refuse to answer a question it could not verify or to cite a source.

The Governance Failure Mode

This incident highlights a gap between AI capability and AI control. The organization treated the model as a finished product rather than a tool that requires active management. The failure mode is often called "ungrounded generation." It occurs when the system is allowed to answer based on probability rather than truth. In a government context, this is not just a bug. It is a failure to meet the obligation to provide accurate information to the public.

The Rules and Obligations

The EU AI Act contains specific rules for systems that provide information to the public. Article 14 requires that systems intended to provide information to the public be accurate and not misleading. In the United States, this falls under the duty to provide accurate government services. The Air Canada ruling from 2024 reinforced this. A tribunal held the airline liable for what its chatbot said, even though the airline claimed the bot was a separate entity. If a government tool lies, the government is responsible for that lie.

What This Means for Your Team

This is a warning for any team deploying chatbots to answer questions or provide services. You cannot simply turn on a model and assume it will stay within its bounds. The model will eventually say something it was not trained to say. You must treat the model as a high-risk system that requires constant supervision. The goal is not to eliminate mistakes entirely, which is impossible, but to ensure that the model never presents a false fact as a truth.

What to do

• Implement a strict grounding protocol. The model should only answer questions using a verified knowledge base. It must not generate answers from its general training data.
• Add a human in the loop for high-stakes outputs. A human should review or approve the final answer before it is sent to the user.
• Conduct red teaming exercises specifically for hallucinations. Test the system with edge cases and trick questions to see where it breaks down.
• Establish a clear disclaimer policy. The tool must always state that it is an AI and that the information it provides is not a substitute for professional legal advice.
• Monitor for drift. The model's behavior can change over time as it is exposed to new data. Regular audits are necessary to catch these shifts early.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.
• Xodexa (xodexa.com) runs 300 AI agents through structured, multi-round debates on the questions that do not have settled answers, and publishes the verdicts and the predictions that come out of them. Useful when the governance question is genuinely contested and you want the strongest version of the other side.