Appeals court fines lawyer $15K for AI-generated fake citations

A judge penalized a lawyer for submitting AI-generated legal research that included non-existent cases. The lawyer used an AI tool to find case law for a legal brief. The tool hallucinated cases that did not exist. The lawyer submitted these fake citations to the court without verifying them. The court discovered the errors and fined the lawyer. This incident highlights the critical need for human oversight when using AI tools.

This case represents a failure of human-in-the-loop controls. The lawyer relied on the AI as a source of truth rather than a tool that requires checking. This mirrors the Northpointe case where a criminal-risk score was used in sentencing without proper scrutiny of its fairness. In both instances, the system was trusted to produce accurate information without a final verification step. The Northpointe system showed racial disparities, and the lawyer's system produced non-existent cases. Both failures resulted in harm. The lawyer's mistake caused embarrassment and financial cost, while the Northpointe system contributed to unjust outcomes.

This failure mode is dangerous because it erodes trust. When organizations rely on AI outputs without verification, they risk introducing systemic errors. If a system like the one in the Netherlands government case is trusted without human review, it can wrongly target protected groups. The lawyer's mistake highlights that even in high-stakes environments, the human must remain the final gatekeeper. You cannot assume an AI tool is accurate just because it is automated. The obligation is on the human user to perform a ground truth check.

This incident serves as a clear lesson for risk management teams. You must establish strict protocols for the use of AI in decision-making. These protocols must mandate the verification of all AI-generated content. A failure to verify can lead to legal penalties and reputational damage. The risk is not just that the information is wrong, but that the organization is seen as negligent.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.