The OWASP LLM Top 10 Was the Warm-Up: What Comes Next

The OWASP Top 10 was about vulnerabilities, not outcomes

When the OWASP Top 10 for Large Language Models first came out, it gave security teams a shared language for prompt injection, insecure output handling, and training data poisoning. It was a warm-up. It told you how an LLM could be attacked. It did not tell you what to do when the LLM simply makes something up and a customer or a court relies on it.

That is the gap the next wave of AI governance is about to fill. Regulators and courts are no longer asking whether your model can be hacked. They are asking whether you can prove you stopped it from telling a damaging lie.

The failure pattern: unsupported claims that reach real people

Take the two most cited examples. In 2023, a lawyer used an LLM to draft a legal filing. The model invented court cases that did not exist. The lawyer filed them. The court sanctioned him. The mechanism was not a prompt injection or a data leak. The model simply generated text that sounded plausible and had no grounding in any real source.

A year later, Air Canada's support chatbot invented a bereavement-refund policy. A customer followed it, spent money, and the airline refused to honor it. A tribunal held the airline liable for what its AI told a customer. Again, no exploit. Just a model that produced an unsupported claim and the company had no process to catch it before it reached a human.

Both are grounding failures. The model did not have a reliable link to approved, verifiable sources, and the organizations did not have a human check on outputs that carried real consequences.

Why this is a governance failure, not just a technical bug

A hallucination is not a security vulnerability in the classic sense. It is a failure of the system to stay within its permitted knowledge boundary. That makes it a risk management problem. The controls you need are not firewalls. They are process controls.

First, you need grounding. The model's answers should be built on a retrieval-augmented generation (RAG) pipeline that pulls from a vetted, versioned source base. If the answer cannot be traced to a source in that base, it should not be sent to a customer or filed in court.

Second, you need human oversight on high-stakes outputs. The EU AI Act's rules on human oversight (Article 14) apply to high-risk systems and require that people can interpret the model's output and intervene. That is not a suggestion. It is a legal obligation for systems in scope. For legal research or customer policy answers, a human check is the difference between a mistake and a liability.

Third, you need to measure the failure rate. You cannot manage what you do not count. Track the unsupported-claim rate on sampled production output, citation coverage on factual answers, and confirmed false statements that reach customers per month. If those numbers are not near zero, you have a live risk, not a theoretical one.

The shift from vulnerability scanning to outcome auditing

Security teams are used to scanning for vulnerabilities and patching. That mindset does not transfer cleanly to LLMs. You cannot patch a hallucination. You can only reduce its likelihood and catch it before it does harm.

That is why the next phase of AI governance will look more like audit than like penetration testing. Regulators will not ask whether you had a prompt-injection filter. They will ask whether you had a process to detect and correct false outputs, whether you logged them, and whether you acted on them.

The OWASP Top 10 gave you the checklist. The next wave demands you show the evidence. If you are building or deploying LLMs for customer-facing or regulated use, start treating grounding and human oversight as core controls, not as nice-to-haves.

ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems, with live intelligence feeds and detection rules, because a control is only as good as the threat it is sized against. But the biggest threat to your reputation may not be an attacker. It may be an answer your model gives with total confidence and no source.

What to do

• Map every LLM output that reaches a customer, court, or regulator. Decide which ones carry legal or financial consequences if wrong.
• For those high-stakes outputs, require grounding in an approved source base and a human review step before release. No exceptions.
• Start measuring your unsupported-claim rate and citation coverage on a sample of production traffic. Set a target near zero and escalate when you miss it.
• Review your incident response plan for grounding failures. If a false claim reaches a customer, who retracts it, how do you notify, and how do you prevent the same claim from recurring?
• If you use a vendor for LLM services, check whether they provide logs, source citations, and human oversight options. If they do not, that is a vendor risk you need to document and mitigate.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.