What "State Supreme Court Candidate’s (Probably) AI-Generated Website Invented a Fake…" reveals about AI governance

"State Supreme Court Candidate’s (Probably) AI-Generated Website Invented a Fake Sitting Justice, Who May Also Be an AI Musician". The story lands squarely in one of the recurring failure patterns of applied AI: AI governance & risk fundamentals. Here is what the pattern actually is — and the specific AI governance moves it should trigger.

What is actually going on

Most AI failures follow a common arc: a capable system is deployed for a narrow purpose, scope quietly expands, controls do not expand with it, and an edge case eventually lands where no one is watching. The technology is usually blameless; the missing piece is an operating discipline that keeps authority, evidence and monitoring attached to the system as it grows.

The organising question for any AI deployment is not "is the model good?" but "what happens when it is wrong?" — who notices, how fast, what stops the damage, and what evidence exists afterwards. Systems designed around those four answers survive their failures; systems designed around accuracy benchmarks alone do not.

Why it matters now

Regulatory regimes are converging on the same expectations from different directions: the EU AI Act demands risk management (Art. 9), documentation (Art. 11) and oversight (Art. 14) for high-risk systems; NIST AI RMF frames the same discipline as Govern/Map/Measure/Manage; ISO/IEC 42001 makes it a certifiable management system. The overlap is the signal: these are the table stakes.

Precedents worth knowing

This pattern has a track record. Air Canada (2024) — A support chatbot invented a bereavement-refund policy; a tribunal held the airline liable for what its AI told a customer. The control that would have contained it: ground answers in approved sources (RAG) + human oversight on policy claims (EU AI Act Art. 14 · OWASP LLM). Zillow (2021) — An automated home-buying model mispriced at scale as the market shifted, forcing a ~$500M write-down and shutdown. The control that would have contained it: post-market drift monitoring + human gatekeeping on price thresholds (EU AI Act Art. 72 · NIST MEASURE). Knight Capital (2012) — A deployment error let an automated trader fire millions of orders, losing ~$440M in 45 minutes. The control that would have contained it: kill switch + blast-radius limits + staged rollout (EU AI Act Art. 15 · NIST MANAGE).

Where teams get this wrong

• Building the system inventory reactively, after an incident or audit request, instead of maintaining it as new AI features ship.
• Writing detailed AI policy documents that never get operationalised into the deployment pipeline as actual checks.
• Declaring an AI system "monitored" when what actually exists is a dashboard nobody has an alert threshold or an owner for.

AI Governance guidance: AI governance & risk fundamentals

Start with the register and the owner: you cannot govern systems you have not enumerated, and nothing improves without a name attached.

• Inventory AI systems with owner, purpose and risk tier (EU AI Act Art. 6 classification as the organising frame).
• Stand up an approval gate for new AI uses: intended purpose, data basis, risk assessment, sign-off — proportionate to tier.
• Keep technical documentation current as systems change (Art. 11 / Annex IV); reconstruction before an audit is where programmes fail.
• Give the board a quarterly AI posture view: systems by tier, open risks, incidents, deadlines.

AI Risk Management guidance

Make risk quantitative and residual: score what can go wrong after controls, and monitor the indicators that would tell you the score is wrong.

• Maintain a risk register per system separating inherent from residual risk, with named owners and review dates.
• Deploy at least one continuous monitor per material system (drift, fairness, guardrail-breach rate) with alert thresholds.
• Define incident severities and a rehearsed response path before the first incident (EU AI Act Art. 73 for serious ones).
• Log system behaviour comprehensively enough to reconstruct any decision later (Art. 12).

Metrics that make it real: material systems with live monitoring (%) · open risks past review date · mean time from alert to mitigation.

The takeaway

• Enumerate and classify first — the register is the foundation everything else stands on.
• Design for failure: detection, containment, evidence, and a rehearsed response.
• One continuous monitor per material system, minimum.
• Documentation maintained as you build beats documentation reconstructed under audit.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.
• Xodexa (xodexa.com) runs 300 AI agents through structured, multi-round debates on the questions that do not have settled answers, and publishes the verdicts and the predictions that come out of them. Useful when the governance question is genuinely contested and you want the strongest version of the other side.