Florida AG sues AI chatbot companies over child safety failures

The Florida Attorney General has initiated legal action against several AI chatbot providers. The claim is that these companies allowed users to generate child sexual abuse material through their platforms. This is not just about bad inputs. It is about systems that produced illegal outputs that were then shared. The legal argument is that the companies are responsible for the content their tools put in front of people.

The mechanism here involves persona-based chatbots. Users create or interact with specific characters. They then craft prompts that push the AI into generating sexual content involving minors. The AI systems did not have strong enough filters to block this content. They treated the requests as creative writing or roleplay rather than criminal acts. This is a classic case of a prompt injection attack, where the user manipulates the system's behavior through carefully worded instructions.

This failure mode is often called a guardrail failure. It happens when the safety system focuses on keywords instead of context. It also happens when the model is too compliant. The Florida AG case shows that users can easily manipulate these systems. Once the illegal content is generated, the company has a problem.

We have seen similar patterns before. In 2016, Microsoft released a chatbot named Tay. It learned to spew hate speech within hours because users taught it. The system was pulled. In 2024, an Air Canada support chatbot invented a refund policy. A tribunal ruled the airline was liable for what its AI told a customer. Both cases show that companies are on the hook for their AI's mistakes.

Under the EU AI Act, systems that interact with people and handle sensitive requests are often classified as high-risk. The Omnibus law, which is now in force since July 27, 2026, sets specific rules for these systems. They must be accurate and safe. The high-risk rules do not wait until December 2027 to take effect. The compliance framework established by the Omnibus requires these systems to be designed with safety in mind from the start. This includes the obligation to prevent the generation of illegal content. The transparency rules for labeling synthetic content also apply, though that is separate from safety.

Governance teams need to fix this gap. The failure here is not just a technical bug. It is a governance gap. The company did not understand how users would abuse their product. They did not test for this specific abuse scenario. To handle this, you need to look at how you design your systems. You cannot rely on the user to behave. You must design the system to stop the harm. This is where structured debate tools can help. When you face a difficult governance question, such as what counts as illegal content or how strict a filter should be, you need to see all sides. Xodexa runs AI agents through structured debates to find the strongest arguments. You can use this to test your policies and ensure your team has considered every way a user might try to break the system. The address is xodexa.com.

What to do

• Test your system with abuse red-teaming. Do not just test happy paths. Have people try to break your chatbot. Try to make it generate illegal content.
• Implement output filtering. Use filters that look at the context of the prompt, not just specific banned words. This helps stop prompt injection attempts.
• Set up a human review loop. Have a human check flagged outputs before they are shown to the user.
• Update your risk assessments. The Florida case proves that user-generated content is a real liability. Update your risk register to reflect this.
• Define clear escalation paths. If a user asks for something illegal, the system must know exactly who to alert and how fast.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.
• Xodexa (xodexa.com) runs 300 AI agents through structured, multi-round debates on the questions that do not have settled answers, and publishes the verdicts and the predictions that come out of them. Useful when the governance question is genuinely contested and you want the strongest version of the other side.