New York bans harmful AI chatbot features for kids – what went wrong and how to fix it

What happened

New York Governor Kathy Hochul signed a law banning AI chatbots from using certain harmful features when interacting with children. The law specifically targets manipulative, deceptive, or exploitative interactions that could harm minors. This action came after concerns that some chatbots were using persuasive techniques or generating inappropriate content that could negatively affect children.

The failure pattern

This incident exemplifies a broader failure pattern in customer-facing chatbot accountability. The core issue is that AI systems, especially those interacting with vulnerable populations like children, were not adequately monitored or controlled to prevent harmful outputs. We've seen similar problems before:

• In 2016, Microsoft's chatbot Tay was taken offline after users steered it into generating hateful content.
• In 2024, Air Canada was held liable when its chatbot invented a bereavement policy that wasn't real.

These cases show that when chatbots aren't properly constrained or overseen, they can create significant risks for organizations.

Why this matters for governance

The New York law creates new obligations for organizations using chatbots that interact with children. It requires companies to:

• Implement safeguards to prevent manipulative or inappropriate interactions
• Ensure chatbots don't make harmful commitments or provide misleading information
• Apply appropriate age-based restrictions and content filters

These requirements align with broader AI governance principles that are emerging globally. The law essentially forces companies to think about how their AI systems might behave in the real world, rather than just focusing on technical capabilities.

Key controls that prevent this

To avoid similar issues, organizations should implement several key controls:

• Output guardrails: Systems that automatically block or modify harmful content before it reaches users.
• Grounded responses: Ensuring chatbots can only provide information that exists in approved knowledge bases.
• Red-teaming: Regular testing where teams try to manipulate the chatbot into producing harmful content.
• Human oversight: Especially for sensitive topics or populations like children.
• Logging and monitoring: Tracking all interactions to detect and respond to problematic behavior.

What to do

1. Review all customer-facing AI systems for interactions with children or other vulnerable populations.
2. Conduct a red-team exercise to identify potential failure points in your chatbot systems.
3. Implement output guardrails that align with your organization's policies and applicable laws.
4. Establish clear human oversight procedures for AI systems that make commitments or provide sensitive information.
5. Set up monitoring systems to track key risk indicators like authority-breach rates and disputed statements.

Building a comprehensive AI governance program can help organizations manage these risks more effectively.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.