Why LLM jailbreaks are prompt engineering, not software bugs

When a large language model gets jailbroken, developers often look for a software patch. But as recent analyses show, jailbreaks and prompt injections are not code bugs. They happen at the prompt level. The model is not crashing or failing to compute. It is simply following instructions embedded in the input text that override its original safety guidelines. An attacker might hide text inside a retrieved document, an email, or a website. When the assistant reads that content, it treats the hidden text as a valid command, leading to unauthorized data leaks or unintended tool executions.

The risk management failure pattern

This behavior belongs to the AI security and prompt injection failure class. The pattern is well documented. In 2023, employees at Samsung pasted confidential source code and internal notes into a public large language model, exposing trade secrets because the model stored and processed that data outside enterprise boundaries. Similarly, early demonstrations with tools like Microsoft Copilot showed how hidden instructions in web pages could hijack an assistant into exfiltrating private files. The root cause is treating natural language inputs as trusted data rather than untrusted code.

Existing standards and controls

Managing this risk requires moving beyond traditional software security. Under ISO 42001 and data protection rules like the General Data Protection Regulation requirement for secure processing, organisations must implement strict acceptable-use policies and redaction layers for personally identifiable information and trade secrets. For systems with connected tools, the Open Worldwide Application Security Project top ten for large language models and the EU AI Act's rule on high-risk system accuracy and cybersecurity require robust input and output filtering alongside least-privilege tool access.

Monitoring what the model actually did

Prevention alone is never enough because clever prompt injections continue to evolve. To maintain visibility, security teams need specialized tooling to audit model behavior. Platforms like argus.threatclaw.ai record every trace an AI application produces and scan it for prompt injection, jailbreaks, and data leaks, including the attacks hidden inside retrieved documents and tool results rather than what the user typed. Governance decides what an AI agent is allowed to do, but monitoring shows what it actually did.

What to do

• Define a clear acceptable-use policy that restricts what employees can paste into public or internal language models.
• Implement automated filtering for sensitive information like secrets and personal data before input reaches any language model.
• Enforce least-privilege tool access so an assistant cannot execute dangerous commands without human confirmation.
• Track key risk indicators such as your injection block rate against a maintained attack corpus and the mean time to detect anomalous agent behavior.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.