California’s AI-Adapted Cybersecurity Plan Requires Immediate Input/Output Validation Overhaul

The California Cybersecurity Plan (CCP) Update: An AI-First Shift

The California Governor’s Office of Emergency Services (CalOES) has announced the next phase of the California Cybersecurity Plan (CCP), explicitly acknowledging the evolving threat landscape driven by Artificial Intelligence. This update moves beyond traditional IT security to address the integration of Generative AI into state infrastructure and operations. While this is a policy and strategic announcement rather than an incident, it signals a critical shift for practitioners: the state’s security posture now explicitly includes the governance of Large Language Models (LLMs) and agentic AI systems. For AI governance and security professionals, this mandate requires an immediate reassessment of existing cyber controls to address the specific failure patterns of AI, particularly prompt injection and data exfiltration.

The Mechanism: Prompt Injection and the Hijacking of the "Action Path"

The primary risk highlighted by this shift is the rise of prompt injection attacks, a class of vulnerability where an attacker inserts instructions into the context window of an LLM that override the intended system prompt. Unlike traditional code injection, which targets the execution environment, prompt injection targets the semantic understanding layer of the model.

The mechanism operates by exploiting the model's tendency to prioritize the most recent or seemingly authoritative instruction. Attackers often use "prefix injection" (adding malicious text to the beginning of user input) or "postfix injection" (adding text after the user's request, often with line breaks to bypass instruction following). A concrete precedent of this occurred in the Bing/Copilot demos of 2023, where hidden instructions embedded in web pages and emails hijacked the assistants into leaking sensitive data or performing unauthorized actions, bypassing the developers' guardrails.

A more consequential real-world failure involved Samsung employees in 2023 who pasted proprietary source code and internal meeting notes into public Generative AI interfaces. This occurred because the "action path"—the sequence of steps the model takes to fulfill a request—was not constrained. The model, thinking it was merely processing text, inadvertently transmitted sensitive intellectual property. This failure mode highlights that the vulnerability lies not just in the model's generation, but in the unrestricted access the model has to tools, APIs, and data sources without secondary verification.

Why This Matters Now: Aligning State Security with EU AI Act and ISO Standards

The urgency of the CCP update is underscored by the impending regulatory reality. As the EU AI Act moves toward enforcement, public sector and state-level systems utilizing Generative AI are increasingly classified as "high-risk" under Annex III criteria (e.g., management of public services, law enforcement). The Act mandates strict transparency obligations (Article 50) and technical requirements for high-risk systems, including the mitigation of systemic risks.

Furthermore, organizations operating under ISO/IEC 42001 (the first international standard for AI Management Systems) are required to treat AI security as a core business continuity risk. The CCP update effectively forces state actors to demonstrate compliance with the principle of "Security of Processing" (GDPR Article 32) by addressing the unique vectors introduced by LLMs. The failure to adapt the CCP to include AI-specific controls—such as input/output filtering and least-privilege tool access—would constitute a governance gap that could lead to non-compliance with both ISO 42001 and the EU AI Act once applicable deadlines are reached.

Where Teams Get This Wrong: Specific Pitfalls in AI Security Implementation

Despite the clarity of the threat landscape, many teams implementing AI governance for the California state sector (and similar environments) make three specific technical and procedural errors:

1. Relying on "Jailbreak Detection" as a Standalone Control: Teams often assume that a detector model can identify malicious intent before the user query is processed. This is flawed because jailbreaks are increasingly sophisticated (e.g., using Chain-of-Thought reasoning to bypass filters) and detectors often have high false-negative rates. Without a hard boundary on the "action path," a successful jailbreak leads to immediate compromise.

2. Neglecting the "Hidden Context" Attack Vector: In the context of web-based assistants or RAG (Retrieval-Augmented Generation) systems, teams fail to sanitize the metadata or hidden instructions in documents retrieved by the system. If a malicious PDF is retrieved and processed, the LLM might follow a hidden command embedded within the document structure, similar to the Bing attack, without the user ever typing a malicious prompt themselves.

3. Granting Full API Access by Default: A common architectural mistake is granting the LLM write access to production databases or file systems. In the Samsung incident, the failure was allowing the model to process sensitive text in a public environment. In a state infrastructure context, allowing an LLM to execute DELETE commands or write sensitive logs without a human-in-the-loop or a strict API gateway results in a broken access control scenario.

Governance Guidance: From Policy to Technical Enforcement

The CCP update implies that governance for state AI systems must move from abstract policy to technical enforcement. Governance teams must:

• Implement Input/Output Filtering Layers: All interactions with LLMs must pass through a strict guardrail layer. This layer should perform regex-based filtering for known malicious command structures (e.g., SQL injection patterns, system commands) and semantic analysis for jailbreak attempts.
• Enforce the Principle of Least Privilege for Tools: The "action path" must be constrained. LLMs should not have direct write access to production databases. Instead, they should only be allowed to read from specific, non-sensitive data stores, or write to a staging area that requires manual review and approval before production deployment.
• Establish a State AI Security Board: Analogous to the CalCIC, organizations must create a cross-functional board (including ML engineers, security architects, and legal counsel) that reviews the security posture of every new AI agent deployed within the state ecosystem.
• Map AI Risks to ISO 42001 Clauses: Governance documentation must explicitly map identified AI risks (like prompt injection) to specific clauses in ISO 42001, such as Clause 6.1 (Actions to address risks and opportunities) and Clause 8.3 (Control selection), ensuring that technical controls are formally approved and documented.

Risk Management Guidance: Measuring the Unmeasurable

To operationalize the CCP's new requirements, risk management teams must define and track specific Key Risk Indicators (KRIs) that are unique to AI security:

• Injection Block Rate Against a Maintained Attack Corpus: This metric measures the effectiveness of the guardrail layer. It is calculated as (Number of malicious prompts blocked / Total number of malicious prompts in the corpus). A healthy rate is typically above 95%, but this must be tested against a constantly evolving corpus of jailbreaks.
• Sensitive Tool Calls Without Secondary Authorization (Target: 0): This metric tracks the number of times the AI agent attempts to call sensitive APIs (e.g., database writes, file system access) without a secondary human approval step or a whitelist check. This directly addresses the "action path" hijacking failure mode.
• Mean Time to Detect (MTTD) Anomalous Agent Behavior: This measures the speed at which the security monitoring systems identify deviations from the agent's standard behavior pattern, such as suddenly attempting to access unauthorized file types or communicating with external domains not in the allowlist.

The Takeaway

• Adapt Your Cyber Controls: The California CCP update confirms that Generative AI is now a primary vector for cyber threats. Your security architecture must treat LLMs as high-risk assets requiring input/output validation and strict API gatekeeping.
• Sanitize the Context: Guardrails must extend beyond the user prompt to include the retrieval context (RAG), ensuring that hidden instructions in documents do not hijack the model.
• Restrict the Action Path: Never allow an LLM direct write access to production systems. The "action path" must be segmented and require secondary verification for any sensitive operation.
• Define AI-Specific KRIs: Move beyond traditional cyber metrics and track Injection Block Rates and unauthorized tool call frequencies to quantify AI risk effectively.
• Align with EU AI Act Standards: As the EU AI Act enforcement dates approach, the CCP's guidance serves as a necessary precursor for ensuring state systems meet the transparency and safety obligations of Annex III high-risk categories.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.