CWE-1426 covers a product that invokes a generative AI or ML component whose behavior and output cannot be directly controlled, but does not validate those outputs, or validates them too little, against the intended security, content or privacy policy.
If your application passes what an LLM says into a browser, a database, a shell or another agent, this applies to you.
How it happens
A model's output is shaped by prompts, training and chance. You cannot guarantee what it will produce. Treat it as you would any user input and you are fine. Treat it as trusted because your own code asked for it, and you have a hole.
The record's example is CVE-2024-3402, a chain where a GUI for the ChatGPT API validated input but did not properly sanitize or validate the model's output, leading to cross site scripting (CWE-79).
// Vulnerable pattern: model output inserted as HTML.
chatBox.innerHTML = reply;
// Fixed: treat it as text.
chatBox.textContent = reply;
What an attacker gains
MITRE lists integrity and execution of unauthorized code or commands, varying by context. In an agent setting, output could cause unpredictable agent invocation, controlling or influencing agents the output triggers. The impact depends on what access is granted to the surrounding system.
How to find it
MITRE suggests dynamic analysis using known prompt injection and other attack techniques, adapted to the specific model or system, with results interpreted manually or automatically. Design review helps too, but works best alongside dynamic testing.
How to fix it
From the record:
- Run the model in an untrusted, non privileged space. Its output cannot be trusted, so give it as little access as possible.
- Use semantic comparators. These identify objects that look different but mean the same thing.
- Add an external moderator. Components outside the system, called supervisors or guardrails, monitor output.
- Train with good and bad examples to guide preferred outputs.
Alongside those, apply normal output handling: encode for the context where the text lands.
Quick check
Why is validating the prompt not enough?
The weakness is about the response. In CVE-2024-3402 the input was validated but the output was not, which led to XSS.
