To test an LLM app for improper output handling, list every place model output goes next (browser, database, shell, file system, email, another service), push harmless marker strings through the model into each one, and confirm the receiving code encodes or validates them. Treat the model like any other untrusted user.
What is improper output handling?
OWASP defines LLM05 as insufficient validation, sanitization and handling of LLM outputs before they reach other components. Because a prompt can steer what the model writes, whoever controls the prompt gets indirect access to whatever consumes the output.
The OWASP page lists the usual consequences: cross-site scripting, CSRF, SSRF, privilege escalation, or remote code execution on a backend. It also names things that raise the impact, such as an LLM with excessive privileges, indirect prompt injection, and missing context-specific output encoding.
Note the difference from prompt injection. Injection is about what goes in. LLM05 is about what your own code does with what comes out.
How do you find the sinks?
Start with a data flow sketch, not a scanner. For each feature that calls a model, write down where the text goes. OWASP's examples give you a ready list of categories to look for:
- Output passed to a system shell or to functions such as exec or eval.
- Generated JavaScript or Markdown rendered in a browser.
- Generated SQL run without parameters.
- Output used to build file paths.
- Generated text dropped into email templates without escaping.
Add the ones the page does not spell out but your app may have: values written to logs that another tool parses, fields sent to a webhook, and arguments handed to an agent tool.
How do you test each sink without an attack payload?
Use marker strings that are inert but would be visible if mishandled. You are testing whether the layer after the model changes or escapes the text, not trying to cause harm.
- For rendering sinks, instruct the model (in a test environment) to include a string with HTML special characters and Markdown link syntax. Then look at the page source. The characters should appear encoded, and links should not become clickable unless you intended that.
- For query sinks, have the model produce a value containing a quote character and a SQL comment marker. Check the query log. The value should arrive as a bound parameter, never as part of the statement text.
- For path sinks, have the model produce a file name with parent directory markers. The code should reject it or resolve it inside a fixed base directory.
- For command or code sinks, the test is mostly a code review: find any call that takes model text and executes it. If one exists, that is the finding, no probing needed.
- For email and message sinks, include a fake link with mismatched display text and check how your template treats it.
Run these through the real model, not by mocking the output. Models vary, and a unit test that bypasses the model proves nothing about the pipeline.
What should pass and what should fail?
A pass means the text arrives at the next system unchanged in meaning but safe in form: encoded for HTML, bound as a parameter, confined to an allowed path, or rejected with a logged reason.
A fail is any case where model text changes the structure of what the next component does. Record the sink, the marker you used and the observed result. Fixes belong in the receiving code, not in the prompt. A system prompt that says "never output HTML" is not an encoding control.
What to do
- Draw the data flow for each model-backed feature and mark every sink.
- Apply the OWASP guidance: zero-trust treatment of model output, validation per ASVS, and context-aware encoding for the target (HTML for web pages, parameterization for databases).
- Set a strict Content Security Policy on any page that shows model output.
- Remove exec and eval style sinks. Replace them with a fixed set of allowed actions.
- Add the marker tests above to CI so a new sink cannot ship unchecked.
- Log unusual output patterns and review them, as OWASP recommends for detection.
- Reduce what the model's tools can do, so a bad output has a small blast radius.
Quick check
Is LLM05 the same as prompt injection?
No. Prompt injection manipulates the model's input. Improper output handling is the failure of your own code to treat the output as untrusted. The two combine, but you fix LLM05 in the receiving components.
Can a system prompt prevent improper output handling?
Not reliably. A prompt instruction can lower the odds of bad output but cannot guarantee it. Encoding, parameterization and validation in the consuming code are the controls that hold.
Do I need real attack strings to test this?
No. Inert marker strings with special characters show whether the next layer escapes or binds correctly. If you need a working exploit to prove a sink is unsafe, the sink was already unsafe by design.
Sources
Last reviewed: 18 October 2026.
