In March 2023, security researcher Johann Rehberger showed that Bing Chat could be manipulated into embedding a hidden markdown image tag in its response — one whose URL silently sent a user's private conversation history to an attacker-controlled server the moment the browser rendered the reply. No malware, no phishing link clicked, no traditional exploit. The browser simply did what it always does with an image tag, and the chat interface simply did what it always does with the model's text: displayed it as trusted content. That gap — between what an LLM outputs and what the receiving system assumes about that output — is insecure output handling, and it now sits at position LLM05 on the OWASP Top 10 for LLM Applications (2025 edition). As LLMs get wired into browsers, databases, shells, and internal APIs, the same gap is showing up as stored XSS, SQL injection, SSRF, and remote code execution across production AI products.
What Is Insecure Output Handling in LLM Applications?
Insecure output handling is the failure to validate, sanitize, or encode an LLM's generated text before passing it to a downstream component — a browser DOM, a SQL client, a shell, an API call, or another agent — that will interpret it as code or a trusted instruction rather than as plain data. OWASP first named this class of flaw "Insecure Output Handling" (LLM02) in its August 2023 Top 10 for LLM Applications v1.0, then renamed and reframed it as "Improper Output Handling" (LLM05) in the 2025 revision to emphasize that the problem isn't unique to any one sink — it's a pattern that recurs everywhere an application treats model output as inherently safe. The core issue is that LLM output is attacker-influenceable: a prompt injection hidden in a document, email, or web page an agent reads can steer what the model writes, and if nothing downstream re-validates that text, the injection rides straight through to execution. It is functionally the same bug class as failing to encode user input before rendering it in HTML — except the "user input" is now anything the model has read, not just what a human typed into a form.
How Does Insecure Output Handling Lead to Remote Code Execution?
It leads to RCE when an application takes the LLM's generated code and executes it directly instead of running it in a sandbox. LangChain's LLMMathChain, which asked a model to translate a math question into a Python expression and then evaluated that expression with Python's exec(), shipped this exact flaw: CVE-2023-29374, disclosed in April 2023, let an attacker craft a prompt that caused the LLM to emit arbitrary Python instead of a math expression, which the chain then executed on the host with no isolation. Two months later, a near-identical pattern surfaced in LangChain's PALChain component as CVE-2023-36258, disclosed in June 2023 — again, LLM-generated Python treated as trusted code and handed to an interpreter. Both were fixed by adding sandboxing, but the underlying lesson generalized poorly across the ecosystem: any "code interpreter," "math solver," or "agent executor" pattern that runs LLM output through exec(), eval(), or a shell call is one crafted prompt away from arbitrary code execution, and 2023–2024 saw the same class of finding reported against multiple other LangChain-adjacent and custom agent frameworks.
How Does Insecure Output Handling Enable Cross-Site Scripting and Data Exfiltration?
It enables XSS and exfiltration when a front end renders LLM output as HTML or markdown without sanitization, letting the model's text execute as active content in the victim's browser. Rehberger's February 2023 disclosure against Bing Chat is the canonical case: by getting the model to include a markdown image reference pointing to an attacker's server with the conversation encoded in the URL's query string, the chat UI's own markdown renderer auto-loaded the "image" and leaked the data — no click required. The same pattern shows up in custom-built chatbot widgets: a React or Vue front end that pipes a model's response into dangerouslySetInnerHTML or v-html for "nicer formatting" will happily execute a <img src=x onerror=fetch(...)> payload that a prompt injection convinced the model to output, turning a support chatbot into a stored XSS vector against every user who reads that response. This is precisely why OWASP's guidance treats LLM output the same as any other untrusted input reaching a browser — HTML-encode by default, and treat markdown-to-HTML rendering of model text as a feature that needs an allowlist, not a formatting nicety.
How Does Insecure Output Handling Cause SQL Injection and SSRF in AI Pipelines?
It causes SQL injection and SSRF when text-to-SQL or tool-calling agents concatenate model output directly into a query or a request URL instead of treating it as a parameter. Text-to-SQL is one of the most common enterprise LLM patterns — a user asks a natural-language question, the model generates SQL, and the application runs it against a production database — and if that generated SQL is executed as a raw string rather than bound through parameterized queries, a prompt injected via a support ticket, a document, or a manipulated field can steer the model into emitting UNION-based exfiltration or destructive statements the application then runs with its own database credentials. NVIDIA's AI red team documented the SSRF variant in its 2023–2024 LLM application assessments: agents that let a model choose a URL to fetch — for "look this up" or "summarize this page" tool calls — without restricting the target to an allowlist can be steered into requesting internal metadata endpoints or private network addresses, exactly the way classic web SSRF works, except the attacker never touches the server directly; they only need to influence what the model decides to fetch.
Why Do Traditional Output-Encoding Defenses Fall Short Against LLM Output?
Traditional output encoding falls short because it assumes a fixed, developer-authored template with a small number of well-understood insertion points, while LLM output is dynamically generated, unbounded in structure, and directly steerable by anyone who can influence the model's input — including third parties whose content the model merely reads. A developer can HTML-encode a username field with confidence because a username is short and its shape is known in advance; an LLM's multi-paragraph response, by contrast, might legitimately need to contain code blocks, links, or formatted data, so a blanket "strip everything" policy breaks the product while a permissive policy reopens the hole. Worse, because prompt injection lets an attacker who never directly interacts with the vulnerable system still control what the model outputs — via a poisoned document, email, or web page an agent reads — the trust boundary isn't at the user's keyboard anymore, it's wherever any content the model ingests originates. That's a much larger and less auditable attack surface than the one classic output-encoding rules were built for, which is why OWASP's 2025 guidance for LLM05 recommends context-aware output handling per destination (HTML encoding for browser rendering, parameterized queries for SQL, sandboxed interpreters for code, allowlisted destinations for network calls) rather than a single sanitization pass applied once at the model's output boundary.
How Safeguard Helps
Safeguard treats LLM-integrated applications the same way it treats any other software supply chain: every sink an LLM's output can reach — a database client, a shell, a browser render path, an outbound HTTP call — is a dependency edge that needs provenance and policy enforcement, not blind trust. Safeguard's static analysis flags the specific code patterns behind incidents like CVE-2023-29374 and CVE-2023-36258 — LLM output passed to exec(), eval(), raw SQL execution, or unsanitized HTML rendering — so a text-to-SQL agent or a code-interpreter integration gets caught in CI before it reaches production, not after a prompt injection finds it first.
For teams building on LangChain, LlamaIndex, or custom agent frameworks, Safeguard's SBOM and vulnerability tracking cover the AI stack itself, so when a CVE lands against a chain or tool-calling library your pipeline depends on, you know within minutes which services are affected rather than discovering it from an incident report. Safeguard's policy engine lets teams codify the OWASP LLM05 controls as enforceable gates — requiring parameterized queries on any LLM-generated SQL path, sandboxing on any code-execution path, and an explicit allowlist on any agent that can trigger outbound requests — so insecure output handling gets stopped at the architecture level instead of depending on every engineer remembering to sanitize every model response by hand. In a threat model where the attacker doesn't need direct access to your system, only the ability to influence what your model reads, that structural enforcement is what keeps a clever prompt from becoming a production breach.