On March 20, 2023, a bug in an open-source Redis library caused ChatGPT to expose the first message of some users' new conversations to strangers, along with payment details — names, email addresses, and the last four digits of credit card numbers — for roughly 1.2% of ChatGPT Plus subscribers active during a nine-hour window. A month later, Samsung banned generative AI tools internally after employees pasted confidential semiconductor source code and internal meeting notes into ChatGPT three separate times in 20 days. These are not edge cases. Sensitive information disclosure — where an LLM reveals training data, system prompts, proprietary code, credentials, or user PII it was never meant to expose — is now formally tracked as LLM02:2025 in the OWASP Top 10 for LLM Applications. As enterprises wire LLMs into support bots, coding assistants, and internal knowledge tools, the blast radius of a single leaky prompt or misconfigured RAG pipeline keeps growing.
What Is Sensitive Information Disclosure in LLMs?
Sensitive information disclosure in LLMs is the unintended exposure of confidential data — training data, system prompts, API keys, PII, or proprietary business logic — through a model's outputs. OWASP formalized this as LLM02:2025 in its Top 10 for LLM Applications, distinguishing it from traditional data breaches because the leakage path runs through natural-language generation rather than a database dump or file export. The disclosure can happen three ways: the model memorized and regurgitates training data verbatim, a crafted prompt tricks the model into revealing its system instructions or a prior user's context, or an application feeds sensitive documents into a retrieval-augmented generation (RAG) pipeline without access controls, letting any authenticated user query data they shouldn't see. Unlike a SQL injection that returns rows from a table, an LLM leak returns fluent prose that can look like a normal, helpful answer — which is exactly why it's so easy to miss in code review or QA.
Why Do LLMs Leak Sensitive Data In The First Place?
LLMs leak sensitive data because they memorize portions of their training corpus and because application layers routinely pass secrets into the context window. In "Extracting Training Data from Large Language Models" (2021), Carlini et al. demonstrated that GPT-2 could be prompted to regurgitate more than 600 verbatim training examples, including individuals' names, phone numbers, and email addresses, simply by generating text and filtering for memorized sequences. Beyond memorization, three application-layer patterns account for most real-world leaks: system prompts stored as plain strings and extractable via prompt injection (Microsoft's Bing Chat "Sydney" system prompt was leaked by a Stanford student within days of its February 2023 launch), RAG pipelines that index internal documents without per-user permission filtering, and developers hardcoding API keys or database credentials directly into prompts or fine-tuning datasets that later get logged, cached, or used to train a downstream model.
What Are Real-World Examples Of LLM Sensitive Information Disclosure?
Real-world LLM disclosure incidents span consumer chatbots, enterprise SaaS, and internal coding assistants. Samsung's April 2023 incidents involved an engineer pasting proprietary source code into ChatGPT to check for bugs, another employee sharing code to optimize a defective chip-testing program, and a third converting a confidential meeting recording into notes via the tool — all recoverable by OpenAI as training signal unless a user opted out. In August 2024, security firm PromptArmor disclosed a prompt injection flaw in Slack AI that let an attacker post a message in a public channel instructing the AI assistant to exfiltrate data from private channels the attacker couldn't otherwise access, surfacing it back through a rendered link. Italy's data protection authority, the Garante, temporarily banned ChatGPT in March 2023 over concerns that it processed personal data unlawfully and lacked an age-verification mechanism, lifting the ban only after OpenAI added disclosures and controls in late April 2023. Each case traces back to the same root cause: sensitive content entered a context window with no downstream control over where the model's output could travel.
How Is This Different From Traditional Data Leakage Risks?
LLM sensitive information disclosure differs from traditional data leakage because the attack surface is the prompt and the model's generative behavior, not a fixed API endpoint or file share. A conventional data loss prevention (DLP) tool scans outbound traffic for known patterns — a credit card regex, a labeled file — but an LLM can paraphrase, summarize, or reconstruct sensitive content in ways that evade pattern matching entirely, since the "leak" is synthesized text rather than an exact copy. Traditional access control also assumes a static permission boundary at query time; RAG architectures instead flatten many documents into a shared embedding space, so a single unscoped retrieval call can pull content across tenant or user boundaries that the underlying vector database was never configured to isolate. This is why OWASP treats prompt injection (LLM01) and sensitive information disclosure (LLM02) as adjacent, compounding risks: an injected prompt is frequently the delivery mechanism, and disclosure is the payload.
How Can Security Teams Detect And Prevent LLM Sensitive Information Disclosure?
Security teams prevent LLM sensitive information disclosure by controlling what enters the context window and validating what leaves it, rather than trusting the model to self-censor. Concretely, that means: enforcing document-level and row-level access control before content reaches a RAG retriever (not after), stripping secrets and PII from fine-tuning and few-shot datasets before training, treating system prompts as security-sensitive configuration rather than static strings, and instrumenting output filtering for known-sensitive patterns (keys, tokens, internal hostnames) at the API boundary. Vendors like Snyk, Wiz, and Aqua have extended existing SAST/SCA tooling to flag hardcoded secrets feeding into AI pipelines, but the harder problem is tracing whether a specific vulnerable dependency, exposed credential, or overly-permissive RAG index is actually reachable by an LLM-facing feature in production — most teams still answer that question manually, if at all, which is where automated reachability analysis becomes necessary rather than optional.
How Safeguard Helps
Safeguard's reachability analysis traces whether secrets, sensitive data paths, and vulnerable dependencies identified across your SBOM are actually exposed to LLM-facing code paths, so security teams can prioritize the handful of findings tied to real exploitability instead of triaging every match a scanner surfaces. Griffin AI continuously analyzes your codebase and dependency graph to flag risky patterns specific to LLM integrations — unscoped RAG retrieval calls, hardcoded API keys entering prompt context, system prompts stored without access controls — and explains the exploit path in plain language. Safeguard's SBOM generation and ingest give teams a live inventory of every model, library, and data connector feeding an AI feature, closing the visibility gap that lets shadow AI integrations go unreviewed. When a fix is identified, Safeguard opens an auto-fix pull request with the remediation already scoped, cutting the time between detection and a merged fix from weeks to hours.