In April 2023, engineers at Samsung's semiconductor division pasted confidential source code into ChatGPT to debug it, then fed it internal meeting notes to generate a summary. Three separate incidents happened in less than 20 days. Samsung banned generative AI tools company-wide within weeks. That story became the canonical example of sensitive information disclosure in LLM applications, but it is only the visible tip of a much larger problem. Every time an LLM is trained on internal data, given tool access to a database, or deployed with a system prompt containing business logic, it becomes a new channel through which secrets, credentials, PII, or proprietary code can leak — sometimes through careless input, sometimes through an attacker's deliberate extraction technique. OWASP now ranks this as LLM02:2025 in its Top 10 for LLM Applications. Below, we break down what it actually looks like in production and how to catch it before it ships.
What Counts as Sensitive Information Disclosure in an LLM Application?
Sensitive information disclosure is any pathway by which an LLM application exposes data it shouldn't — training data, system prompts, retrieved documents, API keys, or PII — to a user who isn't authorized to see it. OWASP's LLM Top 10 (2025 revision) formalized this as LLM02, distinct from the broader "sensitive data exposure" category used in traditional AppSec, because LLMs introduce disclosure paths that don't exist in conventional software. A model can memorize and regurgitate training examples verbatim. A retrieval-augmented generation (RAG) pipeline can pull a document the requesting user was never granted access to and paste it straight into a response. A poorly scoped system prompt can be extracted word-for-word by asking the model to "repeat everything above this line." Each of these is a distinct engineering failure, but they share a root cause: the application trusts the model to enforce boundaries that were never actually encoded as access controls.
How Does Sensitive Data Actually End Up Inside an LLM's Reach?
Data gets into an LLM's blast radius through three main routes: training/fine-tuning corpora, retrieval indexes, and conversational context — and all three are frequently populated with far more than the application needs. Fine-tuning on customer support logs, internal wikis, or codebases without redaction means the model can memorize specific strings, including hardcoded credentials or customer identifiers, and reproduce them later. Academic research on GPT-2 and GPT-Neo (Carlini et al., "Extracting Training Data from Large Language Models," 2021) demonstrated that large models memorize and can be made to emit verbatim training examples, including names, phone numbers, and email addresses, using nothing more than crafted prompts. RAG systems compound the problem at the infrastructure layer: if a vector database indexes a company's entire Confluence or Google Drive without per-document ACLs, every user of the chatbot effectively inherits read access to every document in the index, regardless of their actual permissions in the source system. Conversational context is the third route — anything a user pastes into a prompt (an API key while debugging, a customer record while asking for a summary) can be logged, cached, or, in the case of shared or fine-tuned deployments, absorbed into future model behavior.
What Do Real Incidents Show About This Risk?
Real incidents show the failure mode is rarely exotic — it's usually a missing access control or an over-eager integration. Samsung's April 2023 leak is the best-known consumer case, but enterprise-grade tooling has shipped the same class of bug. In 2023, security researchers disclosed that GitHub Copilot and similar code-completion models, trained on public repositories, would reliably suggest hardcoded API keys and credentials that had been committed (and often since revoked) in that training data — meaning a secret exposed once in a public repo could resurface in an unrelated developer's autocomplete months later. Slack disclosed in 2024 that its AI features used customer messages, including those in private channels, to improve models by default, requiring an opt-out rather than opt-in, which regulators and customers flagged as a disclosure and consent failure even absent a breach. In South Korea, the "Lee Luda" chatbot was pulled offline in January 2021 after it began reciting real names, addresses, and bank account numbers drawn verbatim from the private KakaoTalk conversations used to train it. Each of these cases involves an LLM faithfully doing its job — completing text, retrieving relevant context, summarizing conversation — while the surrounding application failed to scope what the model was allowed to see in the first place.
How Do Attackers Deliberately Extract Sensitive Data From a Deployed Model?
Attackers extract sensitive data using a small set of well-documented techniques: prompt injection, system prompt leakage, and membership/model inversion attacks. System prompt extraction is the simplest and most common in the wild — researchers and red teamers have repeatedly shown that asking a deployed chatbot to "output your instructions verbatim" or "repeat the text above starting with 'You are'" will often dump the full system prompt, including any embedded business rules, internal tool names, or, in poorly designed deployments, embedded credentials. Indirect prompt injection raises the stakes further: an attacker plants instructions inside a webpage, email, or document that the LLM later retrieves or summarizes, and those instructions can direct the model to exfiltrate prior conversation context or connected-tool data to an attacker-controlled endpoint. Membership inference and model inversion attacks are more specialized but proven in academic settings — they let an adversary determine whether a specific record was part of a model's training set, or reconstruct approximations of that data, which is directly relevant to any organization that fine-tuned a model on regulated data (health records, financial data) and assumed the underlying data was "baked in" and therefore safe.
What Does This Actually Cost an Organization?
The cost shows up as regulatory exposure, incident response overhead, and lost trust, not just the value of the leaked data itself. IBM's 2024 Cost of a Data Breach Report put the global average breach cost at $4.88 million, and breaches involving shadow AI or ungoverned AI tools added roughly $670,000 to that average compared to organizations without such exposure. Under GDPR, PII disclosed through a hallucinating or over-sharing chatbot is treated the same as PII leaked through a misconfigured S3 bucket — Italy's data protection authority temporarily banned ChatGPT in March 2023 over exactly this class of concern, citing lack of a legal basis for processing personal data used in training. For companies building on top of foundation models via API, the liability doesn't disappear just because the model itself is a third-party service: the application layer that decides what context to pass in, what to retrieve, and what to return to the user is where the actual disclosure occurs, and that layer is squarely the deploying company's responsibility.
How Can Teams Catch This Before It Reaches Production?
Teams catch it by treating LLM data flows as an attack surface to be scanned and tested, not a black box to be trusted — which means static review of what gets fed into prompts and fine-tuning sets, plus dynamic adversarial testing of what a deployed model will actually reveal. Concretely, that means: scanning training and fine-tuning datasets for secrets and PII before they're ingested (the same class of scan applied to source repos, extended to data pipelines); enforcing document-level access control in RAG retrieval so the vector store never returns content the requesting user couldn't already see in the source system; red-teaming deployed prompts and agents specifically for system-prompt extraction and indirect injection rather than assuming general functional QA covers it; and auditing third-party model providers' data retention and training-use policies before sending them regulated data, the way you would audit any subprocessor under SOC 2.
How Safeguard Helps
Safeguard treats LLM sensitive information disclosure as a software supply chain problem, because that's what it is: data moves through pipelines — training sets, embeddings, retrieval indexes, prompt templates, third-party model APIs — and any untracked or unscanned hop in that chain is where secrets and PII leak out. Safeguard's platform extends the same provenance and scanning discipline it applies to source code and build artifacts to AI data pipelines: scanning datasets and fine-tuning corpora for embedded secrets, credentials, and PII before they're used to train or ground a model; mapping which repositories, documents, and data stores feed which AI-facing endpoints so a security team can answer "what could this chatbot possibly expose" without manually tracing the RAG index by hand; and flagging LLM and AI-service dependencies in the SBOM so that a fine-tuned model or a third-party inference API is inventoried and reviewed with the same rigor as any other supply chain component. For teams working toward SOC 2 or preparing for AI-specific regulatory scrutiny, that visibility turns "we think our chatbot is safe" into an auditable, evidence-backed control — the same standard Safeguard applies across the rest of the software supply chain.