In February 2023, a Stanford student named Kevin Liu got Microsoft's newly launched Bing Chat to reveal its confidential internal codename, "Sydney," along with its entire system prompt, just by asking the right series of questions. No malware, no exploit code, no buffer overflow — just carefully chosen words. That moment introduced millions of people to the prompt injection attack, a technique that manipulates a large language model into ignoring its original instructions and following an attacker's instead. More than two years later, prompt injection attacks are still ranked as the single most critical risk in the OWASP Top 10 for LLM Applications, a position they have held since the list's first release. Unlike a typical software bug, this attack doesn't exploit broken code — it exploits the fact that language models cannot reliably tell the difference between an instruction and a piece of data. That distinction is exactly why, despite enormous investment from every major AI lab, nobody has fully closed the hole.
What Is a Prompt Injection Attack?
A prompt injection attack is any technique that inserts unauthorized instructions into the text an LLM processes, causing the model to act on the attacker's commands instead of (or in addition to) the developer's original instructions. The term itself predates ChatGPT: researcher Simon Willison coined "prompt injection" in a blog post on September 12, 2022, describing how a GPT-3-powered Twitter bot could be hijacked by simply tweeting a new instruction at it. The core mechanism hasn't changed since. An LLM application typically works by concatenating a system prompt ("You are a helpful support assistant, only answer questions about our product") with user input or retrieved content, then feeding the whole thing to the model as one continuous block of text. If an attacker can control any part of that block — a chat message, an uploaded résumé, a webpage the model reads to answer a question — they can plant instructions that look, to the model, exactly as authoritative as the developer's own.
How Is Direct Prompt Injection Different From Indirect Injection?
Direct prompt injection means the attacker types the malicious instruction straight into the chat box themselves; indirect injection means the attacker plants it somewhere the model will later read on its own. Kevin Liu's Bing Sydney disclosure was direct — he typed the manipulative prompt himself. A more dangerous pattern emerged over the following two years as LLMs gained the ability to browse the web, read documents, and call external tools. In August 2024, security firm PromptArmor disclosed a case of indirect prompt injection in Slack AI: an attacker could post a message in a public Slack channel containing hidden instructions, and when a completely different user later asked Slack AI an unrelated question, the assistant would read the poisoned message as part of its context and exfiltrate private data the attacker never had access to. Security researcher Johann Rehberger documented similar indirect injection chains throughout 2023 and 2024 against ChatGPT plugins, GitHub Copilot Chat, and Google's Gemini and Bard — in each case, the payload arrived not from the user's keyboard but from a document, email, or web page the model was asked to summarize.
Why Can't Models Just Be Trained to Ignore Malicious Instructions?
Models can be trained to resist obvious attacks, but training reduces the success rate rather than eliminating it, because there is no architectural wall separating "instructions" from "data" inside a transformer. Every token — system prompt, user question, retrieved document, tool output — flows into the same context window and gets attended to using the same mechanism. This is the crux of LLM input manipulation: the model has no built-in notion of trust level for the text it's reading, only patterns learned from training data about what a follow-up instruction tends to look like. OpenAI's 2024 "instruction hierarchy" research and similar work from Anthropic and Google DeepMind have shown real improvements — teaching models to weight system-level instructions more heavily than content encountered later — but every published evaluation, including OpenAI's own, still reports a nonzero attack success rate against adversarial inputs crafted specifically to evade the hierarchy. Compare it to spam filtering: after 25 years of machine learning progress, spam still reaches inboxes, because the attacker gets to see the defense and iterate against it. Prompt injection defense faces the same asymmetry, compressed into a much younger field.
What Real-World Incidents Show the Scale of the Problem?
Real incidents span everything from embarrassing to financially damaging, and they've been documented consistently since Bing Chat's launch. In December 2023, a Chevrolet dealership's customer-service chatbot was manipulated through injected conversational instructions into agreeing to sell a 2024 Tahoe for one dollar "no take backs," a case that went viral precisely because it showed how easily a production deployment could be steered off-script. Rehberger's GitHub Copilot Chat research showed that a malicious instruction hidden inside a pull request or source file comment could cause the assistant to leak private repository data or inject misleading suggestions back to a developer who never saw the payload directly. Academic red-teaming has quantified the exposure too: multiple 2023-2024 studies benchmarking retrieval-augmented generation systems found attack success rates above 80% for unprotected pipelines when the injected instruction was placed inside retrieved documents rather than the user's own message. The common thread across every incident is that the vulnerable systems were doing exactly what they were built to do — read and act on external content — which is also precisely what made them exploitable.
Why Is Prompt Injection Defense So Hard to Get Right?
Prompt injection defense is hard because effective mitigations require trade-offs between security, usability, and cost that most teams aren't equipped to make correctly on their first attempt. Input sanitization and instruction-hierarchy prompting catch known attack patterns but miss novel phrasing; output filtering catches some data exfiltration but breaks legitimate use cases that resemble it; sandboxing tool access limits blast radius but also limits what the assistant can usefully do. Layering all three still leaves gaps, because — as OWASP's LLM01:2025 entry states plainly — there is currently no fix that eliminates prompt injection risk entirely, only measures that reduce the attack surface and the impact of a successful attack. The problem compounds in real deployments because injection payloads rarely arrive through a single, auditable channel. They ride in through a chained agent's tool outputs, a third-party plugin's API response, a vector database populated by a scraper, or an MCP server maintained by someone outside the security team entirely — each one an untracked dependency in what is, functionally, a software supply chain for AI behavior.
How Safeguard Helps
That supply-chain framing is exactly where Safeguard focuses. Prompt injection attacks don't just live in a single prompt template — they travel through the full chain of tools, plugins, MCP servers, retrieval pipelines, and third-party integrations that modern AI agents depend on, and most organizations have no inventory of that chain at all. Safeguard extends the software supply chain security practices teams already trust for their code — provenance tracking, dependency visibility, and continuous monitoring — to the AI systems built on top of it. That means mapping every tool and data source an LLM agent can reach, flagging integrations that accept untrusted external content without isolation, and continuously monitoring for the kind of anomalous instruction-following behavior that indicates an injection attempt succeeded. Rather than promising to "solve" prompt injection with a single filter, Safeguard treats it the way mature security programs treat any risk that can't be fully eliminated: reduce the attack surface, instrument every entry point, and make sure that when an attack does get through, it's detected before it turns into a data leak, a hijacked transaction, or a headline. If your team is deploying LLM agents against real data and real tools, understanding your AI supply chain is no longer optional — it's the difference between a contained incident and an open door.