LLM security is the set of practices, tools, and controls that protect large language model applications — the model weights themselves, the training and fine-tuning data, the prompts users send, the outputs the model generates, and the plugins, agents, and vector databases wired around them — from manipulation, data leakage, and supply chain compromise. It sits at the intersection of classic application security and a genuinely new attack surface: a model that ingests untrusted natural-language input and can be talked into ignoring its own instructions. OWASP published the first Top 10 for LLM Applications in August 2023 and revised it for 2025 because real incidents kept piling up — Samsung banned generative AI tools company-wide in May 2023 after engineers pasted proprietary source code into ChatGPT, and by February 2024 JFrog researchers had found roughly 100 malicious models sitting on Hugging Face Hub, several configured to execute code the moment they loaded. LLM security covers all of it, from the model file on disk to the prompt in transit.
What Is LLM Security?
LLM security is the discipline of protecting every layer of a large-language-model application stack — model weights, training and fine-tuning pipelines, prompt handling, output handling, agent/plugin permissions, and the orchestration frameworks and vector stores (LangChain, LlamaIndex, Pinecone, Weaviate) that connect them — against manipulation, exfiltration, and supply chain compromise. It differs from "AI safety" or "alignment," which focus on a model behaving as intended in the abstract; LLM security assumes an adversary is actively trying to make the deployed system misbehave, leak data, or execute unauthorized actions. The field has three practical layers: the model layer (weights, fine-tuning data, provenance), the application layer (prompts, outputs, agent permissions, retrieval-augmented generation pipelines), and the infrastructure/supply chain layer (the Python packages, containers, and model files an organization pulls in to run any of it). OWASP's Top 10 for LLM Applications — first released in 2023, updated for 2025 — is the closest thing to an industry-standard taxonomy, covering risks from prompt injection (LLM01) and sensitive information disclosure (LLM02) through supply chain (LLM03), data and model poisoning (LLM04), and excessive agency (LLM06).
What Are the Top LLM Security Risks in Production?
The risk that shows up most often in live deployments is prompt injection, where crafted input causes a model to override its system instructions, followed closely by sensitive information disclosure and supply chain compromise. OWASP's 2025 list ranks prompt injection as LLM01 specifically because it's the one vulnerability class unique to generative AI with no direct legacy analogue. Sensitive information disclosure (LLM02) covers a model regurgitating training data, API keys embedded in a system prompt, or a customer's PII pulled from a retrieval index and shown to the wrong user. Supply chain risk (LLM03) covers everything from a compromised base model on Hugging Face to a vulnerable version of langchain or transformers sitting in requirements.txt. Excessive agency (LLM06) is the risk that grew fastest through 2024 as teams shipped agentic systems: an LLM-driven agent with a file-write tool, a code-execution tool, and a database-write tool has three separate blast radii if any one of its inputs is compromised. Unbounded consumption (LLM10) — an attacker triggering runaway API cost or denial of service through recursive agent loops or oversized context requests — rounds out the list as a cost and availability concern, not just a confidentiality one.
How Is Prompt Injection Different From SQL Injection?
Prompt injection differs from SQL injection because there's no syntactic boundary between "instruction" and "data" for a language model to parse — SQL injection exploits unescaped input concatenated into a query string that a database parser can still be taught to sanitize, while prompt injection exploits the fact that a transformer model reads system instructions and untrusted user or document content as the same token stream. You can parameterize a SQL query with near-total confidence it stops injection; you cannot yet fully separate "trusted instruction" from "untrusted content" inside a context window. That gap produces two attack patterns: direct prompt injection, where a user types the attack straight into a chat box, and indirect prompt injection, where the attack is hidden in a document, webpage, or email the model is asked to summarize or act on. Security firm PromptArmor disclosed in August 2024 that Slack's AI assistant could be manipulated through messages posted in a public channel to exfiltrate secrets shared in private channels the attacker had no access to, entirely through indirect prompt injection. Microsoft's Bing Chat ("Sydney") was similarly jailbroken within days of its February 2023 launch through prompts hidden in webpages it was asked to browse.
What Vulnerabilities Live in the LLM Software Supply Chain?
The LLM supply chain carries vulnerabilities at every layer an organization doesn't build itself: orchestration frameworks, model files, and package registries. CVE-2023-29374, disclosed in April 2023, is a concrete example — LangChain's LLMMathChain passed LLM-generated text into Python's exec() to evaluate math expressions, so an attacker who could influence the model's output through prompt injection could achieve arbitrary code execution on the host running the chain. Model files carry their own risk: PyTorch .bin and .pt checkpoints are frequently just pickled Python objects, and pickle deserialization can execute arbitrary code on load — which is exactly what JFrog's security research team found in February 2024, when they identified roughly 100 malicious models on Hugging Face Hub, some designed to open a reverse shell to the uploader's infrastructure the moment a data scientist loaded them locally. Package registries add a third vector: typosquatted PyPI and npm packages mimicking popular AI libraries like openai and langchain have shown up repeatedly since 2023, targeting developers who mistype an install command. None of these require a flaw in the model's reasoning at all — they're conventional supply chain attacks that happen to ride in on the AI boom's dependency graph.
What Real-World Incidents Show the Stakes?
The clearest real-world incidents span data leakage, brand damage, and legal liability, not just theoretical exploits. Samsung employees leaked confidential source code and meeting notes into ChatGPT on three separate occasions in early 2023, which Bloomberg reported and which led Samsung to ban generative AI tools company-wide in May 2023. In December 2023, a customer got a Chevrolet dealership's ChatGPT-powered chatbot to agree, in writing, to sell a 2024 Chevy Tahoe for one dollar — a prompt-injection-adjacent stunt that went viral and forced the dealership to shut the bot down. In February 2024, Canada's Civil Resolution Tribunal ruled that Air Canada was legally liable for bereavement-fare advice its website chatbot hallucinated, rejecting the airline's argument that the bot was "a separate legal entity" — a ruling that reframed improper output handling as a contract and consumer-protection liability, not just a security bug. And the August 2024 Slack AI disclosure showed that even a mature SaaS vendor's first-party AI feature could be turned into a private-channel data exfiltration path using nothing more than a crafted message and a rendered link.
How Do Security Teams Operationalize LLM Security?
Security teams operationalize LLM security by mapping controls to a named framework and then enforcing them at build time, not just after deployment. NIST released its AI Risk Management Framework in January 2023 as the first widely adopted cross-industry playbook for identifying and governing AI risk, and MITRE's ATLAS knowledge base catalogs adversary tactics against AI systems the same way MITRE ATT&CK does for traditional networks. In practice, this turns into concrete engineering controls: least-privilege API keys and scoped tool permissions for any agent that can write files, call other services, or touch a database; input and output filtering at the application boundary rather than trusting the model to self-censor; signed and provenance-checked model artifacts instead of pulling arbitrary checkpoints from public hubs; and dependency scanning tuned to catch known-vulnerable versions of langchain, transformers, and similar frameworks before they reach production. Because most of the incidents above trace back to a dependency, a permission, or an unfiltered output rather than the model's reasoning itself, the highest-leverage fixes are the same supply chain and access-control discipline security teams already apply everywhere else — just extended to cover model files, prompt pipelines, and agent tool grants.
How Safeguard Helps
Safeguard extends the same supply chain rigor security teams apply to open source dependencies to the AI stack itself. Reachability analysis determines whether a vulnerable function in a package like langchain or transformers — say, the code path behind CVE-2023-29374 — is actually called by your code, so teams can prioritize the handful of exploitable findings instead of triaging every CVE in the tree. Griffin AI reads that same dependency and call-graph context to explain why a given LLM library or model-loading path is risky and to draft a fix, rather than surfacing a bare CVE ID and leaving engineers to reverse-engineer the impact. SBOM generation and ingest give security teams a real inventory of every model, package, and container feeding an AI application, closing the visibility gap that let 100 malicious Hugging Face models go unnoticed by their downloaders in early 2024. And auto-fix PRs turn a validated, reachable finding directly into a reviewable pull request that bumps the vulnerable dependency or patches the unsafe call, so remediation ships in the same workflow engineers already use instead of sitting in a backlog.