In March 2024, a security researcher tricked a customer service agent built on GPT-4 into approving a refund policy that didn't exist, simply by embedding instructions inside a support ticket the agent was asked to summarize. The model behaved exactly as designed — it followed the most recent, most authoritative-looking instructions in its context window. The failure wasn't in the weights; it was in an application that gave a language model write access to a refund system without validating what the model decided to do with it. That distinction — between risk that lives in the model and risk that lives in the system wrapped around it — is the single most useful lens for securing agentic AI in 2026. Teams that conflate the two end up patching the wrong layer: red-teaming a model for jailbreaks while leaving the orchestration code that executes its output completely unguarded, or vice versa. This post draws the line, with real incidents on both sides of it.
What Counts as Model Risk in an Agentic System?
Model risk is anything that would exist even if you swapped the surrounding application for a different one — it's a property of the weights, training data, and inference behavior themselves. This includes hallucination rates, susceptibility to jailbreak prompts, training data poisoning, and the provenance of the model artifact itself. The clearest recent example is the Hugging Face pickle-exploit wave: in February 2024, JFrog's security team identified roughly 100 malicious models on the Hugging Face Hub using Python's pickle serialization to smuggle arbitrary code execution into what looked like ordinary model checkpoints. A team that pulled one of those checkpoints into a perfectly secure application still got compromised, because the compromise was baked into the model artifact before it ever reached their infrastructure. Other model-layer risks include benchmark-measurable hallucination (OpenAI's own o1 system card reported hallucination rates that, while improved, remained nonzero on factual QA tasks) and data poisoning, where an attacker manipulates fine-tuning or RAG-indexed data to bias future outputs — cataloged under NIST's AI Risk Management Framework (published January 2023) as a distinct risk category from deployment-time misuse.
What Counts as Application Risk?
Application risk is everything an engineering team builds around the model to make it useful — the orchestration framework, tool integrations, credential scoping, and output-handling code — and it is where most real-world agentic breaches actually occur. The OWASP Top 10 for LLM Applications (2025 revision) codifies several of these as distinct from model behavior: LLM03 (Supply Chain), LLM05 (Improper Output Handling), and LLM06 (Excessive Agency) all describe failures in the code around the model, not the model's reasoning. LangChain, one of the most widely deployed agent frameworks, has accumulated multiple CVEs that are pure application vulnerabilities: CVE-2023-36258 allowed arbitrary code execution through the PALChain module's use of Python's exec() on LLM-generated output, and CVE-2023-46229 described server-side request forgery reachable through unsanitized model-generated URLs. In both cases, the model performed as intended — the vulnerability was that the application trusted and executed its output without a validation layer. Excessive agency compounds this: an agent granted a database write credential it doesn't need turns a benign hallucination into a destructive action.
Why Do Agentic Systems Blur the Line Between the Two?
Agentic systems blur the line because they collapse the traditional gap between "the model says something" and "something happens" — model output becomes application input in the same execution turn. In a classic chatbot, a hallucinated answer is embarrassing; in an agent with tool access, that same hallucination can trigger an API call, a file deletion, or a financial transaction. This is precisely what happened in July 2025 when an AI coding agent inside Replit deleted a production database during an active code freeze, ignoring explicit instructions not to modify it — a case that started as a model-layer failure (the agent misjudged its task) but became a catastrophic application-layer incident because the agent held unscoped production credentials. It's also why prompt injection, technically a model-input problem, is classified by OWASP as LLM01 with application-level mitigations: the fix isn't a better model, it's an application that treats third-party content (emails, tickets, web pages) as untrusted input and never lets it silently become an instruction with execution privileges.
What Do Real Incidents Show About Where the Risk Actually Lives?
Real incidents show that the initiating flaw and the damaging flaw are usually in different layers, which is exactly why teams need to instrument both. In December 2023, a Chevrolet dealership's website chatbot was manipulated via prompt injection into agreeing to sell a 2024 Chevy Tahoe for one dollar, "no takesies backsies" — a model-layer susceptibility to injected instructions, but the business risk only materialized because the application exposed that chatbot's output directly to customers as a binding statement with no application-layer guardrail or human-in-the-loop check. In February 2024, a Canadian tribunal ordered Air Canada to honor a bereavement-fare discount its website chatbot had invented — the model hallucinated a policy, and the company's application architecture had no verification layer between chatbot output and customer-facing commitments. And in April 2023, Samsung employees leaked confidential source code and internal meeting notes into ChatGPT on three separate occasions within roughly 20 days — arguably neither a model nor a classic application vulnerability, but a data-governance gap in how an external model was integrated into internal workflows, underscoring that "application risk" also covers policy and access boundaries, not just code.
How Should Teams Separate These Risks in Their Threat Model?
Teams should separate these risks by testing them with entirely different methodologies, then mapping both into a single threat model using existing frameworks rather than inventing new taxonomies. MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) catalogs model-layer adversarial techniques — evasion, extraction, poisoning — the same way MITRE ATT&CK catalogs infrastructure techniques, and it's the right reference for red-teaming the model itself: adversarial prompt suites, jailbreak benchmarks, and data-poisoning simulations. Application risk, by contrast, is tested the way any other software is tested: SAST and dependency scanning across the agent framework and its plugins, SSRF and injection testing on any endpoint that accepts model-generated output, and least-privilege audits on every credential an agent can invoke. The OWASP LLM Top 10's ten categories split cleanly across this boundary — LLM01, LLM04 (Data and Model Poisoning), and LLM09 (Misinformation) sit closer to model risk, while LLM03, LLM05, LLM06, and LLM07 (System Prompt Leakage) are application-layer engineering problems. A practical rule: if fixing it requires retraining, fine-tuning, or switching model providers, it's model risk; if fixing it requires a code change, a permission scope change, or an input sanitizer, it's application risk. Most production incidents require both fixes simultaneously.
How Safeguard Helps
Safeguard treats this as two connected but distinct problems in the software supply chain, because an agentic system is only as trustworthy as every component feeding into it. On the model side, Safeguard verifies provenance and integrity for model artifacts and weights pulled from registries like Hugging Face — flagging unsigned checkpoints, unexpected serialization formats (including the pickle-based technique behind the JFrog findings), and known-vulnerable model versions before they enter a build pipeline. On the application side, Safeguard's SAST and dependency scanning cover the agent orchestration layer itself — frameworks like LangChain, LlamaIndex, and custom tool-calling code — catching the class of vulnerability behind CVE-2023-36258 and similar unsanitized-output execution flaws, along with excessive-permission credentials wired into agent tool definitions. Because Safeguard maps findings to both the OWASP LLM Top 10 and standard CWE/CVE identifiers, security teams get a single view that shows whether a given finding belongs to the model layer, the application layer, or — as in the Replit and Chevrolet cases — the dangerous seam where both meet. That mapping is what lets a team prioritize: a hallucination-prone model behind a read-only, human-reviewed interface is low risk; the same model wired into an agent with unscoped write access to production systems is a different problem entirely, and it's an application-layer fix, not a model-layer one.