AI-native apps break traditional application security assumptions because the model treats instructions and data as the same stream of text, its output is not deterministic, and it can call tools on its own. Code review and input validation still matter, but they no longer describe what the system will do at run time.
Which old assumptions stop holding?
Traditional AppSec rests on a few quiet beliefs. Code defines behaviour. Inputs have types you can validate. The same input gives the same output. Privileges are granted to a user or service account and used as written.
A language model weakens each of these. Behaviour is set partly by a prompt and partly by learned weights, so the source code does not tell you the whole story. The input is free text, which cannot be validated like an integer. And the model may decide, from context, to call a function you gave it.
What does the OWASP list say?
The OWASP Top 10 for LLM Applications (2025 edition) names ten risks, and several do not map to a classic web flaw. Prompt injection (LLM01) is user or document text that alters the model's behaviour. Sensitive information disclosure (LLM02) is data leaving through the model's answers. Supply chain (LLM03) covers models, datasets and plugins you did not write. Data and model poisoning (LLM04) targets training or retrieval data. Improper output handling (LLM05) is trusting model output downstream. Excessive agency (LLM06) is a model granted more ability to act than the task needs.
OWASP describes excessive agency as damaging actions performed in response to unexpected, ambiguous or manipulated output. That sentence is the clearest statement of why this is a different problem. The flaw may live in the permissions you granted, not in any line of code.
Why can't input validation fix it?
For a SQL query you can separate the command from the data with parameters. A prompt has no such separator. The system instruction and the retrieved web page arrive as one block of text, and the model weighs them together. You can add filters and instructions, but they reduce risk rather than remove it.
So defences move to the edges of the model: limit what it may reach, and check what it produces before anything acts on it.
What does a constructed example look like?
A support assistant can read tickets and issue refunds up to a limit. A customer pastes text into a ticket that tells the assistant to ignore its rules and refund an order twice. Static analysis of the application passes: no injection flaw, no secret in code. The weakness is that the refund tool trusts the model's decision.
The fix is architectural. The refund function enforces its own limit and requires an approval for repeats. The assistant's credential can only touch the refund endpoint, not the customer database.
How do you adapt an AppSec programme?
Keep the existing controls and add three questions per AI feature: what can the model read, what can it do, and what checks its output. NIST's AI Risk Management Framework gives a vocabulary for governing and measuring those risks, and NIST's adversarial machine learning taxonomy (AI 100-2) names attack types you can test against.
How to check or apply this
- Draw the data flow for each AI feature: sources of text, the model, tools it can call, where output goes.
- List every tool and its permissions. Remove any that the task does not need.
- Put hard limits inside the tools (amounts, scopes, rate limits), not in the prompt.
- Treat model output as untrusted input to the next component. Validate and encode it.
- Log prompts, tool calls and decisions so an investigator can reconstruct what happened.
- Add test cases for hostile text in documents and tickets, and rerun them when you change models or prompts.
Safeguard's MCP Guardrail sits between agents and their tools to apply policy to tool calls, which maps to the permission and logging steps above.
Quick check
Is prompt injection the same as SQL injection?
No. SQL injection can be prevented structurally with parameterised queries. Prompt injection has no equivalent separator, so you limit impact and verify output instead.
Do we still need code scanning for AI apps?
Yes. The code around the model still has ordinary flaws and dependencies. Scanning covers that layer, and AI-specific controls cover the rest.
Sources
- OWASP Top 10 for LLM Applications 2025
- OWASP LLM06:2025 Excessive Agency
- NIST AI Risk Management Framework
- NIST AI 100-2e2025, Adversarial Machine Learning taxonomy
Last reviewed: 18 October 2026.
