Generative AI gives application security teams two jobs at once. They must secure applications that contain models, and they must review code that models help write. The risks are catalogued by OWASP and NIST, so teams do not need to invent a framework. They need ownership, rules and a review path.
What are the risks in applications that use models?
The OWASP Top 10 for LLM Applications, 2025 edition, gives a shared vocabulary. Its ten entries are prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption.
Several of these look familiar to an appsec engineer. Improper output handling is the old rule that you do not trust data from another component, applied to model output. Supply chain covers the models, datasets and plugins you pull in. Unbounded consumption is a resource exhaustion problem with a cost attached.
Others are new in shape. Prompt injection exists because instructions and data share one channel. Excessive agency appears when a model is given tools and permissions beyond what its task needs.
What changes when developers use AI assistants?
Code arrives faster, from a source that does not understand your threat model. Reviewers see more diffs and sometimes less intent behind them. Assistants can suggest packages, and a suggested name may not exist or may belong to someone else. We cover that in a separate post on package hallucinations.
None of this means banning the tools. It means that the checks you apply to human code, such as static analysis, dependency checks and secrets scanning, must run on every change regardless of author, and must run automatically.
How does NIST frame it?
NIST released its Generative AI Profile, NIST AI 600-1, on 26 July 2024. It sits on the AI Risk Management Framework, which NIST released on 26 January 2023 and which is organized around four elements: Govern, Map, Measure and Manage. The profile helps organizations identify risks specific to generative AI and suggests actions that fit their goals. For an appsec team, Govern is where policy and accountability live, and Map is where you list each model, data source and tool connection.
Constructed example: the helpful support bot
A company adds a support assistant that can look up orders and issue refunds. It reads customer messages, so a message can contain instructions. If the assistant can refund without limit, a crafted message may talk it into paying out. The fixes are classic appsec: give the assistant only a narrow refund tool with a hard cap enforced outside the model, require human approval above a threshold, and log every tool call. The model's good behavior is never the control. The permission boundary is.
What do teams struggle with?
Ownership. Product teams add model features without telling security. An inventory of where models are used is the first fix.
Testing. Model output is not deterministic, so tests need ranges and adversarial cases, not one expected string.
Third party exposure. Prompts may carry customer data to an outside provider. Check what is sent and what the provider retains.
Agent permissions. As assistants gain tools, identity and access become the central question. Safeguard offers MCP Guardrail and agent identity controls for teams who want a policy layer between agents and the tools they call.
How to check or apply this
- List every application feature and internal tool that uses a model, with an owner.
- Map each to the OWASP LLM categories and note which apply.
- Remove permissions an assistant does not need and enforce limits outside the model.
- Treat model output as untrusted input before it reaches a browser, shell or database.
- Run static analysis, dependency and secrets checks on all code, including AI assisted changes.
- Decide what data may be sent to external model providers and write it down.
- Log tool calls and review them.
Quick check
Is prompt injection fixed by a better system prompt?
No. A system prompt is guidance the model may not follow. Limit what the application allows the model to do, and enforce those limits outside the model.
Do AI written changes need a different review process?
They need the same automated checks as any change, run every time, plus attention to suggested packages and to code the author cannot explain.
Sources
Last reviewed: 18 October 2026.
