Build a frontier AI security roadmap in four stages that mirror NIST's AI Risk Management Framework: set governance and an inventory, map each use case and its risks, measure with tests and red teaming, then manage with controls and monitoring. Do the inventory first. You cannot secure model use you do not know about.
What does "frontier AI" mean here?
The term has no single standards definition in the sources reviewed for this post. This guide uses it for the most capable general-purpose models that organizations call through APIs or run themselves, and for the agents built on them. The advice applies equally to smaller models.
Stage 1: Govern
NIST's AI RMF describes GOVERN as a cross-cutting function infused throughout AI risk management that enables the other three. It includes policies, risk tolerance, accountability, and documentation, which NIST says can improve transparency, human review and accountability.
Concrete outputs: an owner for AI risk, an approved-use policy, a list of data classes that may not go to external models, and a process for approving new AI uses. Without these, each team decides alone.
Stage 2: Map
MAP is where you identify what each AI system is for, where it runs, what data it touches and what could go wrong. Build an inventory with these fields: use case, model and provider, data sent, tools or actions the system can invoke, users, and owner.
NIST's generative AI profile (AI 600-1) is a companion to the RMF for generative systems. It organizes risks specific to or made worse by generative AI, and includes information security among them.
Stage 3: Measure
Test before you trust. The OWASP Top 10 for LLM applications (2025) is a practical test list: prompt injection, sensitive information disclosure, supply chain weaknesses, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption.
NIST's adversarial machine learning taxonomy (AI 100-2e2025) adds vocabulary for attacks across the lifecycle: evasion, poisoning, privacy and abuse. It also notes that mitigations come with trade-offs and that models resistant to attack remain an open problem, so measure results on your own use cases instead of trusting general claims.
Run red team exercises that match your real tool access. A chatbot with no tools has a small blast radius. An agent that can send email, read files and call internal APIs does not.
Stage 4: Manage
Controls follow the measured risks.
Limit agency. Give agents the narrowest tool set and the least privilege. Require approval for high-impact actions, such as payments, deletions and external messages.
Control data. Block restricted data classes from leaving, and decide where prompts and outputs are logged, mindful that logs hold sensitive content.
Validate outputs. Treat model output as untrusted input to the next system, especially when it feeds code execution, queries or markup.
Secure the supply chain. Models, plugins, tool servers and packages are dependencies. Inventory them and review changes.
Monitor and respond. Trace model and tool calls, watch for abuse and cost spikes, and write an incident playbook for AI-specific failures.
A constructed example
A company wants an internal agent that reads support tickets and drafts refunds. In the Govern stage, the CFO's office sets a refund approval limit. In Map, the team records that the agent reads customer data and can call a refund tool. In Measure, a red team plants an instruction inside a ticket telling the agent to refund a different customer. The agent complies in testing. In Manage, the team restricts the tool to the ticket's own customer, requires human approval above a small amount, and logs every call. Retesting shows the planted instruction no longer works.
How to check or apply this
- Name an owner and publish an approved-use policy within a month.
- Build the inventory of AI uses, including shadow ones found by asking teams.
- Rank uses by data sensitivity and tool power.
- Test the top five with the OWASP list and a red team.
- Add controls for agency, data, output handling and supply chain.
- Turn on tracing for model and tool calls with content capture limited.
- Review quarterly, since models and uses change.
Safeguard's relevant capabilities are guardrails, MCP Guardrail for tool calls, agent identity and policies, along with dependency scanning for the packages AI projects pull in. They support stages 2 to 4, but governance remains your own.
Quick check
What should come first in an AI security roadmap?
Governance and an inventory of AI uses. Everything else, from testing to controls, depends on knowing what is in use and who owns it.
Is the OWASP LLM Top 10 a standard?
It is a community risk list, not a certification standard. It works well as a checklist for testing but does not replace a risk assessment.
Sources
- NIST AI 100-1, Artificial Intelligence Risk Management Framework
- NIST AI 600-1, Generative AI Profile
- NIST AI 100-2e2025, Adversarial Machine Learning
- OWASP Gen AI Security Project, Top 10 for LLM applications
Last reviewed: 18 October 2026.
