Adversarial AI attacks are deliberate attempts to make a machine learning system behave wrongly, leak data or reveal itself. NIST's taxonomy sorts them by lifecycle stage, attacker goal, capability and knowledge. Defense starts with knowing which stage an attacker could reach, then limiting access, checking data and monitoring outputs.
Where does the standard vocabulary come from?
NIST published AI 100-2 E2025, "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations," in March 2025. It is written by authors from NIST, academia and industry, including the UK AI Security Institute. It arranges attacks in a hierarchy that covers key types of machine learning methods, stages of the lifecycle, and the goals, capabilities and knowledge of attackers. It also describes mitigations and open challenges.
MITRE's ATLAS is a companion knowledge base that catalogs adversary tactics against AI systems in a style familiar to ATT&CK users.
What are the main attack families?
Using the taxonomy's framing, think in four questions about what an attacker wants:
- Make it wrong at use time (evasion): change an input slightly so the model misclassifies it. A sticker on a sign, a few altered pixels, or text crafted to slip past a filter.
- Make it wrong by corrupting training (poisoning): insert bad samples or labels into training data so the model learns a flaw, or a hidden trigger.
- Make it reveal things (privacy attacks): infer whether a record was in the training set, or recover pieces of training data or the model itself.
- Make it misbehave on purpose (abuse and prompt attacks on generative systems): get a language model to ignore its instructions or produce what it should refuse.
The report notes that generative systems add attacks beyond the older predictive-model categories.
Why is this hard to defend against?
A model learns patterns from data, and an attacker only needs to find inputs where the pattern fails. Defenses often trade accuracy or cost. Many published defenses were later broken by adaptive attackers, which is why NIST frames mitigations with caveats rather than guarantees. Do not expect a single control to close this category.
A constructed example
A spam filter retrains weekly on messages users mark as "not spam." An attacker creates thousands of accounts and marks their own campaign messages as safe, week after week. The next model passes the campaign. No one hacked a server. The attack went through the data pipeline. The defenses are plain: limit how many labels one account can influence, weight trusted reviewers more, keep a held-out test set the attacker cannot touch, and alert when accuracy on it drops.
Does this apply if you only use a hosted model?
Yes, though differently. You do not control the model's training data, so poisoning of the base model is the provider's problem. You still control what you feed it, what it can call and what you do with its output. If you fine-tune, use retrieval from your own documents, or let users supply content, you create your own paths for poisoned input. The question to ask is which of the model's inputs an outsider can influence, and what the system does next with its answer.
How to check or apply this
- List your models, and for each note the training data source, who can add to it, and who can query the model.
- Map threats by stage using the NIST taxonomy: training, deployment, and query time.
- Treat training data like code. Track its origin, verify it, and restrict write access.
- Rate limit and monitor queries to deployed models to slow extraction and probing.
- Test models with adversarial inputs before release and after each retraining.
- Keep a clean evaluation set that is never part of any feedback loop.
- For language model applications, separate untrusted text from instructions, and limit the actions the model can take.
- Verify the provenance of pretrained models and datasets you download.
Safeguard scans dependencies, and our MCP Guardrail and policies can limit which tools an AI agent may call, which narrows what a manipulated model can do.
Quick check
Is prompt injection an adversarial attack?
Yes. It is an attack on generative systems that tries to override their instructions through crafted input.
Can I just filter bad inputs?
Filters help but are not complete. Attackers adapt, so combine filtering with access limits, monitoring and testing.
Sources
Last reviewed: 18 October 2026.
