Adversarial attacks on machine learning deliberately manipulate a model or its data to cause wrong, harmful or revealing behaviour. NIST's taxonomy groups them by attacker goal, capability and the stage of the model life cycle they target, such as training, deployment or querying.
What does NIST's taxonomy cover?
NIST's report AI 100-2 provides a taxonomy and defines terminology in adversarial machine learning. It arranges concepts in a hierarchy that includes the main types of machine learning methods, the life cycle stages of attack, and attacker goals, objectives, capabilities and knowledge. It also describes challenges in the life cycle of AI systems and methods for mitigating and managing the consequences of attacks.
The value is a shared vocabulary. When a team says "we were attacked", the taxonomy lets them say which stage, with what knowledge, aiming at what.
What are the main attack families?
These are the broad groups the field uses, summarised in plain words. Check the NIST report for the precise definitions.
Evasion. At the time a model is used, an attacker alters an input to cause a wrong output. A modified image that a classifier misreads, or text crafted to slip past a content filter.
Poisoning. During training or data collection, an attacker inserts or alters data so the model learns something wrong, or contains a hidden trigger. OWASP lists data and model poisoning as LLM04 in its 2025 list for LLM applications.
Privacy attacks. An attacker queries a model to learn about its training data or the model itself, for example to test whether a record was in the training set.
Abuse and misuse of generative systems. Prompt injection, where text changes the model's behaviour, is LLM01 on the OWASP list. Excessive agency, LLM06, covers a model with too much ability to act.
How does attacker knowledge change things?
The taxonomy separates what the attacker knows. With full knowledge of the model and weights, attacks can be precisely crafted. With only the ability to send queries and read answers, attacks need more probing but are still possible. Planning for a capable attacker with query access is a reasonable default for a public model.
What does a constructed example look like?
A company runs a classifier that flags abusive messages. An attacker discovers that adding odd spacing and look-alike characters keeps the meaning for a human but drops the model's score. The classifier now misses abuse. That is evasion.
Separately, the company retrains monthly on user reports. A group submits many false reports against a rival's harmless messages, hoping to teach the model to flag them. That is poisoning, aimed at the data pipeline.
What defences are practical?
There is no complete fix, and NIST is open about managing consequences rather than eliminating attacks. Practical steps:
- Test adversarially. Include altered inputs in your evaluation set, and rerun after each model change.
- Protect the data pipeline. Control who and what can write to training data, record where data came from, and review sudden shifts.
- Limit query exposure. Rate limit, authenticate, and monitor for probing patterns.
- Add layers around the model. Validate inputs and check outputs with rules that do not depend on the model.
- Limit consequences. Do not let a single model decision trigger an irreversible action without a second check.
- Plan response. Know how to roll back to a previous model.
How to check or apply this
- List each model you run, where its training data comes from, and who can change it.
- For each, write which attack families apply, using the NIST taxonomy as the checklist.
- Add adversarial examples to evaluation and run them on each release.
- Rate limit and log queries to externally reachable models.
- Keep versions of models and data so you can roll back.
- Review the OWASP LLM Top 10 for any language model features.
Safeguard's MCP Guardrail can check tool calls that an AI agent makes, which addresses the consequences step for agent-based systems.
Quick check
Is adversarial machine learning the same as prompt injection?
Prompt injection is one attack against language model applications. Adversarial machine learning is the broader field that also includes evasion, poisoning and privacy attacks.
Can these attacks be fully prevented?
No. NIST discusses mitigating and managing consequences. Layered controls reduce risk.
Sources
- NIST AI 100-2e2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
- OWASP Top 10 for LLM Applications 2025
- NIST AI Risk Management Framework
Last reviewed: 18 October 2026.
