A neural network is a function made of layers of simple units. Each unit multiplies its inputs by learned weights, adds them up and applies a nonlinear step. Training adjusts the weights until outputs match examples. Because it learns from data, attackers can target the data, the inputs or the model itself.
How does one actually compute?
Take a tiny constructed network that flags suspicious logins. It takes three inputs: failed attempts in the last hour, whether the country is new, and whether the hour is unusual. Each input is multiplied by a weight, say 0.8, 1.5 and 0.4. The sum, plus a bias, goes through an activation function that squashes it into a number between 0 and 1. A value above 0.5 means "suspicious".
A real network stacks many layers of such units, with thousands to billions of weights. The first layers pick up simple features and later layers combine them. No one writes the rules by hand. Training does it.
How does training work?
Start with random weights. Feed a batch of examples with known answers. Measure the error between prediction and answer with a loss function. Use backpropagation, which applies the chain rule of calculus to work out how each weight contributed to the error, then nudge the weights in the direction that reduces it. Repeat for many passes.
Two consequences matter for security. The model is a compressed record of its training data, so whatever was in the data can shape, and sometimes leak through, the behaviour. And the learned rules are not readable, so you cannot audit them by reading code.
What attacks apply?
NIST's report Adversarial Machine Learning, NIST AI 100-2 gives a shared taxonomy of attacks and mitigations for predictive and generative AI systems. In broad terms, it groups attacks by the stage they hit and what the attacker wants.
Poisoning. Corrupt the training data so the model learns something harmful, such as a hidden trigger that makes it misclassify when a pattern appears.
Evasion. At runtime, craft an input that looks normal to a person but pushes the model to a wrong output. Small, deliberate changes to an image are the standard example.
Privacy attacks. Probe the model to learn whether a record was in its training data, or to reconstruct parts of it.
Model extraction and theft. Query a model enough to copy its behaviour, or steal the weights outright.
Supply chain. Pretrained models and datasets come from outside. A tampered download carries the problem into your system.
The report is the place to check precise definitions, which keep evolving, rather than relying on summaries.
What should defenders do?
Treat the model like any other asset with an owner, a version and a lineage. Know where the training data came from and who could change it. Test the model against adversarial inputs, not just clean test data. Limit and monitor queries, because both extraction and probing need volume. And do not let the model's decision be the only control on a high-impact action.
NIST's AI Risk Management Framework gives a way to organise this work across governance, mapping, measuring and managing risks.
How to check or apply this
- Inventory every model in use: source, version, hash, training data lineage and owner.
- Restrict write access to training and evaluation datasets, and log changes.
- Verify downloaded models by hash and prefer data-only file formats.
- Add adversarial and out-of-distribution cases to your test suite, and rerun after every retrain.
- Rate limit and log queries to model endpoints. Alert on systematic probing.
- Keep a human or rule-based check on high-impact outputs.
- Define how to roll back to a previous model version quickly.
- Document limits: what inputs the model was and was not built for.
Quick check
Is a neural network the same as AI?
No. Neural networks are one family of machine learning methods inside the broader field of AI. Many AI systems use other techniques.
Can a model be backdoored without touching its code?
Yes. Changing training data or weights can alter behaviour while the surrounding code stays identical, which is why model provenance and hashes matter.
Sources
Last reviewed: 18 October 2026.
