AI-based intrusion detection learns a profile of normal behavior for users, hosts or networks, then flags activity that deviates from it. It can catch threats that have no known signature. It also produces false positives, can be trained on bad data, and can be slowly fooled, so it works best next to other methods.
What are the main detection methods?
NIST SP 800-94, the guide to intrusion detection and prevention systems (IDPS), describes three methods.
Signature-based detection compares observed events with patterns of known threats. NIST says it is very effective against known threats but largely ineffective against unknown threats, evasion and many variants. Changing one filename can defeat a signature.
Anomaly-based detection compares definitions of normal activity with observed events to find significant deviations. The IDPS builds profiles for users, hosts, network connections or applications by watching typical activity over a period, then uses statistical methods to compare current activity to thresholds. NIST's example is a network where web traffic averages 13 percent of border bandwidth during workdays, and an alert when it is much higher.
Stateful protocol analysis compares activity to vendor-developed profiles of how a protocol should behave and tracks the state of communications.
What people call AI or machine learning detection is, in most products, an advanced form of the anomaly approach. The profile is learned by a model instead of set by hand.
What are the benefits?
The main benefit, in NIST's words, is that anomaly detection can be very effective at finding previously unknown threats. A new piece of malware that sends large volumes of email or opens many connections will differ from the host's normal behavior even if no one has seen the malware before.
CISA's joint guidance on living off the land techniques recommends similar thinking: build or acquire automation, such as machine learning models, to review logs continually against baselines and alert on anomalies, and use user and entity behavior analytics. It matters most where attackers use legitimate tools that signatures cannot flag.
Where does it fail?
NIST SP 800-94 lists problems that still apply to newer models:
- Training period. Profiles take days or weeks to build, and malicious activity during that time can become part of normal.
- Static profiles go stale as systems change, while dynamic profiles can be evaded by an attacker who increases activity slowly enough to be treated as normal.
- Rare legitimate events. A monthly maintenance job with large file transfers may not appear in training, and then triggers an alert.
- False positives, especially in diverse or dynamic environments.
- Explainability. Analysts often struggle to say why an alert fired and to confirm it is real, because of the complexity of the profile.
The NIST report on adversarial machine learning (AI 100-2) adds a layer: it provides a taxonomy of attacks on machine learning systems by life cycle stage and attacker goals, capabilities and knowledge. A detection model is itself a system that can be attacked, for example through poisoned training data. This article did not review the report's details, so read it before relying on a model in an adversarial setting.
A constructed example
A company enables anomaly detection on its file servers. In week one, alerts flood in because the quarterly backup test was not in the training period. The team marks that job as known and the noise drops. In week six the system flags a finance user who downloaded 3,000 files at night. It is a real incident: a stolen password. No signature would have matched, because the attacker used a normal download feature. The same system missed an earlier compromised account whose owner always worked late.
How to check or apply this
- Define what you want to catch: account abuse, data theft, lateral movement. Pick data sources for each.
- Run the model in alert-only mode and record what it flags for 30 days before it can block anything.
- Keep signature and rule detection running. Treat anomaly detection as an addition.
- Create a feedback process: each alert is tagged real, benign or unknown, and the tags go back into tuning.
- Keep the training data clean: avoid training on periods with known incidents.
- Test with a planned exercise, such as a red-team user downloading unusual volumes, and measure time to alert.
- Ask for explanation: what feature or behavior triggered this alert? Reject tools that cannot say.
Quick check
Is AI detection better than signatures?
They catch different things. Signatures are precise for known threats. Anomaly models can flag unknown ones but produce more false positives. NIST says organizations should consider using more than one type of detection technology.
Can attackers evade anomaly detection?
Yes. NIST describes slowly increasing malicious activity so a dynamic profile treats it as normal, and notes that malicious activity during training can become part of the profile.
Sources
- NIST SP 800-94, Guide to Intrusion Detection and Prevention Systems (IDPS)
- NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology
- CISA, Identifying and Mitigating Living Off the Land Techniques
Last reviewed: 18 October 2026.
