A detector that is 99 percent accurate can still produce alerts that are wrong about 99 times out of 100. The reason is the base rate. When real attacks are rare among the events you inspect, even a small false positive rate produces far more false alarms than true hits.
The worked example
The numbers below are made up to show the arithmetic. Say your detector catches 99 percent of malicious events and wrongly flags 1 percent of benign ones. Say 1 event in 10,000 is actually malicious, and you inspect 1,000,000 events.
- Malicious events: 100. The detector flags 99 of them.
- Benign events: 999,900. The detector flags 1 percent, which is 9,999.
- Total alerts: 99 plus 9,999, which is 10,098.
Only 99 of those 10,098 alerts are real. That is about 0.98 percent, or roughly 1 real alert per 102 you open. The quoted accuracy was true and still told you almost nothing about alert quality.
The number you actually need
The quantity that matters to an analyst is positive predictive value, the share of alerts that are real. It depends on three inputs: sensitivity, false positive rate, and base rate. Vendors usually quote the first, sometimes the second, and almost never the third, because the third depends on your environment.
The formula, in words: real alerts divided by all alerts, where real alerts equal sensitivity times base rate, and all alerts add the false positive rate times the benign share.
Why base rates differ so much
Base rate is not a fixed fact about the world. It moves with where you point the detector.
- Scanning every outbound connection has a tiny base rate of malice. Scanning only connections to newly registered domains from a server that never makes outbound calls has a much higher one.
- Flagging every dependency with any known advisory has a low base rate of real exposure. Flagging only advisories on code your app actually calls raises it.
- A rule that runs on a privileged admin account sees a different mix than the same rule on every laptop.
This is why narrowing the population before the detector runs is often a better lever than making the detector slightly more accurate. Cutting the benign events by 90 percent does what a 10 times lower false positive rate would do, usually at lower cost.
What to ask when someone quotes accuracy
- What was the base rate in the test data, and does it resemble mine?
- Is the quoted number accuracy, or is it precision and recall reported separately?
- How many alerts per day would this produce on my volume, with a range?
- Who labelled the test set, and could the detector have seen those examples before?
If the answers are vague, run a small pilot and count. Our post on measuring SCA false positives in a pilot walks through a lightweight way to do that.
A practical rule of thumb
Before you buy or tune anything, write down the expected daily event count and your best guess at how many are truly bad. Multiply by the false positive rate. If the result is more alerts than your team can open, the detector is not ready for that population, whatever its headline accuracy says.
Safeguard applies this thinking to dependency findings: reachability analysis narrows a list to the vulnerabilities your code can actually call, which raises the share of findings worth a person's time. You can try it on a repository with the free scan at gold.safeguard.sh/scan.
Quick check
Is accuracy a good way to compare detectors?
Not on its own. Accuracy hides how rare real attacks are. Compare precision and recall, and test on data with a base rate close to yours.
Does a lower false positive rate always fix alert floods?
It helps, but shrinking the population the detector looks at can have the same effect and is often easier to do.
Last reviewed: 19 October 2026.
