How to test a detection rule change without fooling yourself

To test a detection rule change fairly, tune on one slice of data and evaluate on a different slice the tuning never saw. Keep the old rule running in parallel as a baseline, and write down what result would make you roll back before you look.

The mistake

You review last month's 300 alerts, notice that 120 come from one noisy pattern, add an exclusion, and re-run the rule on the same alerts. The false positives fall by 40 percent. That number is meaningless, because you built the exclusion by looking at those exact alerts. Of course it removes them.

This is overfitting, and it is as common in rule writing as in machine learning.

A simple procedure

1. Split by time. Use the earlier weeks to design the change and the later weeks to test it. Time based splits beat random ones because real attackers and noise both change with time.
2. Freeze the change. Do not adjust after seeing the test period. If you need another iteration, collect new data for the next test.
3. Run both versions. Keep the old rule in shadow mode alongside the new one for a period, so you compare on identical traffic.
4. Compare on two sides. Count false alerts removed, and count real alerts lost. You need labelled real cases for the second. Use past incidents and simulation tests, as in measuring detection coverage.
5. Decide with a pre written rule. For example: ship if false alerts fall by at least a third and no known real case is lost.

Example with numbers

Illustrative. Weeks 1 to 4 produced 300 alerts, 120 from one pattern. You add an exclusion for that pattern based on those weeks.

Weeks 5 and 6 produce 150 alerts in the old rule. The new rule would raise 100. So 50 fewer alerts, a one third drop. Of the 50 removed, you review them and confirm 48 are benign and 2 are real.

• False alerts removed: 48.
• Real alerts lost: 2.

Whether to ship depends on those two real alerts. Perhaps the exclusion is too broad and needs narrowing. Without the holdout you would never have seen it.

Guard against drift

• Log every exclusion with owner, reason, date and an expiry.
• Review old exclusions each quarter, since a pattern that was benign last year may now hide something.
• Keep a small set of labelled real alerts as a standing regression test. Run it after every rule change.

Cost of the check

This takes a few extra days per change. For a rule that fires hundreds of times a month, it is cheap compared with the cost of silently going blind. The hours saved by a good change are easy to estimate, as we show in the arithmetic of alert fatigue.

Quick check

Last reviewed: 19 October 2026.