How to measure detection coverage with simulated attack tests

To measure detection coverage, list the attack behaviours you care about, run a safe simulation of each, and record which ones raised a useful alert. Divide detected by tested and report the result with a range, because small test lists give wide uncertainty.

Step 1: choose the behaviours

Pick what attackers would plausibly do in your environment. A list of 20 to 40 is realistic for a first pass. Examples:

• Creating a new admin account outside change control.
• Disabling logging on a host.
• Reading many secrets in a short time.
• A build job calling out to an unknown host.
• A new dependency pulling a postinstall script that runs a shell command.

Use a published behaviour framework as a menu if you like, but weight the list toward your own systems. Coverage of behaviours you cannot encounter is not worth reporting.

Step 2: define "detected"

Decide in advance what counts.

• Alert fired and reached a human or a ticket.
• Alert fired but was buried in noise. Count as missed, or as a separate partial category.
• Logged but no alert. Not detected.

Also record time to detect. A detection that fires two days later is a different result from one that fires in two minutes.

Step 3: run safe tests

Use a controlled environment or approved test accounts. Tell the owners of the systems. Record the exact time of each action so you can find the matching alert later.

Step 4: compute and report with a range

Illustrative result: 25 behaviours tested, 18 detected, so 72 percent. With only 25 tests the uncertainty is wide. A standard interval (Wilson, 95 percent) runs from about 52 to 86 percent.

Report both: "We detected 18 of 25 tested behaviours, which is about 72 percent, and the true share is plausibly between roughly half and 85 percent." If someone asks why the range is so wide, the answer is that 25 tests is a small sample, and our post on the rule of three explains the same logic.

Step 5: turn misses into work

The percent matters less than the list of what was missed. For each miss, decide:

1. Is there a data source we are not collecting?
2. Is there a rule we need to write?
3. Is the alert there but drowned out? That is a precision problem. See precision versus recall.

Then re-run the same tests after the fix. Keep a regression list so an old gap does not reopen when a rule changes.

Pitfalls

• Testing only what you already detect. A test list built from your current rules will score near 100 percent and teach you nothing.
• Treating the percent as a grade across time. If the list changes, the number is not comparable. Version the list.
• Counting coverage without counting noise. A rule that fires on everything has full coverage and no value.
• Running tests once. Environments change. Repeat at a steady cadence.

Quick check

Last reviewed: 19 October 2026.