Vanity vs decision metrics in security reporting

A vanity security metric describes activity: scans run, alerts raised, attacks blocked. A decision metric changes what someone does next, because it has a target, an owner and a threshold that triggers action. If a number can rise or fall without anyone acting differently, it belongs in an appendix at best.

The three tests

Run each metric through three questions.

1. What decision does it inform? If you cannot name one, drop it.
2. What level counts as bad? A metric without a threshold is trivia.
3. Who acts when it crosses the line? No owner means no change.

Common vanity metrics

Number of vulnerabilities found. More scanning finds more. A rising count can mean better coverage. By itself it says nothing about risk.

Attacks blocked. Internet noise is constant. A large number may show that a tool is working, or only that the internet is hostile.

Training completion. Useful as a compliance check. It says little about behavior unless paired with something like phishing report rates.

Tool coverage claimed by the vendor. Coverage means little until you know which assets are actually in scope.

Better replacements

• Instead of vulnerabilities found, track time to fix critical issues on systems that matter, with the share past your target.
• Instead of attacks blocked, track incidents that reached production data, and how long containment took.
• Instead of training completion, track how many phishing reports come from staff, and how fast the team responds.
• Instead of claimed coverage, track share of known assets with the key control verified and the size of the unknown asset pool.

When prioritizing fixes, it helps to use outside signals about real exploitation. CISA describes its Known Exploited Vulnerabilities catalog as the authoritative source of vulnerabilities exploited in the wild, and advises using it as an input to prioritization. A metric such as "known exploited issues open past their due date" is much more decision relevant than a raw count.

Build a short dashboard

Pick five to eight decision metrics. For each, define the formula in one sentence, the data source, the owner, the target and the action on breach. Freeze the definitions for a year so trends mean something. If you must change one, mark the break in the chart.

Watch for gaming

Any metric that becomes a target can be gamed. If fix time is measured, teams may close tickets without fixing. Pair speed with a quality check, such as sampling closed items to confirm the fix.

Quick check

Sources

• CISA Known Exploited Vulnerabilities Catalog

Last reviewed: 19 October 2026.