Simpson's paradox is when a trend that holds in every group reverses once you combine the groups. In security it appears when teams face different mixes of work. A team can patch faster in every severity band and still show a worse overall on time rate, simply because it handles harder items.
The example
Illustrative patch on time figures for two teams. Each handled 100 items.
Team 1:
- Critical: 64 of 80 on time, 80 percent.
- Low: 20 of 20 on time, 100 percent.
- Overall: 84 of 100, 84 percent.
Team 2:
- Critical: 15 of 20 on time, 75 percent.
- Low: 78 of 80 on time, 97.5 percent.
- Overall: 93 of 100, 93 percent.
Team 1 does better on critical items (80 versus 75) and on low items (100 versus 97.5). Yet the overall number ranks Team 2 higher, 93 versus 84.
Why it happens
Overall rate is a weighted average, and the weights differ. Team 1 mostly handles critical items, which are harder to fix on time. Team 2 mostly handles low items, which are easy. The mix, not the skill, drives the total.
In real organisations the mix differs for reasons that have nothing to do with performance: one team owns a legacy service with many critical findings, another owns a small new app.
Where this shows up
- League tables of teams by percent of vulnerabilities fixed within SLA.
- Before and after comparisons of a program, when the type of work changed between the periods.
- Vendor comparisons that report one blended detection rate across easy and hard cases.
- Scanner comparisons on different test sets of different difficulty.
How to avoid being fooled
- Break results out by the factor that sets difficulty, such as severity, system age or exposure, before comparing groups.
- Compare like with like. Rank teams within the same severity band, or compare each team against its own previous period.
- If you need a single number, standardise. Apply the same mix of work, for example the company wide proportion, to every team's group rates and compute a weighted result.
- Show counts. A reader who sees 20 critical items for one team and 80 for another understands at once why the totals mislead.
A related question: which view is right?
Neither the group view nor the total is always correct. It depends on what you want to decide. If you are judging how well a team handles critical work, use the critical band. If you are measuring how much of the whole organisation's backlog is cleared, the total answers it. Be explicit about which question the chart answers.
For the counts and denominators behind any such chart, read percent versus count in security metrics.
Quick check
Does Simpson's paradox mean the data is wrong?
No. Every number is correct. The paradox comes from combining groups of different size and difficulty, so the total says something different from the parts.
How can I spot it early?
Whenever you compare groups, ask whether their mix of work differs. If it does, split by that factor before drawing conclusions.
Last reviewed: 19 October 2026.
