For a rough false positive rate within plus or minus 10 points, review about 100 randomly chosen findings. For plus or minus 5 points, about 385. If your list is a few hundred items long, a correction shrinks those numbers. Below about 30 reviewed items, treat the result as a hint, not a measurement.
The formula
For a yes or no question such as "is this finding a false positive", the sample size for a 95 percent confidence level is:
n = 1.96 squared times p times (1 minus p), divided by e squared
Here p is your guess at the false positive rate and e is the margin of error you can live with, both as fractions. If you have no guess for p, use 0.5. That is the worst case and gives the largest n.
Three worked sizes
The arithmetic below is exact for the formula, applied to made up situations.
- Guess p of 0.5, margin 0.10: about 96 findings.
- Guess p of 0.5, margin 0.05: about 385 findings.
- Guess p of 0.2, margin 0.05: about 246 findings.
Notice the second and third. If you already believe most findings are real, you need fewer samples for the same margin, because there is less uncertainty to pin down.
The small list correction
The formula assumes a very large population. When the list is short, you can sample fewer. The adjusted size is n divided by (1 plus (n minus 1) divided by N), where N is the total number of findings.
With N of 500:
- 385 becomes about 218.
- 96 becomes about 81.
With N of 100, you would review nearly everything to get a tight margin. At that size just triage them all.
Sampling rules that keep the answer honest
- Pick at random. Number the findings and use a random generator, or sort by a random key.
- Sample within groups if you care about them. Sample rules, languages or severities separately if you want a rate for each, because the blended number can hide a noisy rule.
- Deduplicate first. The same issue repeated in 40 places is one decision.
- Have the same person or the same written definition label every item. Inconsistent labelling adds error the formula does not account for.
- Record unknowns. Items nobody could judge should be counted and reported, not silently dropped.
When 30 items is still useful
A 30 item sample cannot give you a tight number, but it can rule things out. If 25 of 30 are false positives, the rate is very unlikely to be below 60 percent, which is enough to decide a tool needs tuning before a wider rollout. If you saw zero false positives in 30, read our post on the rule of three before celebrating.
Reporting the result
Say the sample size and the margin together: "about 40 percent false positives, plus or minus 10 points, from 96 reviewed findings." A bare percentage invites readers to assume more precision than you have.
Quick check
Do I need the whole backlog reviewed to know the false positive rate?
No. A random sample of a few hundred is enough for a margin of about 5 points, regardless of how big the backlog is, as long as the sample is truly random.
Is a bigger sample always better?
Only if labelling stays consistent. Past a few hundred items, tired reviewers add more error than extra items remove.
Last reviewed: 19 October 2026.
