Have two or three developers triage the same random sample from each scanner while a timer runs, and record minutes to a decision for each finding. Compare the median time and the share that needed escalation. Raw counts reward noisy tools, while triage time shows the real cost.
Why counts mislead
A scanner that reports 800 findings is not twice as good as one that reports 400. If half of the 800 are duplicates or irrelevant, the team spends time on rows that carry no risk. The cost of a tool is closer to the sum of time spent deciding than the number of rows. Related ideas appear in measuring scanner noise in a trial.
Define a decision
Agree what counts as done. A triage decision is one of:
- Fix now.
- Fix later, with a ticket.
- Not applicable, with a reason.
- Need help, escalate.
The clock starts when the developer opens the finding and stops when they record one of these. Writing the reason counts. Actually fixing the issue does not, because that belongs in a separate fix test.
Set up the sample
- Draw 20 findings at random from each scanner's output on the same repositories. Use a fixed random seed or a documented method.
- Remove tool names and branding where possible. Blind triage reduces favoritism.
- Mix the order so reviewers do not see one tool's findings in a block.
- Give reviewers the same instructions: decide using whatever the finding shows, plus normal tools such as your code editor.
Use at least two reviewers, and let some of the sample overlap so you can compare their decisions.
What to record
| Field | Why |
|---|---|
| Minutes to decision | The main measure |
| Decision | Fix now, later, not applicable, escalate |
| Needed outside lookup? | Shows missing context in the tool |
| Confidence of reviewer | Low confidence hints at unclear evidence |
| Same decision as second reviewer? | Shows how clear the finding is |
Use a simple stopwatch or a timestamped form. Do not ask reviewers to report time from memory.
How to compare
Use the median, not the mean, because a few long investigations skew an average. Report the middle half of the range as well, so readers see the spread. Then compare:
- Median minutes per finding for each scanner.
- Share escalated or marked low confidence.
- Share of identical findings, by overlap, where reviewers disagreed.
- Share that needed an outside lookup, meaning the tool did not show enough.
Then combine with volume for a total estimate. Multiply the median minutes by the number of findings that adjudication confirmed as real, and add the time for those that were not real. A scanner with fewer, clearer findings can cost far less than one with more.
Cautions
Short is not always better. A quick decision on a finding with thin evidence may be a wrong decision. Pair time with a check of correctness on a subset. Reviewer experience also matters: a senior engineer is faster than a new hire, so use the same people for every tool.
Reviewers also learn the repositories as they go, so later findings get faster. Rotate which tool's findings come first.
Try it yourself
Click through it at your own pace, or open it full screen. Notice what context sits next to each finding, and how that would change your time to decide.
Safeguard shows reachability and fix information alongside findings, which is context a triage clock is meant to reveal. Try the free scan at gold.safeguard.sh/scan.
Quick check
How many reviewers do I need?
Two is the minimum, so you can see whether decisions agree. Three gives a tiebreak.
Should the vendor demo the triage?
No. The point is a developer's unaided time to decision.
Last reviewed: 19 October 2026.
