Add a one click way to mark each finding useful or not useful, record the answer by rule, and review it monthly. Developers already express opinion by dismissing or ignoring comments. A signal captures that opinion cheaply and tells you which rules deserve to be tuned or removed.
Why a signal beats guessing
Without feedback, the only data you have is volume. High volume could mean a risky codebase or a noisy rule. A useful or not useful count per rule separates the two and gives you a defensible reason to change a rule.
It also changes how developers feel. A bot that asks for feedback and then acts on it is a colleague. One that does not is a tax.
Ways to collect it
| Method | Effort for developer | Data quality |
|---|---|---|
| Reaction on the comment (thumbs up or down) | One click | Good if the bot reads reactions |
| Dismissal with a required reason | A few seconds | Richest, includes why |
| Link to a short form in the comment | Medium | Detailed, low response |
| Quarterly survey | Medium | Broad, not per finding |
Combine the first two. A reaction is easy. A dismissal reason is rich. Together they cover most of what you need.
Define the dismissal reasons
Keep the list short and meaningful so that the data can be counted:
- False positive, the finding is wrong.
- Not reachable or not exploitable in our use.
- Test or example code.
- Accepted risk, with expiry.
- Will fix later, with ticket.
Free text can sit beside the choice, but the choice is what you will count. Our post on writing a dismissal comment an auditor will accept covers the free text.
What to measure
For each rule, per month: findings shown, fixes merged, dismissals by reason, and useful versus not useful reactions. Two ratios do most of the work.
- Fix rate. Findings fixed divided by findings shown. Low means people do not believe it or cannot act on it.
- False positive share. Dismissals marked false positive divided by findings shown. High means the rule needs tuning.
Do not set a universal target. Rules differ, and a rule with a low fix rate but high severity may still be worth keeping.
Act on it visibly
Review the bottom five rules each month. For each, choose one action: tune the pattern, lower the severity, move it to informational, or retire it. Post a short changelog where developers will see it. When people see their clicks cause changes, they click more.
Guard against bias
A signal from a vocal few can skew a decision. Look at counts across teams, and treat a single loud thread as a prompt to check the data, not as the data. Also remember that "not useful" can mean "I did not understand it", which is a wording problem, not a rule problem.
Safeguard reachability analysis helps cut findings that are not exploitable before they reach a comment. Try a free scan to see how many findings change after that filter.
Quick check
Should the signal be anonymous?
Usually yes at the report level. Aggregate by rule and team, and keep individual names out of the dashboards.
How many responses do I need?
There is no fixed threshold. Look for a rule with several dismissals for the same reason before changing it.
Last reviewed: 19 October 2026.
