At the end of a pilot, ask developers five short questions: would you keep it on, was it clear what to do, how often was it wrong, how did it affect your speed, and what would you change. Collect answers anonymously, score them simply, and weigh them next to your findings data.
Why survey at all
Numbers about findings tell you how a tool behaves. They do not tell you whether people will live with it. A tool that blocks merges on results developers distrust gets disabled, ignored or worked around. A short survey surfaces that early, while switching is still cheap.
Keep it to five questions. Longer surveys get fewer answers and more guesswork.
The questions
Use a scale from one to five for the first four, and free text for the fifth.
- Would you want this check on every pull request you open? This is the headline question.
- When it flagged something, did you know what to do next? Measures clarity of messages and fix guidance.
- How often did a result turn out to be wrong or irrelevant? Ask for a rough share, such as almost never, sometimes, often.
- How did it affect your time to merge? Waiting for scans and chasing results both count.
- What is the single thing you would change? Free text. Read every answer.
How to run it
Send it to everyone who touched the pilot, not just the volunteers. Make it anonymous or at least confidential. Give it a deadline of two days, and send one reminder.
Aim for a response from most participants. With a small pilot group of five to ten people, even a handful of honest responses is informative. Do not pretend a small sample gives you a precise percentage. Report counts, such as six of eight said yes.
How to weigh it
Put the survey beside your measured data: share of findings labelled real, triage time, setup effort. Then use a simple rule.
- If data looks good and developers say yes, proceed.
- If data looks good and developers say no, find out why. Often it is placement or message clarity, which a vendor can sometimes fix.
- If data looks poor and developers say yes, be careful. Enthusiasm for a new tool fades. Check whether the tool is quiet because it finds little.
- If both are poor, stop.
A strong no from developers on question 1 should carry weight even when the numbers look fine. You will be asking those people to live with the tool for years.
Reading the free text
Group answers into themes: speed, noise, clarity, placement, other. If three people independently mention the same pain, treat it as a requirement and put it in the vendor follow up list.
Pitfalls
Do not lead the question. Ask whether they would keep it on, not whether they agree it makes the code safer. Do not run the survey only with the security team, who like the tool by default. And do not let a manager attend the feedback session, which tends to silence criticism.
Quick check
How many developers should answer?
Most of those who used the tool. In a small pilot, report raw counts instead of percentages.
Can a survey overrule the data?
It can override a marginal data result. Treat a clear developer no as a serious risk to adoption either way.
Safeguard's policies and gates can be switched on in stages, and the free scan is at gold.safeguard.sh/scan.
Last reviewed: 19 October 2026.
