Run the pilot in four weeks: define success in week one, install and baseline in week two, test accuracy and workflow in week three, and decide in week four. Fix the repositories and success measures before you start, so the result cannot be bent by whoever likes the demo.
Week 1: set the ground rules
- Pick three to five repositories: one large, one with two ecosystems, one monorepo if you have it, one old and neglected.
- Name the people involved: an engineer, a security reviewer, and someone who owns the budget.
- Write the success measures. Examples: coverage of your manifests, share of reviewed findings that are correct, minutes added to CI, and time to triage a typical finding.
- Write the gates, such as data handling rules that disqualify a tool.
Reuse the weights from the scorecard guide.
Week 2: install and baseline
Connect the tool to the chosen repositories. Record the first results without tuning anything. Note:
- Which repositories it could and could not read
- Total findings by severity
- Setup problems and how long each took
The untuned baseline matters, because it is what developers will see on day one.
Week 3: test accuracy and workflow
Run the checks that vendors rarely show you:
- Review a sample of fifty findings by hand, as in measuring false positives
- Run the five package test from the transitive dependency guide
- Put the tool in a CI test branch and break it on purpose
- Ask two developers to fix a finding using only what the tool tells them, and time it
Week 4: decide
Fill the scorecard from evidence only. Hold a short meeting and decide one of three things: adopt, extend the pilot on a named gap, or reject. Write down why.
If you adopt, plan the rollout: one team first, then more, with a rule for who can accept risk.
Pilot traps
- Testing only the repository the vendor suggests
- Tuning rules during week two, then comparing with an untuned rival
- Counting findings as value. More findings is not better
- Skipping developers, who will decide whether the tool lives
What to keep
Store the repositories, commit hashes, exports, and scorecard. When the contract renews, you can rerun the same test.
Safeguard is a reasonable candidate for the pilot list: SBOM and dependency scanning with reachability analysis, and a free scan at https://gold.safeguard.sh/scan to get a first look.
Who decides at the end
Name the decision maker in week one, not week four. If the budget owner joins only at the end, the pilot turns into a presentation. Give that person a one page summary: the scorecard, three findings that mattered, the cost model in plain words, and the main risk of each option. Keep the summary short enough to read in five minutes. A clear, brief document makes a yes or a no far easier, and either is a good outcome of a pilot.
Quick check
Can the pilot be shorter than 30 days?
Yes for one small repository. Allow the full time if you need workflow and CI testing across teams.
Last reviewed: 19 October 2026.
