Run a SAST pilot on three to five of your own repositories for two weeks, decide your pass criteria first, and measure three things: how many findings a developer would act on, how long triage takes, and whether results land where developers already work. Demo apps tell you very little.
Why a pilot beats a demo
A vendor demo runs on a sample application built to make the tool look good. Your code has its own frameworks, odd build steps and years of history. OWASP's guidance on source code analysis tools lists language support, framework understanding, build requirements and workflow fit as selection criteria. Every one of those is easier to judge on your repositories than on a slide.
Week 0: set the rules before you start
Write down the pass criteria before anyone opens a dashboard. Otherwise the pilot turns into a vote on whichever tool had the nicest interface.
Pick the repositories. Choose one large service in your main language, one small service in a second language, one repository with a messy build, and, if you have it, one older codebase nobody enjoys touching. Include at least one repository where you already know of past security bugs.
Pick two or three people to triage. Name them now. A pilot with no owner collects dust.
Week 1: connect and scan
Wire the tool into the places developers work. That usually means a pull request check and a scheduled full scan. Record the effort: how long did setup take, did anyone need vendor help, did the build need changes?
OWASP notes that many static tools struggle with code that cannot be compiled, because they need the right libraries, build instructions and complete code. If your messy repository fails here, that is a result, not an inconvenience.
Keep a simple log. For each finding, the triager writes one of four labels: real and worth fixing, real but low value, not real, unclear. That log is your data.
Week 2: measure what matters
Count the labels. A tool that produces 400 findings where 20 are real and worth fixing may still be useful, but only if the triage cost is acceptable. Time a few sample triages with a stopwatch.
Check the known bugs. If you seeded the pilot with past vulnerabilities, how many did the tool catch? This is a small sample, so treat it as a smoke test. It will not give you a detection rate.
Ask developers one question: would you want this on every pull request? Their answer matters more than a feature table, because a tool people mute protects nothing.
What to decide at the end
Score each tool on the same sheet: setup effort, share of findings labeled real, triage minutes per finding, known bugs caught, developer verdict, and export format support. OWASP also recommends checking for standard output such as SARIF so results can move between tools.
If two tools tie, pick the one with the lower triage cost. Findings nobody reads have no value.
Safeguard covers the open source dependency side with SBOM and dependency scanning and reachability analysis, which is a separate question from first party code analysis. You can try a free dependency check at gold.safeguard.sh/scan.
Quick check
How many repositories should a SAST pilot cover?
Three to five is enough. Choose them to differ in language, size and build complexity rather than picking the easiest ones.
Should I use the vendor's sample app?
Use it only to confirm the tool runs. Judge accuracy and effort on your own repositories.
Sources
Last reviewed: 19 October 2026.
