Aim for three or four vendors on the final shortlist, cut from a long list of eight to twelve. Fewer than three leaves you without comparison or negotiating room. More than four spreads your testing time so thin that you end up judging brochures instead of evidence.
Why three or four
The limiting resource is not vendor patience. It is your own engineers' time. A fair hands-on test needs the same repos, the same people and the same week for every candidate. Each extra vendor adds setup, calls, support questions and a scoring session.
A rough budget helps. If a proper test of one tool needs a few hours from two engineers plus a couple of hours from a security reviewer, multiply that by the number of vendors and see whether it fits the calendar you actually have. That is a planning exercise, not a benchmark, so use your own numbers.
The funnel
| Stage | Typical size | What you do |
|---|---|---|
| Long list | 8 to 12 | Collect names from peers, standards bodies, your cloud marketplace and your developers |
| RFI cut | 5 or 6 | Short written questions, drop anyone who fails a deal breaker |
| Shortlist | 3 or 4 | Full RFP, demo on your terms, reference calls |
| Hands-on test | 2 or 3 | Run on your own repos with agreed success criteria |
| Decision | 1, with a runner-up | Score, document, negotiate |
These sizes are a starting point, not a rule. A narrow category with few credible options may give you a long list of five. A crowded category may need twelve.
How to cut without politics
- Write the deal breakers first. Deployment model, supported ecosystems, data residency, single sign-on. If a vendor fails one, they are out, and nobody has to argue about it.
- Cut on evidence you can check. Public documentation, a free trial, a sample report. Avoid cutting on logo recognition.
- Keep one outsider. Include one candidate that is not the usual name. It tests your assumptions and sharpens the incumbent's answers.
- Keep the incumbent honest. If you already own a tool, include it in the process. Otherwise you cannot tell whether switching is worth the effort.
- Record why each name left. One line each. It saves you from relitigating in six months.
When to go bigger or smaller
- Go to five if two of your four are close on paper and you want a tie-breaker, and you can afford the extra week.
- Go to two if a hard requirement, such as an on-premises deployment, removes most of the market.
- Go to one only for a renewal or an extension of something already proven in your environment, and write down why.
A worked example: cutting twelve to three
A platform team starts with twelve names. Here is how a realistic cut might go, with every step written down.
| Step | Names left | Cut reason, in one line |
|---|---|---|
| Long list | 12 | Collected from peers, a marketplace and developer suggestions |
| Deal breakers | 9 | Two lack support for the main build tool, one cannot run in the required region |
| RFI answers | 6 | Three could not label capabilities as generally available or roadmap, or declined to answer in writing |
| Environment fit | 4 | Two want a heavy agent on build servers, which the team will not allow |
| Final shortlist | 4 | Includes the incumbent and one unfamiliar candidate |
| After the test | 2 | Two fall behind on triage effort and pull request behavior |
Notice that no step required opinions about vendor quality. Each cut came from a fact or an evidence gap, which makes it easy to explain to a vendor and easy to defend to a manager.
What a hands-on test costs per vendor
Rather than guess, build a simple time budget before you commit to a number. List the tasks, estimate hours for each, and multiply.
- Connect one repository and run the first scan.
- Review a sample of findings with the code owners.
- Open a test pull request and watch the check.
- Export data and check it.
- Raise one support question.
- Score and write notes.
If those six tasks take about two engineers a day or two each, four vendors take most of a working week for the pair, plus calls. If you only have a few engineer days to spend, shrink the list to three. The right size is the one your calendar can support without cutting corners on the test.
Signals that your shortlist is wrong
- All finalists look alike. If the four are near copies, you may be missing a different approach, such as a build-time tool versus a platform.
- Nobody on the team has heard of two of them. That might be fine, but make sure you have checked basic facts such as maintenance history and support terms.
- The incumbent is missing. Add it, or write down why a switch is not on the table.
- A vendor was added late by a senior sponsor. Run them through the same steps, or the process loses credibility.
Communicating the cut
Tell cut vendors the outcome briefly and politely, and offer a one line reason if they ask. Do not share other vendors' names or scores. Keep relationships civil, because you may need them in a future renewal.
Try it yourself
Click through it at your own pace, or open it full screen. Look at what a dependency inventory shows, then ask which shortlisted tools could produce the same view from your own repos.
Quick check
Is a shortlist of two enough?
It can be, if a hard requirement removes the rest. Otherwise two gives you little comparison and little negotiating room.
Should the incumbent be on the list?
Yes, as a fair candidate. Without it you cannot tell whether switching costs are justified.
What if a senior sponsor insists on a fifth vendor?
Add it, but run it through the same steps and the same test. If time is short, drop the weakest candidate by documented criteria, not by politics.
Safeguard can be one of the candidates if dependency scanning, reachability analysis, SBOMs, or policies and gates are on your list. Judge it by the same test as everyone else, and start with the free scan at https://gold.safeguard.sh/scan if you want a quick look.
Last reviewed: 19 October 2026.
