Send every vendor the same demo script before the call, ask them to run it on a repository you supply, and score it live. A scripted demo takes about 90 minutes, removes the polish gap between vendors, and shows how the tool behaves when it is not choosing the example.
Why standard demos mislead
A default demo uses a sample app chosen because the tool handles it well. Findings are tidy, fixes apply cleanly and nothing times out. That is not a lie, it is just not your environment. Your repos have old build files, odd monorepo layouts and private packages.
You cannot fix this by asking tougher questions in the room. You fix it by controlling the inputs.
Prepare the script
Write it as a list of tasks, each with a pass condition. Keep it under ten tasks. Examples:
- Connect a repository using your identity provider login, no vendor staff holding admin access.
- Scan the supplied repo and show the first result time.
- Open one critical finding and show how the tool explains why it matters in this code.
- Mark a finding as accepted risk with a reason and an expiry, then show the audit trail.
- Show a blocking rule in a pull request, then show how a developer gets past a false alarm.
- Export results in an open format.
- Remove a user and show what access they lose.
Share the script at least three days ahead. Vendors who ask to skip a task have told you something.
Supply the code
Choose a repo that is small enough to scan during the call but has real mess in it. Strip secrets and anything that identifies a customer. See how to choose the test corpus for what to include.
If you cannot share code, ask the vendor to scan a public project you name on the day.
Run the call
- Have two people take notes, one on results and one on friction.
- Ask the vendor to share their screen but let your own engineer say what to click next.
- Time each task. Write down the time, not an impression.
- Mark "shown," "described only" or "not shown" for each item. Described only is not a pass.
- Park roadmap talk. Anything beta or planned is recorded as not available today.
After the call
Score within a day while memory is fresh. Use the same 0 to 4 anchors for every vendor, for example 0 means not shown and 4 means shown on your code with no help. Score anchors keep two evaluators from drifting.
Send the vendor a short list of open items with a deadline. Anything not answered in writing stays at its current score.
Try it yourself
Click through it at your own pace, or open it full screen.
Click through the pull request flow and note what a developer sees when a check blocks a merge. That is the same moment you want to watch in every vendor demo.
Quick check
Should procurement attend?
Yes for the commercial section, but keep it separate. Mixing pricing into the technical demo skews scoring.
What if the vendor refuses a script?
Treat it as a finding. A tool that only works on prepared examples will cost you more in rollout than a late start.
Last reviewed: 19 October 2026.
