An SCA proof of concept should end on criteria written before it starts. Use four groups: coverage of your real repositories, signal quality of the findings, fit with developer workflow, and operational basics such as access and export. Give each a pass or fail line and a named person who decides, so the result is a decision rather than an impression.
Why write them first
Without criteria, a trial ends with whoever argues loudest. Vendors will also steer you toward the repositories where their tool looks strongest. Fixing the test set and the pass lines in advance protects you from both problems.
Keep the trial short. One to two weeks is usually enough for SCA, because the questions are mostly about coverage and signal.
A starting table
| Group | Check | Pass looks like |
|---|---|---|
| Coverage | Every ecosystem in the test repositories is parsed | No repository returns an empty or partial inventory without an explanation |
| Coverage | Transitive dependencies are listed | The tool's tree matches your lockfile for a sample of ten packages |
| Signal | Findings are traceable | Each of twenty sampled findings names the advisory, affected range, and fixed version |
| Signal | Noise is manageable | Reviewers mark most sampled findings as relevant or have a way to filter the rest |
| Workflow | Pull request feedback works | A seeded vulnerable dependency is flagged on a pull request within your agreed time |
| Workflow | Fix guidance is usable | At least some fix suggestions apply without a breaking change |
| Operations | Roles and single sign-on work | Read-only and developer roles behave as documented |
| Operations | Data leaves cleanly | Inventory and findings export in an open format |
Adjust the thresholds to your own tolerance, and write the numbers down before the trial.
Build the test set
Choose repositories deliberately.
- One healthy, well-maintained service.
- One legacy repository with messy build files.
- One polyglot repository.
- One repository with a seeded vulnerable dependency you planted, so you know the right answer.
The planted case is the most informative. You know whether the tool should find it, so you can measure a miss.
Decide who judges
Name a single decision owner, and have two developers score the workflow items. Ask them to log friction in a shared note during the week, not afterward.
Close the trial properly
At the end, fill in the table, record any criteria that were unclear, and write a one paragraph recommendation. Then revoke trial credentials and delete test data on the schedule the vendor commits to in writing.
A worked example: filling in the table
Imagine a two week trial with four repositories. At the end of week two, the team fills in results.
| Check | Threshold written in advance | Result | Pass |
|---|---|---|---|
| Ecosystems parsed | All four repositories return a non-empty inventory | Three did, the fourth missed a nested module | Fail, vendor asked to explain |
| Lockfile match | Ten sampled packages match the lockfile | Ten of ten | Pass |
| Traceable findings | Twenty of twenty sampled findings name advisory, range, and fix | Nineteen | Pass with note |
| Seeded vulnerability | Planted dependency flagged on a pull request within the agreed time | Flagged | Pass |
| Roles | Read-only role cannot dismiss findings | Could dismiss | Fail |
The result is mixed, and the table makes the discussion concrete. The vendor can answer the two failures, the team can re-test, and the decision owner can see exactly what is outstanding.
Step by step: run the two weeks
- Day 0: agree the table, the repositories, and the decision owner. Send the table to the vendor.
- Day 1 to 2: connect the repositories with read-only access and run the first scans. Record any setup friction.
- Day 3 to 5: sample findings and trace them. Plant the seeded dependency in a pull request.
- Day 6 to 8: have two developers work with the tool in normal tasks and log friction as it happens.
- Day 9 to 10: test roles, export, and the data handling questions.
- Day 11: fill in the table and list open questions for the vendor.
- Day 12 to 13: re-test whatever the vendor fixed or clarified.
- Day 14: decision meeting, then clean up.
Questions to settle before you start
Who owns the trial data, and when will it be deleted? What credentials does the trial need? Is the trial environment the same as production, or reduced? A reduced trial can hide limits you will meet later.
Beyond the table
Note qualitative items too. How fast did support answer? Was documentation accurate? Did the vendor admit limits plainly? These do not belong in pass or fail columns, but they matter at renewal time.
Make the decision reversible
Even with a good trial, start production use in a few repositories and review after a month. Keep your own export of inventories and decision records so a later change of tools does not lose history.
Avoid common mistakes
- Judging a tool on a repository the vendor chose.
- Changing thresholds after seeing the results.
- Letting the trial run so long that nobody remembers the original goals.
- Leaving test accounts, tokens, and connections active afterward.
Try it yourself
Click through it at your own pace, or open it full screen.
Use it to picture the pull request workflow check, then decide what time limit and block rule you would write into the table.
Quick check
How many repositories should a proof of concept include?
Four to six chosen for variety is enough. More adds noise without adding information.
Should the vendor run the trial for us?
Let them help with setup, but have your own staff run scans and judge results, so you see the day to day experience.
Can a proof of concept use a copy of production code?
Only if your data rules allow it. Check the vendor's data handling terms first, prefer read-only access, and delete the data at the end.
Last reviewed: 19 October 2026.
