Put five things in writing before an outside evaluator starts: scope, independence terms, data handling, the method and who may publish the results. Agree the success criteria first, give every candidate the same access, and keep decision rights with your own team.
1. Scope
State which tools, versions and use cases are in scope. Name the repositories or environments the evaluator may use, and which are off limits. Say what is not being judged, such as pricing or support quality, if you want a narrow technical result.
2. Independence terms
Ask the evaluator to disclose any financial or personal relationship with any candidate. The FTC describes material connections as including personal, family, employment or financial relationships, and says anything of value counts. Use that as your definition. Add a clause that the evaluator will not accept free licenses, travel or gifts from candidates during the work.
3. Data handling
Decide what code or data the evaluator may touch, where results are stored, how long they keep it, and who can see it. A mutual NDA is common for tool tests, and our post on an NDA for a security tool evaluation covers what to include. If the code is sensitive, run tests on a copy or a sample.
4. Method
Agree how ground truth will be set, how false positives will be judged, how many runs are done and what is reported. Ask for the labeling to be done without seeing which tool produced each finding. Require raw outputs to be kept so you can recheck.
5. Publication rights
Decide whether the report is private, shared with candidates or published. If candidates will see it, decide whether they get a chance to correct factual errors, and make sure that right does not become a right to change conclusions. If it will be published, the evaluator should disclose who paid.
Give candidates the same treatment
Offer every candidate the same access, the same time and the same briefing. Tell them the method up front. Do not let one candidate see test inputs that others have not seen.
After the report
Read the method section first, then the numbers, then the conclusions. Ask the evaluator to walk through three findings in detail. If the walk through is vague, the report is weaker than it looks.
Try it yourself
Click through it at your own pace, or open it full screen.
Notice which fields an outside evaluator would need to see to judge a finding, and decide whether they would need access to them.
Where Safeguard fits
Safeguard offers SBOM and dependency scanning, policy gates, and compliance and evidence automation. A free scan at gold.safeguard.sh/scan is available if you want to include it in a test.
Quick check
Should vendors review the evaluator's draft?
They can check facts about their own product. They should not edit conclusions, and you should keep a record of any corrections.
Sources
Last reviewed: 19 October 2026.
