A real pen test report shows what the tester did, which findings were verified by exploitation, and how weaknesses chained together. A relabeled scan shows long lists of generic findings with scanner wording and no proof. Check for evidence, narrative and manual work before you accept the report.
Why this matters
NIST SP 800-115 draws the line clearly. Vulnerability scanners check only for the possible existence of a vulnerability, while the attack phase of a penetration test exploits it to confirm. NIST also notes scanners have high false positive rates and findings need validation. A report with neither validation nor exploitation is a scan.
Seven signs
| Sign | Real pen test | Relabeled scan |
|---|---|---|
| Evidence | Request and response, screenshots, steps to reproduce | Plugin names and generic descriptions |
| Verification | States which findings were exploited | No distinction between possible and proven |
| Chaining | Describes how issues combined | Flat list sorted by score |
| Context | Explains impact in your system | Copied advisory text |
| Scope detail | Lists what was tested and what was not | Just a target list |
| Manual work | Mentions business logic, access control checks | Mostly network and version findings |
| False positives | Removed or flagged | Left for you to sort out |
Questions to ask the provider
- Which findings did you verify by exploitation, and how?
- What did you test manually that a scanner would not?
- Can I see an anonymized example of a finding with its evidence?
- Which tools did you use, and which were scanners?
- How did you test authorization between roles or tenants?
- What was out of scope or untested, and why?
Also see questions to ask a penetration testing provider before you sign.
A scan is not bad
An automated scan is a legitimate and useful product, if it is sold as one. The problem is a mismatch between what the buyer expects and what is delivered. NIST SP 800-53 describes penetration testing as going beyond automated vulnerability scanning and conducted by people with demonstrable skills, so ask for exactly that.
What to do if you got a scan
- Ask the provider for the manual testing notes.
- Request a corrected report or a credit toward real testing.
- Do not forward it to customers as a pen test.
- Run your own scanner regularly so you are not paying for scanner output at pen test rates.
Quick check
Can a good pen test still include scanner output?
Yes. Testers use scanners in the discovery phase. The sign of a real test is verification, manual testing and a narrative, not the absence of tools.
What is the quickest check?
Look at one high severity finding. If it has reproducible steps and evidence of exploitation, the work is likely manual. If it reads like an advisory, it is probably scanner output.
Safeguard handles the automated side so you do not pay a tester for it: SBOM and dependency scanning, reachability analysis and policy gates, plus a free scan.
Sources
Last reviewed: 19 October 2026.
