Test a secrets scanner by building a small repository seeded with fake credentials you control, then scoring what it finds, misses and wrongly flags, on both current files and Git history. Add one real repo for noise. Two weeks is enough if you decide the pass marks before the first scan.
Why does a demo not tell you enough?
Vendor demos use repos where the tool shines. You want to know how it behaves on your code: your languages, your fake-looking test fixtures, your odd internal tokens. A trial that measures only "did it find the AWS key in the README" tells you almost nothing.
The failure modes that cost real time are two. A miss leaves a live credential in history. Noise buries the alerts so people stop reading them. You want both numbers.
How do you build the seeded repo?
Create a throwaway repository and plant fake secrets. Never use live credentials. Include these cases, one per file or commit, and record each in a scoring sheet with a file path and commit id:
- A well-known provider key format with a fake value.
- A private key header block with junk content.
- A connection string with a fake password.
- A high-entropy string in a config file with no recognizable format.
- An internal token pattern your company uses.
- A secret added in commit one and deleted in commit two, so it lives only in history.
- A secret inside a base64 blob, a test fixture and a Markdown code sample.
- A harmless long hash and a UUID, which should not be flagged.
OWASP's Secrets Management Cheat Sheet notes there are many approaches to detection and that open source tools exist, so a trial can also include one open source scanner as a baseline.
What do you measure?
Score every planted item as found, missed or flagged-with-wrong-severity. Then count alerts on the harmless items. For each tool, write down:
- Recall on planted secrets, current files and history separately.
- Noise: false alerts per thousand files on your real repo.
- Whether it can confirm a secret is live. Some tools make a low-privilege call to the issuer, and a few document that. Check whether this can be disabled for air-gapped use.
- Where it runs: editor, pre-commit, CI, platform. Note any surface that does not exist yet. For example, Safeguard's docs say there is no CLI command to install a git hook yet, and that pre-push prevention comes from the IDE extension.
- Whether secret values are redacted in reports and logs.
- Whether allowlists are audited and scoped.
What does week two add?
Week one is correctness. Week two tests daily life. Run the tool in a pull request pipeline and measure scan time. Have two developers work with it and ask what annoyed them. Trigger a real workflow: find a planted secret, rotate it, and see whether the tool helps you close the finding. The OWASP cheat sheet stresses rotation and revocation, so a tool that detects but gives no path to remediation still leaves you the work.
What are the honest limits?
No scanner finds every secret. Pattern matching misses unusual formats, entropy checks flag harmless strings, and verification needs outbound access. Treat any vendor claim of zero false positives as a statement to test. Also remember that detection after the fact does not undo exposure. If a real key was pushed, rotate it.
What to do
- Decide pass marks first: minimum recall, maximum noise, scan time limit.
- Build the seeded repo and the scoring sheet before you install anything.
- Include at least one history-only secret and two decoys.
- Scan your own largest real repo and sample 50 alerts by hand.
- Run each tool in CI for a week and note failures.
- Test the remediation path on one planted secret.
- Score all tools against the same sheet and attach it to the decision record.
See what a scan finding looks like before you plan your trial.
Quick check
Can I use real secrets in the trial?
No. Use fake values that look real. A real credential in a test repo is a leak.
How many seeded secrets are enough?
Twenty to thirty varied cases is a practical size for a trial. That is a judgment call, not a standard. The variety of formats matters more than the count.
Sources
Last reviewed: 18 October 2026.
