Run a two week trial on a repository you control, seeded with known misconfigurations, and score each scanner on what it catches, how much noise it adds, how clear the fix is, and where your data goes. Decide the pass mark before you start, so a demo cannot move the goalposts.
Why trial instead of reading feature lists?
Feature lists say a tool supports Terraform or Kubernetes. They do not say how it behaves on your code, your modules and your pipeline. A trial answers the questions that decide adoption: does it find what matters, will developers tolerate its output, and does it fit your build.
Describe the product category when you ask vendors for access. You are comparing scanners for infrastructure definitions, not particular brands.
Step 1: build the test repository
Create a small repository with three kinds of content.
- Known bad. Ten to twenty misconfigurations you planted on purpose: open network rules, unencrypted storage, overly broad permissions, public buckets. Write down each one and its expected severity before the trial.
- Known good. Several correct resources and modules that a scanner should leave alone. These measure false alarms.
- Real code. A copy of one of your actual modules, with its real quirks, because planted examples are always cleaner than real life.
Never place real secrets in the test repo. HashiCorp's documentation warns that if secret values are in the configuration, Terraform stores them in its state and plan files, which can contain sensitive values such as database passwords or API tokens, and that anyone with access to those files can read them. A scanner that reads plan output deserves the same care as the state itself.
Step 2: test the inputs it accepts
Scanners differ in what they read. Check each against your actual workflow.
- Raw configuration files in the repository.
- A rendered plan. Terraform can emit machine readable output with
terraform show -json, which shows the plan, configuration and current state, and requires Terraform 0.12 or later. HashiCorp notes that sensitive values appear in plain text in that output. - Modules pulled from remote sources.
- Your other formats, such as Kubernetes manifests or Helm charts.
A tool that reads just raw files may miss values computed at plan time. A tool that needs a plan needs credentials or state access in the pipeline, which is a security decision in itself.
Step 3: score what matters
Use a sheet with these columns, filled per scanner.
| Measure | How to score |
|---|---|
| Catch rate | Planted issues found, divided by planted issues |
| False alarms | Findings on the known good resources |
| Clarity | Does each finding name the file, line and fix? Rate 1 to 5 |
| Severity sense | Do severities match your pre written expectations? |
| Speed | Minutes on the real module, locally and in CI |
| Suppression | Can you accept a risk with a reason, expiry and owner? |
| Output formats | SARIF supported? |
| Data path | Does the code or plan leave your environment? |
SARIF is worth asking about. The OASIS specification defines it as a standard format for the output of static analysis tools, so a tool that emits it can feed code scanning views and other tooling without custom work.
Step 4: test the workflow, not just the scan
In week two, move from lab to life. Run the scanner in a pull request on a branch with one planted mistake. Check that the result appears where developers look, that a fix can be suggested or applied, and that a suppression requires a reason. Ask two developers to use it and note where they get stuck.
What are the honest limits of a trial?
A planted test measures known patterns and cannot show what you do not know to plant. Two weeks will not reveal long term upkeep, such as how quickly new cloud services are covered. Ask vendors for their update cadence in writing, and check release notes for evidence.
What to do
- Write the pass mark first: a minimum catch rate, a ceiling on false alarms, and a data path rule.
- Build the test repo with known bad, known good and one real module.
- Confirm no real secrets are in the repo or any plan file.
- Run each scanner on identical inputs and fill in the sheet.
- Run the strongest candidate in a real pull request, with two developers.
- Ask for update cadence, support terms and data handling in writing.
- Decide against the pass mark, and keep the sheet.
Quick check
How many scanners should be in the trial?
Two or three. More than that dilutes attention, and the score sheet makes a fair comparison harder to complete.
Should I trial with production code?
Use a copy of one module, with secrets removed. Do not give a trial tool access to production credentials or live state.
Sources
- HashiCorp: terraform show, fetched 18 October 2026
- HashiCorp: Manage sensitive data in Terraform, fetched 18 October 2026
- OASIS: SARIF v2.1.0, fetched 18 October 2026
Last reviewed: 18 October 2026.
