Run ten pass or fail tests on an API of your own: import your spec, stay logged in, catch a planted object access flaw, catch a planted role flaw, handle write methods safely, report clearly, run in CI, and export evidence. A demo on someone else's sample app proves very little.
Set up before you start
Pick one internal API with known quirks. Plant two flaws on purpose in a test environment: one endpoint that skips the ownership check, and one admin function that a normal user can call. Create the accounts described in OWASP's testing guidance: two ordinary users that own different objects, plus a lower role and a higher role. Without planted flaws you cannot tell a good tool from a quiet one.
The ten tests
| # | Test | Pass looks like |
|---|---|---|
| 1 | Import your spec, including an old or incomplete one | Operations appear, and the tool tells you what it could not parse |
| 2 | Compare the imported list with traffic you know exists | Gaps are reported, not hidden |
| 3 | Authenticate and stay authenticated through token expiry | The tool refreshes, or tells you it lost the session |
| 4 | Find the planted ownership flaw using two users | The finding names both accounts and the object |
| 5 | Find the planted role flaw | The finding names the role and the function |
| 6 | Run in a mode that sends no write methods | The tool states which checks it skipped |
| 7 | Read three findings cold | Each has a request, a response and a plain explanation |
| 8 | Scan twice and compare | Repeat findings are recognized, not duplicated |
| 9 | Run it from a CI job and trigger a failure | The exit status is documented and you can tune what fails |
| 10 | Export results for a reviewer | Dated, scoped, attributable output |
Notes on the trickier tests
Test 2. OWASP's reconnaissance guidance says documentation may be inaccurate or incomplete and should be verified, and it lists older spec versions as a source of deprecated but still-working endpoints. A tool that trusts your spec blindly will miss exactly those.
Test 3. The same guidance says that when fuzzing authenticated APIs you need to reproduce session handling and recompute dynamic fields like nonces or signatures for each request. A tool that cannot do this will quietly scan as an anonymous user.
Test 4. Account A creates objects, account B tries to reach them by changing IDs. That is the procedure in the guide. A tool that tests with one account cannot do it.
Test 6. Ask what is skipped. A passive-only mode is useful for production but will not find authorization flaws. The open-source ZAP API scan documents this: its safe mode skips the active scan and runs a baseline.
Test 9. The ZAP script treats every alert as a warning by default and uses exit codes to distinguish warnings from failures, with a config file to set each rule to ignore, warn or fail. That is the kind of control to look for in any tool.
Write the criteria first
Decide the pass line before the trial: for example, both planted flaws found, session kept for the whole run, no write methods in safe mode. Then score each test yes or no. If a vendor wants to change the criteria halfway, note that.
What a trial cannot tell you
It will not show how the tool scales across hundreds of services, or how support behaves in an incident. Plan separate checks for those.
Worked example: scoring a trial on a booking API
A team runs a two-week trial against a staging booking API with two planted flaws: the appointment read route skips the owner check, and the clinic settings route lacks a role check. They prepare four accounts and a spec that is a few months out of date. Here is a filled-in scorecard for one imaginary tool, to show what the notes should look like.
| # | Test | Result | Notes |
|---|---|---|---|
| 1 | Import spec | Pass | 112 operations loaded, 9 skipped with a clear reason |
| 2 | Compare with known traffic | Partial | Tool did not know about 6 routes seen in logs, and said nothing |
| 3 | Stay authenticated | Fail | Session dropped after about 30 minutes, later requests all returned 401, no warning |
| 4 | Planted ownership flaw | Not tested | Session had expired before reaching it |
| 5 | Planted role flaw | Pass | Finding named role and route |
| 6 | Mode without write methods | Pass | Report listed skipped checks |
| 7 | Read three findings cold | Partial | Two clear, one had no request or response attached |
| 8 | Repeat scan | Pass | Same findings recognized as existing |
| 9 | CI run and exit status | Pass | Documented exit codes, configurable failure threshold |
| 10 | Export for reviewer | Partial | Export lacked the account names used |
Reading the scorecard: tests 3 and 4 are linked. The tool lost its session silently, and as a result it never reached the planted ownership flaw. A reviewer who only looked at the findings count would have seen a short report and perhaps assumed the API was in good shape. The scorecard exposes that the clean result was not a result.
Deciding
The team set a pass line in advance: both planted flaws found, session maintained for the whole run, no unexplained skips. The tool met one of three. That does not mean the tool is useless. It means a follow-up is needed: ask the vendor whether session refresh is configurable, rerun with a longer token lifetime, and retest. The trial plan allows one retry per failed test, with the cause recorded.
Notes for running your own
- Keep the same accounts and the same planted flaws for every tool you trial. Without a fixed yardstick, comparing results is guesswork.
- Time-box setup. If it takes more than a day to get a tool authenticated against your API, that is data about the effort you would face in production use.
- Ask for the tool's own list of what it could not test, and compare it with yours.
- Involve a developer in test 7. If an engineer cannot reproduce a finding from the report in ten minutes, the report is not good enough for daily use.
- Record support responses too. How quickly and clearly the vendor answers when a test fails is part of the evaluation.
Closing the trial
End with a one-page summary: scorecard, pass line, retests, open questions, and a recommendation to proceed, retry or stop. File it with the test records, so the decision is traceable.
Pitfalls in running a trial
Trials fail in predictable ways, and most of them are about preparation, not the tool.
- Changing the target halfway. If the API changes during the trial, results from week one and week two cannot be compared. Freeze a branch or an environment.
- Giving the vendor all the setup. If the vendor configures everything for you, you learn how their best engineer behaves, not what your team will face. Do at least the authentication setup yourselves.
- Counting findings as the score. More findings is not better. Count how many were real, how many were duplicates, and how many were the planted flaws.
- Ignoring effort. Record hours spent on setup, tuning and triage for each tool. Those hours are a running cost.
- Skipping the unhappy paths. Test what happens when the target is down, the token expires, or the spec fails to parse. Real life is full of these.
- Not involving the people who will live with it. A developer and a security engineer should both read the reports and say whether they would act on them.
If you keep to the scorecard, you can compare tools or compare a tool against your current approach, without relying on impressions.
Try it yourself
Open the demo and read the example findings the way test 7 describes: is there a request, a response and a plain explanation?
Click through it at your own pace, or open it full screen.
Quick check
How long should a trial run?
Long enough to complete all ten tests and one repeat scan. For a single API with prepared accounts, that is usually days rather than weeks.
Can I use a public demo app instead of my own API?
It can show how the interface works, but not how the tool handles your auth, spec quality or quirks.
Should the trial include a load test?
Not as a security criterion. Keep to the ten tests, then ask separately about scale, such as how many APIs and how many requests the tool can handle in your pipeline.
Sources
- OWASP WSTG: API reconnaissance
- OWASP WSTG: API broken object level authorization
- ZAP API scan documentation
Last reviewed: 19 October 2026.
