For one widely used open source scanner, no. Its documentation now says the AJAX spider is no longer the recommended option for crawling modern web apps and recommends the Client Spider instead. Whatever tool you use, check its crawler against your own app, because crawl coverage sets the ceiling for everything else.
Why the crawler matters most
A scanner can only test what it finds. If the crawl misses the checkout flow or the admin area, no attack rule ever runs there. A clean report then means "clean where it looked". Teams spend hours tuning attack rules and none checking whether the crawler reached the routes.
Definitions
- Traditional spider: fetches pages and follows links in the HTML. Fast, but blind to routes built by JavaScript.
- Browser-driven crawler: loads pages in a real or headless browser, runs the scripts, clicks elements and records the requests that result.
- Client-side crawler: a newer design that inspects what happens in the browser and records the routes and calls it finds.
What the docs say
The ZAP AJAX spider page carries the note that the AJAX spider is no longer the recommended option for crawling modern web apps, and that the Client Spider is now recommended. ZAP's baseline script documentation shows the transition in its options: a flag to use the modern spider in addition to the traditional one, with the AJAX spider as the default and an option to use the client spider instead. Details change between releases, so read the current documentation for the version you run.
How to check crawl coverage
- List your routes. Take them from the router file, the API definition or the sitemap.
- Run the crawl alone, before any attack phase.
- Compare. For each route, mark found or missed.
- Look for patterns. Missed routes behind a login, behind a button click, or behind a dynamic URL point to different fixes.
- Add seeds. Feed missed routes in by hand, or import an API definition.
- Fix authentication first. Many "missed" routes are simply behind a login the crawler lost.
- Rerun and recompare. Record the coverage percentage.
Common reasons a crawler misses pages
| Symptom | Likely cause | Fix |
|---|---|---|
| Only the home page found | Links built by JavaScript | Use a browser-driven crawler |
| Pages behind login missing | Session lost, or logout link followed | Verify login, exclude logout |
| Deep menu items missing | Needs clicks or hover | Raise crawl depth and time |
| API calls missing | Not triggered by UI | Import the API definition |
| Pages found but forms never tested | Tokens or validation | See anti-CSRF handling |
Setting time limits
Browser-driven crawling is slower than link following. Give it enough time to finish, and note when it stops because of a limit. A crawl that hit its limit is an incomplete crawl.
What to record
Note the crawler type and version, depth and time settings, and the number of routes found against expected. That one line is the most useful context for any later report.
A worked example
A team scans a single-page app. The app has 85 routes defined in its router file. They run three crawls in a test environment and count the routes found.
| Crawl | Setup | Routes found | Notes |
|---|---|---|---|
| A | Link-following spider only | 6 | Most routes are built by script |
| B | Browser-driven crawler, default depth, no login | 31 | Public pages found, protected ones missed |
| C | Browser-driven crawler with login verified and logout excluded | 72 | Remaining 13 need clicks on nested menus |
The counts are illustrative. The pattern is typical. The first big gain comes from using a browser-driven crawler, and the second from fixing authentication. The last gap is closed with depth, time and manual seeds.
For crawl C, the team verified the login by checking for an element that only appears when signed in. They also excluded the logout path, since a crawler that follows a logout link loses its session and quietly finishes the crawl as an anonymous user.
Closing the last gap
- List the 13 missed routes and look for a shared trait. Here they all sit behind a nested menu that opens on hover.
- Raise crawl depth and the time limit, and rerun.
- Seed any remaining routes by hand from the router file.
- Add the router file as a standing check: a small script compares routes in code with routes the crawler found, and fails if the gap grows.
Choosing a crawler for a new app
| Question | If yes | If no |
|---|---|---|
| Is the front end built with a script framework? | Use a browser-driven crawler | A link-following spider may be enough |
| Is there an API definition? | Import it as a second source of routes | Capture traffic from tests or a proxy |
| Does the app need a login? | Verify login and session checks | Skip |
| Are there infinite lists or calendars? | Set depth and count limits | Skip |
Why tool documentation matters
Crawler recommendations change. The ZAP documentation shows how: the AJAX spider is now described as no longer the recommended option, with the Client Spider recommended instead. Teams that never reread documentation keep using older settings for years. Put a quarterly reminder on the calendar to read the release notes of your scanner and rerun your coverage check.
Infinite spaces and traps
Some apps generate endless URLs, such as calendars that let you click to next month forever. A crawler can spend its whole time budget there. Use exclusion patterns and depth limits, and review the crawl log for repeated URL shapes.
Report the coverage
Put the crawl coverage number at the top of every scan report. Readers should see "72 of 85 known routes visited" before they see a finding count. It frames the whole result honestly.
Where Safeguard fits
Crawl coverage is a web testing matter. For the dependencies that your front end and back end ship, Safeguard offers SBOM and dependency scanning with reachability analysis, and a free scan at gold.safeguard.sh/scan.
Try it yourself
Click through it at your own pace, or open it full screen.
Look at how many pages the scan discovers before testing begins.
Quick check
How do I know my crawler found everything?
Compare its results with a list of your real routes. Missing routes were never tested.
Should I switch crawlers right away?
Test the new one on a copy of your app first and compare coverage with your current results.
What if my app has no list of routes to compare against?
Build one from the router file, an API definition, the sitemap or captured test traffic. Even an imperfect list shows large coverage gaps.
Sources
Last reviewed: 19 October 2026.
