Measure the gap between when a public advisory appears and when your scanner first reports it on a project that uses the affected version. Record three timestamps, repeat for several advisories, and compare the median and the worst case. The same test works for open source tools and managed platforms.
Definition
Advisory to detection lag is the time from first public availability of an advisory to the first scan result that flags it in your environment. It includes the scanner's data refresh, and, for a scheduled scan, the wait for the next run.
What you need
- A throwaway repository with a few pinned dependencies, so no real code is at risk.
- Access to a public advisory source such as the GitHub Advisory Database, which lists the published date and affected versions.
- A spreadsheet with five columns: advisory id, ecosystem, published time, first seen in scanner, lag.
Steps
- Choose three to five advisories published in the last week, from at least two ecosystems.
- For each, add a dependency on a vulnerable version to the throwaway repository. Use a version the advisory lists as affected.
- Note the advisory's public timestamp.
- Trigger a scan, or wait for the scheduled one. Record the time the finding appears.
- Compute the lag. Separate two parts: data lag (the scanner did not know) and schedule lag (it knew, but had not run).
- Repeat monthly for a quarter. One data point tells you little.
How to read the results
| Pattern | Likely cause | What to do |
|---|---|---|
| Finding appears within hours, every time | Fast data refresh | Keep measuring, no action |
| Some ecosystems lag, others do not | Source coverage differs | Ask which source feeds the slow ecosystem |
| Lag matches your scan schedule | Schedule lag | Scan on new advisory, not only on a timer |
| Findings never appear | Matching or coverage gap | Report to maintainers or vendor with evidence |
Keep expectations honest. The GitHub Advisory Database states that unreviewed advisories are published automatically from the NVD feed, without validity or completeness checks, and that alerts are generated only for reviewed ones. Some scanners treat reviewed and unreviewed records differently, so a miss may be a policy choice, not a bug. Ask.
Make it part of buying and operating
If you are evaluating tools, run the test during the trial on all candidates at once, with the same advisories. If you run an open source scanner, put the test on the calendar and assign it to the scanner owner. If you pay a vendor, put your measured numbers in the next review meeting. Evidence beats a slide.
Try it yourself
Click through it at your own pace, or open it full screen. Look at how quickly a new advisory is tied to the components you actually run.
Safeguard includes zero-day discovery and continuous monitoring, and a free scan is available at gold.safeguard.sh/scan if you want another data point for your test.
Quick check
How many advisories are enough for the test?
Three to five per round, repeated monthly, gives a usable pattern. A single advisory can be an outlier.
Should I test with real projects?
Use a throwaway repository. It gives clean timestamps and avoids touching production code.
Sources
Last reviewed: 19 October 2026.
