A security upgrade can change speed, memory use and startup time, and a passing test suite will not show it. Before merging a fix, run the same short benchmark on the old and new versions, compare a few numbers against a threshold you set in advance, and merge only if the gap is small and explained.
Why fixes change performance
Many vulnerability fixes add work. Input gets validated, a limit gets enforced, a parser stops taking a shortcut, a cryptographic default gets stronger. That is the point of the fix, and a small cost is often the price. The risk is a large cost nobody expected, such as a regular expression change that is slower on long strings, or a new check that runs once per item in a list of thousands.
Other fixes make code faster, for example by capping resource use. You still want to know, because speed changes can shift timing assumptions in your own code.
What to measure
Keep the list short so people actually run it.
| Measure | Why it matters |
|---|---|
| Median and 95th percentile latency of one or two key endpoints | Users feel the tail, not the average |
| Peak memory during a typical job | Fixes that buffer input can raise it |
| Cold start or boot time | Matters for serverless and containers |
| Build time and bundle size | A new transitive dependency can inflate both |
| CPU time for the heaviest background job | Catches per item overhead |
A simple before and after routine
- Pick one repeatable scenario. A load script against a staging copy, or a fixed batch job with a fixed input file. Use the same data every time.
- Run it on the current main branch three times and record the numbers. The spread between those runs tells you how much noise to expect.
- Run it on the fix branch three times, on the same machine, with nothing else running.
- Compare against a threshold agreed beforehand. For example, "no more than a 10 percent slowdown at the 95th percentile unless the pull request explains it." The number is yours to choose. What matters is choosing it before you see the result.
- Write the result in the pull request. A single line with the before and after numbers saves the next reviewer the same work.
Do not trust one run. Laptops throttle, shared runners vary, and caches warm up. If the difference is smaller than the spread between your own repeat runs, call it noise.
When the fix is slower
Do not skip the fix. Decide with evidence. Check whether the slow path is on your hot path at all, since a slow function you rarely call may not matter. Look for a configuration option that lets you keep the protection and recover speed, such as a size limit set to your real maximum. If neither works, merge behind a feature flag or canary so you can watch it in production, and keep a rollback ready.
If the vulnerable code is not reachable from your application, you may have more time to schedule the upgrade carefully. Safeguard provides reachability analysis to help you see whether a vulnerable function is called, plus policies and gates on pull requests to hold a change that fails your own checks. You can try a free scan at gold.safeguard.sh/scan.
Quick check
Should every dependency bump get a benchmark?
No. Run it for fixes in code on your hot path: parsers, serializers, HTTP and database layers, crypto. For a patch to a build tool or a test helper, the normal test run is enough.
How big a slowdown is too big?
That is a business choice. Set a threshold per service in advance, tighter for latency sensitive paths, and record exceptions in the pull request.
Last reviewed: 19 October 2026.
