Detect a silent behavior change by comparing outputs between old and new versions on the same inputs, asserting invariants that must always hold, and watching business metrics, not just error rates, during a staged rollout. Silent changes do not throw exceptions, so you need checks that look at results, not failures.
What silent changes look like
- A date parser now reads day and month in the other order.
- A rounding rule changes, so totals differ by a cent.
- A serializer drops a field set to null, or orders keys differently.
- A default timeout, page size or retry count changes.
- A validation library becomes stricter or looser, so some requests are now rejected or accepted.
- A hashing or encoding default changes, breaking comparison with stored values.
Semantic Versioning allows incompatible changes only in a major version, but a bug fix in a patch release is allowed to change behavior that was wrong, and some users relied on the wrong behavior. So silent change is possible at any level.
Technique 1: Output diffing on recorded inputs
Collect representative inputs, run them through the old and new builds, and diff the results. This is the idea behind characterization tests. See characterization tests before upgrading legacy code. Normalize timestamps, ids and ordering first, or every run will differ.
Technique 2: Shadow comparison
For pure functions, such as parsers and calculators, call both implementations in production, return the old result, and log differences. Keep it to operations without side effects, log only identifiers and a difference summary, and never log sensitive values. This is the comparison step used in branch by abstraction. See dark launching a replacement library.
Technique 3: Invariants and property checks
State rules that must hold whatever the implementation: a total equals the sum of its lines, a round trip through encode and decode returns the original, a sorted list is sorted, an authorization check denies an unauthenticated request. Property based tests generate many inputs and check these rules, which finds cases you would not think to write down.
Technique 4: Metrics that reflect results
Error rate will not move for a silent change. Watch results instead.
| Metric | Silent change it can reveal |
|---|---|
| Rejected request rate | Stricter or looser validation |
| Average and distribution of order totals | Rounding or pricing changes |
| Rows returned per query | Pagination or filter changes |
| Authentication failures | Token or cookie handling changes |
| Cache hit rate | Key or serialization changes |
| Share of null fields in responses | Serialization changes |
Compare these per version during a canary, as Google's SRE material recommends, since a small canary is diluted in global totals. See canary release metrics.
Technique 5: Read the release notes with a search
Search the changelog for words like default, now, no longer, stricter, deprecated and removed. Each hit is a candidate for a targeted test.
Security relevance
Silent changes are not only functional. A validation library that becomes looser, a token check that no longer rejects an expired value or a cookie flag that changes default can reopen a weakness. Include authorization and input validation cases in your invariants.
Safeguard's dependency scanning shows which components changed between builds, and its policies and gates can block a pull request that introduces a risky package, so the diff you review is complete. Free scan: gold.safeguard.sh/scan.
Quick check
Why not just rely on error monitoring?
Silent changes return results without raising errors, so error rate stays flat. Compare outputs and result-level metrics instead.
Is shadow comparison safe for write operations?
No. Use it for pure, side-effect-free operations only, otherwise you risk duplicate writes or external calls.
Sources
- Semantic Versioning 2.0.0
- Canarying Releases, Google SRE Workbook
- Branch By Abstraction, Martin Fowler
Last reviewed: 19 October 2026.
