Keep custom rules working through upgrades by pinning the tool and language versions they run on, running the rule's test suite against every version you support, and reading release notes before bumping anything. Syntax trees and plugin APIs change, and a rule that silently stops matching is worse than one that fails loudly.
What changes under a rule
- The tree. Python's documentation states that the abstract syntax itself might change with each Python release. It also records changes such as the 3.13 change to AST node constructors, where omitted fields now take defaults, and a note that in Python 3.15 omitting some fields will raise an error.
- The engine API. ESLint warns that its core rules are not part of the public API and that building on them is fragile and will most likely break at some point. The documentation recommends copying a similar core rule into your project instead.
- Config format. ESLint's navigation includes a migration guide to flat config and a page for migrating to v10, which shows that configuration conventions move between major versions.
- Language features. New syntax appears that your rule never imagined, such as a new way to declare or call something.
The dangerous failure: silent misses
When an engine upgrade breaks a rule loudly, you notice. When a syntax change makes the rule match nothing, builds stay green. Guard against this with a positive control: a test file that must produce a finding, run on every version.
Version pinning
- Pin the linter or analyzer version in the project, with a lock file.
- Pin the rules package version separately.
- Upgrade one thing at a time, so a failure has one suspect.
- Upgrade in a branch, run tests, and compare findings on the real code before and after.
Upgrade checklist
| Step | What to do |
|---|---|
| Read release notes | Look for removed or changed APIs and node types |
| Run rule tests | Every version in your support matrix |
| Run positive control | Confirms the rule still fires |
| Diff findings on real code | Number and location should match, or you can explain why |
| Update docs | Note any behavior change in the changelog |
| Roll out | Pin new version in one project first |
Test across a matrix
If you support several runtimes, run the rule tests on each. A CI matrix with the oldest and newest supported versions catches most issues. Add the next version as soon as it ships in pre release form, if your tool allows.
When to rewrite
Rewrite when you find yourself patching around engine quirks. A rule built on a stable, documented API such as a public visitor interface survives longer than one that reaches into internals. Prefer the documented extension points.
Quick check
Why not always use the latest tool version?
Latest is fine if you test the upgrade. The risk is an untested bump that changes what your rules match without any visible error.
What is a positive control?
A sample file that should always trigger a given rule. If it stops triggering, something broke in the rule, the tool or the config.
Safeguard's scanning covers dependencies and SBOMs, and its reachability analysis helps show which findings your code can actually call. You can run a free scan on a repository.
Sources
- Python documentation: ast module
- ESLint: Custom Rules
- ESLint: Bulk Suppressions (navigation lists migration guides)
Last reviewed: 19 October 2026.
