Treat a policy change like a release. Replay the new rule against recent pull requests to see what it would have blocked, run it in a non blocking mode first, tell developers what changes and when, and keep the previous version ready for rollback.
Why policy changes cause more pain than the first rollout
The first rollout of a gate is expected to be disruptive, so people prepare. A later tweak, such as lowering a severity threshold or adding a license rule, arrives without ceremony. Builds that passed on Friday fail on Monday and nobody knows why. Trust in the gate drops, and the next change gets resisted.
The five step change process
- Write the change as a diff. Keep policy in version control so that the change is a reviewable pull request with an author and a reason. A change made in a web console with no history is hard to roll back.
- Replay on history. Run the proposed policy against the last 30 days of merged pull requests. Count how many would have been blocked and look at five of them by hand. If the number is large, the rule is too broad or you need a baseline for existing findings.
- Run in observe mode. Some platforms let you do this natively. GitHub's documentation for rulesets says that when a ruleset runs in Evaluate mode you can see actions that would have passed or failed had it been active. Availability depends on your plan, so check the current docs. If your tooling has no such mode, a non required check that reports but does not block gives the same effect.
- Announce. Post what changes, the date it starts blocking, one example of a failure, and where to ask for an exception. Two working weeks of notice is a sensible default for rules that will block routinely.
- Enforce, watch, and keep the rollback. Monitor the block rate and the exception request rate for the first week. Revert the policy file if the rate is far above what the replay predicted.
What to measure after the change
| Signal | Healthy | Worrying |
|---|---|---|
| Block rate versus replay estimate | Close to the estimate | Several times higher |
| Exception requests | A few, with clear reasons | A spike in the first days |
| Re-runs of failed checks | Flat | Rising, which suggests confusion |
| Time to unblock a failed gate | Same as before | Longer |
Changes that deserve extra care
Lowering a severity threshold, switching from warn to block, and adding a new category of finding all change what developers see daily. Changes that only make a message clearer or add an allowed exception carry little risk and can ship quickly.
Keep a short changelog in the repository that holds the policy. One line per change, with date, reason and link to the pull request. Auditors and new team members both read it.
Try it yourself
Click through it at your own pace, or open it full screen. Look at what the pull request check shows a developer, and imagine your new rule appearing there.
Safeguard policies, gates and guardrails can be applied to pull requests, with a free scan at gold.safeguard.sh/scan to check your starting point.
Quick check
How much notice should developers get before a new blocking rule?
Two working weeks is a common default for rules that will block often. A rule that rarely triggers can have less notice.
What if the replay shows far too many blocks?
Narrow the rule, baseline existing findings so only new ones block, or phase it by repository. Do not ship it and rely on exceptions.
Sources
- GitHub Docs, Managing rulesets for a repository
Last reviewed: 19 October 2026.
