Because a flag can be flipped in production after the build, the same artifact must pass tests with the flag on and with it off. With many flags, test the combinations that matter, not all of them. Pay special attention to the off path, which often rots first and is the one you fall back to in an incident.
The source of the problem
Martin Fowler's article explains it with a tax calculation example. When an artifact moves through a delivery pipeline, you cannot know whether a flag will later be on or off in production. That is the purpose of flags. So to validate every code path that might go live, you must test the artifact in both states.
With one flag, that means doubling at least some of your tests. With several flags, the number of possible states multiplies quickly.
Why security cares
Each flag state is a distinct program. The off state may skip a check that the on state added, or the reverse. A flag turned on in an emergency may activate code that never ran in a test environment.
The off path is dangerous in a quiet way. When you roll back by turning a flag off, you rely on old code that nobody has exercised since the flag was added.
A practical approach
Start with the default states. Run the full suite with every flag at its production default. This is what ships.
Then flip each flag one at a time. Run the tests tied to that feature with the flag in its non-default state, leaving others at default. This gives you coverage per flag at a cost that grows linearly.
Then add pairs for known interactions. If two flags touch the same function or data, test them together. Document these pairs next to the flags.
Do not try to test every combination. If flags are independent, the pairwise approach catches most problems. If you cannot make flags independent, treat that as a design smell and plan to remove one.
Make the pipeline do it
Parameterize tests by flag state, using a local file or in-memory provider rather than the live flag service. The OpenFeature documentation notes that providers can resolve values from a locally stored file, which suits tests since no network is needed.
Fail the build if a test run did not exercise a flag's declared states. Some teams keep a manifest listing each flag and the states required in CI.
Include security tests
For flags near authentication, authorization or data access, the negative tests matter most. With the flag on, can a user still not reach another tenant's data? With the flag off, is the old check still in place?
Remove flags to reduce combinations
Every removed flag halves the states you carry. Set an expiry date on release flags and delete the code path when it passes. See the related post on feature flag types and lifetimes.
Gate the merge
A pull request gate can require that these test jobs pass before a flagged change merges. Safeguard policies and gates work this way for the rules you define, and the demo below shows the idea.
Try it yourself
Click through it at your own pace, or open it full screen. Notice how a gate blocks a merge until checks pass, then picture one check per flag state.
Quick check
Do I need to test every combination of flags?
No. Test defaults, each flag flipped alone, and known interacting pairs. Remove flags to shrink the state space.
Why test the off state?
It is the fallback in an incident, and it may not have run since the flag was added.
Sources
Last reviewed: 19 October 2026.
