Safeguard
Security Guides•

Game day plan: what happens when a security control goes down

A step by step game day for rehearsing the loss of a security control such as a scanner, identity provider or secrets store, with roles and review.

Safeguard Security Team
3 min read

A game day rehearses your response to a simulated event, using the same people and procedures you would use for real. For a security control outage, pick one control, write the scenario, run it with observers, and finish with a review that produces tracked fixes.

What AWS says a game day is

The AWS Well-Architected reliability guidance describes game days as simulations of events in production-like environments to test systems, processes and team responses. The goal is to perform the same actions the team would perform if the event were real. Its anti-patterns include documenting procedures but never exercising them, excluding business decision makers, and blaming teams for failures.

Step 1: choose the scenario

Pick one realistic failure. Examples: the identity provider is unreachable for thirty minutes; the secrets store returns errors; the vulnerability scanner in the pipeline times out on every build; the log pipeline stops delivering events.

Step 2: write the script

Two pages at most. The starting condition, what the facilitator will inject, what the responders see, and the decision points. Include a question that needs a business answer, such as whether to ship a fix while a control is down.

Step 3: assign roles

Facilitator, who runs the clock. Responders, who act as they would on call. Observer, who writes down timings and confusion. A business stakeholder for decisions. Tell everyone who is involved in advance, as AWS advises, including support.

Step 4: run it

Inject the fault in a safe environment or at small scope. Let responders use real runbooks. Do not coach unless safety requires it. If something goes wrong beyond the plan, roll back and stop.

Step 5: review

Within a few days, hold a short retrospective. Ask what was detected and how fast, which runbook step was wrong or missing, who was unsure of their authority, and whether the control failed open or closed. Turn each answer into a ticket with an owner.

Step 6: repeat

Run a different scenario next quarter, or repeat the same one after fixes. Regular repetition builds the habit.

Quick check

How long should a game day take?

Plan for a few hours: briefing, a run of an hour or so, and a review. Keep the first one small.

Do I need special tooling?

No. A facilitator, a script and a shared document are enough. Fault injection tools help later.

Sources

Last reviewed: 19 October 2026.

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.

Self-healing security runs on Safeguard.

Your first fix PR is minutes away.

No sales call required, even your agent can complete the purchase over MCP.