Test a security automation in four stages: unit tests on its rules, replayed sample events in a sandbox project, a dry run in production that only logs what it would do, and a limited rollout to one team. Move on only when each stage matches your expectation.
Stage 1: test the rules alone
Write the decisions as plain functions: given a finding, which team, which priority, which message. Feed in examples including ugly ones: a missing owner, a very long title, a finding with no fix, a duplicate. Confirm the output for each.
Stage 2: sandbox project
Create a separate project in your ticket system and a separate chat channel. Point the automation at them with a separate service account. Replay saved events, including the same event twice, to prove duplicates do not create duplicate tickets.
Stage 3: dry run in production
Run the real automation against real events, but make it write to a log instead of acting. Read the log for a week. Questions to ask:
- Would it have created tickets nobody wants?
- Did it pick the right owner?
- Were there bursts that would trip chat limits?
Stage 4: staged rollout
Turn it on for one team that agrees to help. Watch for a week. Then widen.
Failure cases to inject on purpose
| Failure | What to confirm |
|---|---|
| Ticket system returns an error | Automation retries with a delay, then raises an alert |
| Chat API returns HTTP 429 | It waits for the Retry-After value |
| Same webhook delivered twice | One ticket only |
| Receiver slow to answer | Event is queued, reply is still fast |
| Expired token | Clear error, alert to an owner |
| Owner lookup finds nothing | Fallback owner used |
Keep a kill switch
A single setting that stops all writes, readable without a deploy. Test that it works. Write down who may use it and how.
Keep the test plan
Save the sample events and expected outputs in the repository. Rerun them whenever you change a rule. That also gives you a short piece of evidence for change management.
Quick check
What is a dry run?
The automation runs on real events but only records what it would have done. It shows real behavior with no side effects.
Why replay the same event twice?
Senders retry deliveries. Replaying proves your logic is idempotent and will not create duplicates.
Safeguard's autonomous remediation is off by default with every category on Manual, which suits a staged rollout. Try a free scan at gold.safeguard.sh/scan.
Last reviewed: 19 October 2026.
