Train on-call engineers for security incidents with a one page runbook, clear escalation rules and short, regular drills that replay realistic scenarios. Rehearse the first thirty minutes: recognise, contain, preserve evidence and escalate. Long documents are not read at three in the morning, so practice matters more than paperwork.
The gap in most on-call training
On-call engineers handle outages well. They know dashboards, restarts and rollbacks. A security event looks different. The right first move may be to preserve evidence before restarting something, or to avoid alerting an intruder, or to escalate to someone outside engineering. Without practice, people use outage habits on a security problem.
What to put on the one page runbook
- How to tell a security event from an outage, with three or four examples.
- Who to call first, and who after that, with a way to reach them.
- What to do immediately: isolate an account or system, and what not to do, such as wiping a machine.
- What to record: times, actions and observations, in a shared place.
- Who may talk to customers or the press, and who may not.
Keep contact details current. A dead phone number is the commonest runbook failure.
A drill format
| Part | Time | What happens |
|---|---|---|
| Brief | 5 min | Facilitator sets the scene in two sentences |
| Play | 20 min | Engineers talk through what they would do, step by step |
| Injects | within play | Facilitator adds a new fact every few minutes |
| Debrief | 10 min | What was unclear, slow or missing? |
A tabletop drill needs no tooling. Run one a quarter at first, then adjust.
First scenarios to rehearse
- A credential for a cloud account appears in a public repository.
- A dependency in your build turns out to be malicious.
- An employee reports a strange login prompt.
- A customer reports seeing another customer's data.
Pick the one closest to your real risk. If you have a past incident of your own, replay it, because familiar systems make it concrete. See replay a past incident as a training exercise.
Debrief questions
- Where did we hesitate, and why?
- Which step needed information nobody had?
- Which contact or tool would have failed?
- What one change would help most next time?
Turn the answers into a short list of fixes with owners, and check them at the next drill.
Legal and regulatory clocks
Some obligations start a notice clock. Do not guess in a drill. Ask your legal or compliance contact what applies and add a line to the runbook on when they must be told.
Where Safeguard fits
Safeguard offers asset discovery, continuous monitoring and compliance evidence automation, which help responders know what they own and keep records of what was done. It does not offer training courses or incident response services. Try a free scan at gold.safeguard.sh/scan.
Quick check
How often should on-call engineers run security drills?
Quarterly is a good starting rhythm. Move to less often once responses are smooth, and run an extra one after any real incident.
Do drills need special tools?
No. A facilitator, a scenario and a timer are enough to expose most gaps.
Last reviewed: 19 October 2026.
