If your secrets manager is unreachable, running services should keep using the secrets already in memory, and new starts should fail loudly rather than fall back to a default or stale credential. Cache for a bounded time, retry gently, and alert early so the outage is fixed before restarts pile up.
Two moments, two answers
The outage hits at two different points in a service's life.
Already running. The service loaded its secrets at startup. If it keeps them in memory, a short outage changes nothing until a secret needs rotating. This is the safe, boring case.
Starting or restarting. The service needs secrets now and cannot get them. This is where the design choice shows up.
Options at startup
| Approach | Behaviour | Risk |
|---|---|---|
| Fail to start | Pod or process exits with a clear error | Restarts and scale ups stall during the outage |
| Use a local encrypted cache | Start with the last known secret | Cached value may be revoked or rotated |
| Use a baked-in default | Start with a fallback credential | Unsafe: a weak shared secret becomes the real one |
| Start degraded | Run features that need no secret, refuse the rest | More code paths to test |
Never ship a hardcoded fallback password or key to keep the lights on. That is exactly the situation CWE-636 warns about, where a failure drops the system to a less secure state. If a default is ever necessary, it should deny access, not grant it.
Bound the cache
A cache solves outages and creates a revocation delay. Pick a maximum age that matches how fast you must cut off a leaked secret. Short is safer; long survives longer outages. Write the number down and test it, the way you would test token revocation in a security chaos experiment.
Keep cached secrets in memory where you can. If you must write them to disk, encrypt them and restrict the file permissions.
Rotation during an outage
Rotation is the awkward part. If a secret rotates while the store is down, services holding the old one will eventually fail. Overlap the old and new values for a window so both are valid, and do not start a rotation when the store is unhealthy.
Operational steps
- Alert on secrets store latency and error rate, not just on total outage.
- Pre-warm: load secrets at startup, then refresh in the background.
- Use jittered, capped retries so a recovering store is not flooded, see retry storms.
- Add a break glass path for a human to supply a secret, with logging. See break glass access when SSO is down for the same idea applied to identity.
- Practise the outage in staging.
Quick check
Is caching secrets in memory safe?
It is the usual compromise. Memory is harder to reach than disk, but a process memory dump can expose it, so limit who can attach to the process.
Should the app crash if the store is down at startup?
Usually yes, with a clear error. A loud failure beats starting on a guessed credential.
Sources
Last reviewed: 19 October 2026.
