Design logging so that a failure to write a log never takes down the request that produced it, and never leaks data in the process. Use bounded buffers, non-blocking writes and a fallback, and alert when a source goes quiet. OWASP states that logging failures must not crash the application.
How a log problem becomes an outage
Common chains:
- the log disk fills, writes block, and request threads stall
- the log shipper cannot reach the central store, memory grows, and the process is killed
- a logging call throws an exception inside a payment handler, and the payment fails
- the logger writes to a remote service synchronously, so every request gets slower when that service does
None of these show a bug in business code. All of them hurt users.
Rules to apply
- Wrap log calls so exceptions are caught and swallowed, at most incrementing an internal counter.
- Use asynchronous logging with a bounded queue. When the queue is full, drop low value messages first, and count what you drop.
- Never block a request thread waiting on a remote log service.
- Set disk limits and rotation so one noisy source cannot fill a volume.
- Keep a local fallback, such as standard error captured by the platform, for when the shipper is down.
Decide what is allowed to drop
Not all events are equal. Debug lines can drop under pressure. Security and audit events are different, since losing them can break an investigation or a compliance promise. For those, consider writing to a durable local queue first, then forwarding. If the queue is full, you may face a real choice between refusing the action and proceeding unaudited. Decide that deliberately, per action, and record the decision.
Do not leak while failing
Error handling around logging can itself expose data. A fallback that prints a whole request body to standard error on failure defeats your redaction. Keep the same masking on the fallback path.
Make silence visible
The worst failure is the quiet one. Track the number of events received per source and alert when it drops to zero or falls sharply. Also expose a metric for dropped messages and for the shipper's queue depth. If these rise, you know before the next incident.
Test it
In a staging environment, try three experiments: fill the log volume, block the network to the log store, and make the logger throw. Check that requests still succeed, that latency does not climb, and that your silence alert fires. Many teams discover that the shipper restarts without limits or that a library blocks by default.
Quick check
Should I log synchronously for safety?
Synchronous writes to a local file are often fine. Synchronous calls to a remote service on the request path are the risky kind.
What if an audit event cannot be written?
For high risk actions, you may choose to fail the action. Decide this per action and document it.
Sources
Last reviewed: 19 October 2026.