Detect silent log sources by alerting on the absence of data: expected sources that send nothing, volume that drops sharply, and gaps between event time and ingest time. A quiet log is ambiguous. It might mean a quiet system, or a broken forwarder, or an attacker who turned logging off.
Why silence is the worst failure
Most detection watches for events. If the source stops, the rules never fire, dashboards look calm, and everyone feels reassured. OWASP's Top 10 entry on logging failures lists, among its signs of insufficient monitoring, logs kept only on the local system and attacks that cannot be detected in real time or near real time. A source that silently stops produces the same blindness.
Common causes
- A forwarder crashed or ran out of disk.
- A certificate expired on the log transport.
- A firewall change blocked the collector.
- An API credential for a cloud log feed was rotated.
- Someone disabled auditing, deliberately or by mistake.
- A pipeline rule started dropping a source after an edit.
- UDP loss. RFC 5424 notes syslog has no acknowledgment, so lost messages go unnoticed. See syslog over UDP.
Three checks that catch most of it
1. Heartbeat per source. For every source in your inventory, set an expected maximum silence. Alert if nothing arrives within that period. Chatty sources get minutes. Quiet ones, such as a monthly job, get days.
2. Volume change. Compare the last hour to the same hour in previous weeks. A drop of more than a chosen percentage warrants a look. This catches partial failures where some events still flow.
3. Lag. Compare event timestamp to ingest timestamp. A growing gap means a stuck or overloaded pipeline. Logging both times is explained in log timestamps.
Add synthetic events
Have a scheduled job write a known marker event on each important host or service, such as a line with a unique ID every five minutes, and alert if the marker fails to appear at the other end. It proves the whole path, not just the source.
Watch the watchers
- Alert on agent stop events and on configuration changes to logging.
- Alert on cloud audit trail changes such as disabling a trail or editing its destination.
- Alert on retention setting changes, since shortening retention can erase evidence.
- Monitor collector disk, queue depth and error rates.
Make ownership explicit
Every source needs an owner who is told when it goes quiet. Put owner and expected rate in the source inventory. Without owners, the alert lands in a channel nobody reads.
Test it
Stop the forwarder on a staging host. Time how long until someone is notified. Then fix the gap between what you expected and what happened.
Quick check
Should every source have a silence alert?
Every source that matters for detection or compliance should. Low value sources can be covered by a single volume alert on the whole pipeline.
How do I avoid noisy silence alerts?
Set thresholds per source from its real pattern, use a grace period for deployments, and suppress alerts during planned maintenance with an expiry.
Sources
Last reviewed: 19 October 2026.
