The four golden signals are latency, traffic, errors and saturation. If you can only monitor four things for a service, monitor these. One panel for each, plus alerts on errors and latency, gives a small team a dependable picture of whether users are being served.
The definitions
These come from the Google SRE book chapter on monitoring distributed systems.
- Latency: the time to service a request. Track successful and failed requests separately, because a fast error can hide a slow success problem.
- Traffic: the demand on the system, such as HTTP requests per second or concurrent sessions.
- Errors: the rate of failed requests. These can be explicit, such as a 500 status, implicit, such as a 200 with the wrong content, or by policy, such as any response slower than a set limit.
- Saturation: how full the service is, focused on the most constrained resource. The book notes that performance often degrades before utilization reaches 100 percent, so pick a target below it.
Building the dashboard
One row per signal, in this order:
- Traffic: requests per second, split by route group.
- Errors: failed requests as a ratio of total, not a raw count, so traffic changes do not distort it.
- Latency: median and a high percentile, split into success and failure.
- Saturation: the one or two resources that run out first, such as connection pool use, queue depth or memory.
Put the same four rows on every service dashboard. When an incident starts, nobody should be hunting for where the graphs live.
Which two alerts to start with
Alert on the error ratio and on a high latency percentile. Both are symptoms. Leave traffic and saturation as dashboard panels at first, and add saturation alerts only when you know a resource that fails abruptly.
A sudden drop in traffic is worth a look too. A service that suddenly receives nothing may be broken in a way that raises no errors. Consider a low-traffic alert for services that always have steady demand.
What counts as an error
Decide this once per service and write it down. A 404 for a missing page is usually not a service error. A 200 response containing an error body is. If your logs record an outcome field, you can compute the ratio from the same data.
Mistakes to avoid
- Averaging latency. A mean hides the slow tail that users complain about.
- Counting errors without a denominator.
- Treating saturation as CPU alone. Many services run out of connections or threads first.
Quick check
Do the signals apply to batch jobs?
Adapt them. Use job duration for latency, jobs started for traffic, failed runs for errors and backlog size for saturation.
Which percentile should I plot?
Plot more than one, for example the median and a high one. Choose the high percentile that matches what your slowest users experience.
Sources
Last reviewed: 19 October 2026.
