A shallow health check confirms a process is alive. A deep check also tests its dependencies, such as the database. Deep checks catch more problems but can fail on every server at once when a shared dependency slows, removing the whole fleet from service. Use local checks for automatic removal.
The four kinds of check
The AWS Builders Library article on health checks separates them clearly.
| Type | What it tests | Typical risk |
|---|---|---|
| Liveness | A server process is listening and answers a basic request | Misses a server that answers but cannot do real work |
| Local health | Resources not shared with peers, such as disk access or critical local processes | Low; unlikely to fail on many servers at the same moment |
| Dependency health | Ability to reach adjacent systems, including credentials and configuration freshness | False positives when the dependency itself is the problem |
| Anomaly detection | Compares each server against its peers | Needs fleet-wide monitoring data |
Local checks are the safe middle ground because they test things that belong to one server. A full disk on one machine is a good reason to remove that machine. It is unlikely to happen on every machine simultaneously.
How a deep check cascades
Imagine fifty servers that all query one database. The database slows down. Each server's deep check queries the database, times out and reports unhealthy. The load balancer removes all fifty. Now there is nothing serving, even though many requests could have been answered from cache or with partial results.
The article describes this trade-off directly: a gap in health checking can let a bad server reduce availability until it is noticed, but avoiding the gap risks a complete outage caused by unexpected health check behavior. It also says teams there tend to restrict fast-acting load balancer health checks to local health checks, and use deep checking with the safety of rate-limited automation.
Fail open is a related safeguard. The article notes that some load balancers keep sending traffic to all servers when none appear healthy. It also cautions that this behavior is hard to prove for every kind of overload or partial failure, so it should be tested rather than assumed.
A safer setup
- Use a liveness check for the load balancer's quick decisions, plus a local check for disk and critical processes.
- Run dependency checks separately, and have them raise an alert or a rate-limited action rather than instant removal.
- Add anomaly detection that compares servers against each other, so a single odd server stands out.
- Put a floor on automation. Decide the largest share of the fleet that automation may remove, and require a person for anything beyond it.
- Test the failure of each dependency in staging and record what the checks did.
Why this matters for security tooling too
The same pattern applies to security controls placed in the request path. A gate that calls an external service and treats a timeout as a failure can stop every deployment. Decide in advance whether it fails open or closed, and write down who may override it.
A worked example: fifty servers and one slow database
This example is hypothetical. A service runs on fifty servers behind a load balancer. Each server answers a health check every few seconds. The check does three things: confirms the process is up, writes a small file to local disk, and runs a simple query against the shared database.
At 10:00 the database starts responding slowly because of an unrelated batch job. Here is what happens under two designs.
| Moment | Design A: deep check removes servers | Design B: local check removes, dependency check alerts |
|---|---|---|
| 10:00 | Query takes longer than the check timeout | Same slowdown |
| 10:01 | Several servers report unhealthy and are removed | Local checks still pass, servers stay in service |
| 10:02 | Remaining servers take more load and slow further | An alert tells on call that the dependency check is failing |
| 10:03 | The rest fail their checks | On call investigates the batch job while users see slower pages |
| 10:04 | No healthy servers; users see errors | Slow but working service |
In Design A the health check turned a slowdown into an outage. In Design B the slowdown is visible and a person decides what to do. The AWS article names this risk of cascading failures from health check behavior and describes keeping fast acting checks local.
Step by step: reviewing your own checks
- Find every health check endpoint in your services and every load balancer or orchestrator rule that uses it.
- For each, write what it tests. Sort the tests into four groups: liveness, local, dependency and fleet comparison.
- Mark any dependency test that triggers automatic removal or restart. These are the ones that can cascade.
- Replace those removals with alerts, or with removal that is capped. For example, allow automation to remove no more than a set fraction of servers at once, and require a person after that.
- Check restarts too. A check that restarts a process when the database is slow adds load to the database as every process reconnects.
- Add a fleet comparison. Even a simple dashboard that shows error rate and latency per server will reveal one server behaving differently from the rest.
- Test by slowing a dependency in staging and watching what the checks do.
Liveness and readiness in container platforms
If you use a container orchestrator, the same thinking applies to its probes. A liveness probe that fails when a dependency is slow can restart every container, making recovery harder. A readiness probe that depends on a shared backend can remove every pod from service. Keep liveness probes about the process itself, and be deliberate about what readiness includes. Our post on liveness and readiness endpoints covers exposure and content.
Security angle
Health endpoints should not leak internal details such as versions or connection strings, and should not be a way to trigger expensive work. A deep check that runs a heavy query on every call can itself become a load source, especially if it is public. Keep the check cheap, rate limited and light on information.
Try it yourself
Open the pull request gate demo and look at how a check decides to pass or block a change. Think about what should happen if the checking service itself is slow, and who decides.
Click through it at your own pace, or open it full screen.
Quick check
Should a health check ever test the database?
Yes, but think about what happens when the database is slow for everyone. Prefer an alert or a rate-limited response over automatic removal of the whole fleet.
What does fail open mean in this context?
When no server looks healthy, traffic continues to go to all of them instead of none. It can help, but the source article advises testing it under overload and partial failure.
How often should a health check run?
Often enough to detect a failed server quickly, but not so often that the checks add load. A few seconds is common. Pair it with a threshold of several consecutive failures before acting, so one slow response does not remove a healthy server.
Sources
Last reviewed: 19 October 2026.
