Health check endpoints seem simple but are frequently implemented poorly — either too shallow to be genuinely useful, or too deep in ways that cause cascading failures. This guide covers implementing them correctly.
The Common Health Check Mistakes
Two frequent problems: a health check that always returns "healthy" regardless of genuine application state (providing false confidence), or a health check that transitively checks every dependency so deeply that a single downstream issue cascades into marking many services unhealthy simultaneously.
Distinguishing Liveness from Readiness
See How to Set Up Liveness and Readiness Probes in Kubernetes for the Kubernetes-specific application of this same general principle — even outside Kubernetes, distinguish "is the process alive" (liveness) from "is it ready to serve traffic" (readiness); conflating these causes the problems above.
A Genuinely Useful Liveness Check
app.get('/healthz', (req, res) => {
res.status(200).send('OK');
});Should be extremely simple/fast — confirms the process itself is responsive, without checking external dependencies that shouldn't cause this specific process to be considered "dead."
A Genuinely Useful Readiness Check
app.get('/ready', async (req, res) => {
const checks = await Promise.allSettled([
checkDatabase(),
checkCache(),
]);
const allHealthy = checks.every(c => c.status === 'fulfilled');
res.status(allHealthy ? 200 : 503).json({ checks });
});Checks genuine operational readiness (can this instance actually serve requests correctly) — appropriate for load balancer health checks removing genuinely unready instances from rotation.
Avoiding Deep Transitive Dependency Checks
A readiness check shouldn't transitively check every service your dependencies depend on — check your own direct, immediate dependencies only; deep transitive checking causes exactly the cascading-failure pattern this article warns against.
Setting Appropriate Timeouts on Dependency Checks
async function checkDatabase() {
return Promise.race([
db.query('SELECT 1'),
new Promise((_, reject) => setTimeout(() => reject('timeout'), 2000)),
]);
}A hanging dependency check shouldn't hang the entire health check response — set explicit timeouts, treating a timeout as an unhealthy signal rather than leaving the health check itself unresponsive.
Returning Useful Detail in the Response
{
"status": "unhealthy",
"checks": {
"database": "ok",
"cache": "timeout"
}
}Include which specific check failed, not just an overall pass/fail — genuinely useful for quick diagnosis without needing to dig through logs separately when a health check reports unhealthy.
Excluding Health Check Endpoints from Regular Logging Noise
If health checks run frequently (every few seconds from a load balancer), consider excluding them from regular access logs (or logging at a lower verbosity) to avoid drowning genuine request logs in health check noise.
Testing Health Check Behavior Under Failure Conditions
Deliberately simulate a dependency failure and verify your health check correctly reflects this — a health check that's never actually tested against real failure conditions may not behave as intended when genuinely needed.
Common Errors
Load balancer removes healthy instances during a brief downstream blip — review whether your readiness check is too strict/sensitive to transient downstream issues; consider a small grace period or retry before marking an instance unready for a single failed check.
Continue Reading
- How to Set Up Liveness and Readiness Probes in Kubernetes
- How to Set Up Basic Load Balancing with Nginx
- Understanding the Four Golden Signals of Monitoring
Browse more articles in Advanced Observability & Incident Management.
