A deployment that silently causes problems can go unnoticed until user impact accumulates. This guide covers automated deployment health monitoring with automatic rollback when problems are detected.
Why Automated Health Checking Beats Manual Monitoring
Watching dashboards manually after every deployment doesn't scale and is prone to human oversight, especially for deployments outside normal working hours — automated health checks with defined rollback criteria provide consistent, immediate response.
Defining What "Healthy" Means for Your Deployment
- HTTP error rate below a defined threshold
- Response time within acceptable bounds
- No spike in application-level error logs
- Key business metrics (successful checkouts, for example) not degrading
Basic Health Check Script Integrated into Deployment
#!/bin/bash
deploy_new_version
sleep 60
ERROR_RATE=$(curl -s http://localhost:9090/api/v1/query?query=error_rate | jq '.data.result[0].value[1]')
if (( $(echo "$ERROR_RATE > 0.05" | bc -l) )); then
echo "Error rate too high, rolling back"
rollback_to_previous_version
exit 1
fi
echo "Deployment healthy"A simplified illustration — deploy, wait briefly for traffic to exercise the new version, check a key health metric (here pulled from Prometheus), and automatically roll back if it exceeds an acceptable threshold.
Combining with Canary/Blue-Green Deployment
See Canary Deployments: Gradually Rolling Out Changes and How to Set Up Blue-Green Deployment on a VPS — automated health checking is most powerful combined with these gradual rollout patterns, since you can automatically abort a canary rollout before it reaches full production traffic if health degrades.
Setting Appropriate Wait Time Before Checking
Checking too immediately after deployment may not capture genuine issues that only manifest under real traffic load or after some warm-up period — but waiting too long delays detection and rollback; tune this window based on your application's actual characteristics.
Avoiding False-Positive Rollbacks
An overly sensitive health check threshold can trigger unnecessary rollbacks from normal traffic variance — base thresholds on your genuine historical baseline (see How to Set Up Effective Server Alerting (Without Alert Fatigue) for related alerting-threshold considerations), not arbitrary numbers.
Ensuring the Automated Rollback Actually Works
Test your automated rollback mechanism deliberately (in a controlled scenario) rather than assuming it works correctly — discovering rollback automation is broken during an actual incident defeats its purpose.
Logging Automated Rollback Events
When an automated rollback triggers, ensure it's logged clearly and, ideally, triggers a notification (see How to Set Up Effective Server Alerting) — an automatic rollback is a signal something went wrong and warrants human investigation, not just silent self-correction.
Considering Business Metrics, Not Just Technical Metrics
Technical health (error rate, response time) doesn't always capture every problem — for genuinely critical deployments, consider whether key business metrics (checkout completion rate, sign-up rate) should also factor into automated health assessment, since some regressions manifest there before showing up in technical metrics.
Common Errors
Rollback triggers but old version also has issues (rollback isn't actually clean) — verify your rollback mechanism genuinely restores a fully working prior state, including any related configuration or migration considerations (see How to Set Up Automated Database Migrations in a Deployment Pipeline for schema-related rollback complexity).
Continue Reading
- Canary Deployments: Gradually Rolling Out Changes
- How to Roll Back a Bad Deployment Quickly
- How to Set Up Effective Server Alerting (Without Alert Fatigue)
Browse more articles in DevOps & CI/CD.
