A backup that silently fails is worse than no backup at all — it gives false confidence. This guide covers automating verification that backups genuinely succeeded and completed correctly.
Why Manual Backup Checking Doesn't Scale
Remembering to manually check backup logs regularly is unreliable — automated verification with alerting ensures you're actually notified of failures promptly, rather than discovering a months-long silent failure only when you desperately need to restore.
Step 1 — Ensure Your Backup Script Reports Success/Failure Clearly
#!/bin/bash
if pg_dump mydb > /backups/mydb_$(date +%Y%m%d).sql; then
echo "Backup succeeded"
exit 0
else
echo "Backup FAILED" >&2
exit 1
fiA clear exit code is the foundation for any automated verification — ensure your backup scripts actually reflect genuine success/failure, not just "the script ran" regardless of actual outcome.
Step 2 — Verify Backup File Integrity, Not Just Creation
if [ -s "/backups/mydb_$(date +%Y%m%d).sql" ]; then
echo "Backup file exists and is non-empty"
else
echo "Backup file missing or empty" >&2
exit 1
fiA backup script that "succeeds" but produces an empty or truncated file is a common, dangerous failure mode — verify actual file size/content, not just script exit code alone.
Step 3 — Set Up Alerting on Failure
0 2 * * * /usr/local/bin/backup.sh || curl -X POST YOUR_ALERTING_WEBHOOK -d "Backup failed on $(hostname)"Integrate with your existing alerting system (see How to Set Up Effective Server Alerting) so a backup failure generates an actual notification, not just a silent log entry no one reviews.
Step 4 — Alert on Backup ABSENCE, Not Just Explicit Failure
A more robust pattern uses a "dead man's switch" — the backup process pings a monitoring service on success; if that ping doesn't arrive within the expected window, the monitoring service itself alerts you — catches scenarios where the backup script doesn't even run at all (a cron misconfiguration, for example), not just cases where it runs and explicitly fails.
Step 5 — Periodically Verify Actual Restorability
See How to Test and Verify Your Backups Actually Work — file existence/size checks are a first line of defense, but periodic actual restore testing is the only way to confirm genuine usability; automate this periodic verification where feasible, not just relying on the lighter-weight checks alone.
Step 6 — Monitor Backup Size Trends
Track backup file sizes over time — a sudden, unexplained drop in size can indicate a partial/corrupted backup even when the script reports success, worth flagging for investigation.
Step 7 — Verify Backup Timing/Completion Duration
Track how long backups take to complete — a sudden significant change in duration (much faster, suggesting incomplete work; much slower, suggesting a different issue) is worth investigating even absent an explicit failure.
Building a Backup Health Dashboard
Consolidate backup status across all your systems into a single dashboard view (see How to Build a Monitoring Dashboard for Your Whole Team) — particularly valuable if managing backups across multiple servers/services.
Common Errors
Alerting itself fails silently — periodically test your alerting pathway itself (deliberately trigger a test failure) to confirm notifications actually reach you, not just that the detection logic is theoretically correct.
Continue Reading
- How to Test and Verify Your Backups Actually Work
- How to Set Up Effective Server Alerting (Without Alert Fatigue)
- How to Automate Backup Rotation and Retention
Browse more articles in Backup & Disaster Recovery.
