How to Handle Backup Failures Gracefully

Base de Conhecimento

Backup failures happen — disk full, network issue, database lock timeout. This guide covers building resilience into your backup process so a single failure doesn't leave you without recent, restorable data.

Why Graceful Failure Handling Matters

A backup script that silently fails and never runs again (due to an unhandled error, for example) leaves you increasingly exposed over time — resilient failure handling ensures you're aware of issues promptly and, ideally, that transient failures don't compound into extended gaps in your backup coverage.

Common Backup Failure Causes

Insufficient disk space on the backup destination
Network connectivity issues (for remote backup destinations)
Database lock timeouts or connection issues
Backup script bugs or unexpected data conditions

Building Retry Logic for Transient Failures

#!/bin/bash
MAX_RETRIES=3
RETRY_DELAY=300

for i in $(seq 1 $MAX_RETRIES); do
    if pg_dump mydb > /backups/mydb.sql; then
        echo "Backup succeeded on attempt $i"
        exit 0
    fi
    echo "Backup attempt $i failed, retrying in ${RETRY_DELAY}s..."
    sleep $RETRY_DELAY
done

echo "Backup failed after $MAX_RETRIES attempts" >&2
exit 1

Handles transient issues (a brief network blip, a momentary lock conflict) without requiring manual intervention, while still ultimately alerting if the underlying issue is persistent.

Checking Available Disk Space Before Starting

AVAILABLE=$(df /backups --output=avail | tail -1)
REQUIRED=5000000

if [ "$AVAILABLE" -lt "$REQUIRED" ]; then
    echo "Insufficient disk space for backup" >&2
    exit 1
fi

Fail fast with a clear message rather than starting a backup destined to fail partway through due to insufficient space.

Ensuring a Failed Backup Doesn't Leave Corrupted/Partial Files

pg_dump mydb > /backups/mydb.sql.tmp && mv /backups/mydb.sql.tmp /backups/mydb.sql

Writing to a temporary file first, then atomically renaming only on success, ensures a failed backup doesn't overwrite your previous good backup with a partial/corrupted one.

Never Let a Failed Backup Replace Your Only Good Copy

See How to Automate Backup Rotation and Retention — maintain multiple historical backup generations, not just the most recent; if today's backup fails or is corrupted, you still have yesterday's (or older) good backup as a fallback, rather than being left with nothing.

Setting Up Multi-Destination Backups for Redundancy

See How to Back Up Multiple VPS Instances from One Central Location and object storage backup guides — backing up to multiple independent destinations means a single destination's failure (storage service outage, for example) doesn't leave you completely without a current backup.

Alerting Appropriately on Different Failure Types

Distinguish between a transient failure that self-resolved after retry (worth logging, perhaps a lower-urgency notification) versus a persistent failure requiring genuine attention (should trigger a proper alert, see How to Set Up Automated Backup Verification and Alerting).

Documenting Known Failure Modes and Responses

As you encounter and resolve specific backup failure scenarios, document them — builds institutional knowledge helping you (or a teammate) respond faster to a recurrence.

Common Errors

Backup script fails silently with no indication anything went wrong — the most dangerous failure mode; ensure every backup script has explicit success/failure reporting and integrates with your alerting system, never assuming silent operation means silent success.

Continue Reading

Browse more articles in Backup & Disaster Recovery.

Base de Conhecimento

Categorias

Categorias