Keeping a local directory automatically synchronized with object storage — for backups, static asset distribution, or file archival — is a common practical need. This guide covers setting up reliable automated sync.
Common Use Cases for Automated Sync
- Backing up local files to object storage (see How to Back Up to Object Storage (S3-Compatible))
- Distributing static assets/uploads to a location a CDN can serve from
- Archiving completed data (logs, processed files) off local storage
Using the mc mirror Command (MinIO Client, Works with Any S3-Compatible Storage)
mc mirror /local/directory myminio/mybucket/pathPerforms a one-time sync, copying files that don't yet exist or have changed in the destination — efficient, since it doesn't re-upload unchanged files.
Continuous Sync (Watch Mode)
mc mirror --watch /local/directory myminio/mybucket/pathContinuously monitors the local directory, automatically syncing new/changed files as they appear — useful for near-real-time sync needs rather than periodic batch sync.
Using AWS CLI Sync (For AWS S3 or Compatible Services)
aws s3 sync /local/directory s3://mybucket/pathSetting Up Periodic Automated Sync with Cron
0 * * * * mc mirror /local/uploads myminio/mybucket/uploads >> /var/log/storage-sync.log 2>&1For scenarios not needing real-time sync, a periodic cron-scheduled sync is simpler and less resource-intensive than continuous watch mode.
Handling Deletions (Mirror vs One-Way Copy)
mc mirror --remove /local/directory myminio/mybucket/pathBy default, sync tools typically only add/update, not delete — add the remove/delete flag deliberately if you want the destination to genuinely mirror local deletions too; understand this is a meaningfully different behavior with real data-loss implications if misconfigured.
Excluding Specific Files/Patterns
mc mirror --exclude "*.tmp" --exclude ".git/*" /local/directory myminio/mybucket/pathVerifying Sync Completeness
mc diff /local/directory myminio/mybucket/pathPeriodically verify local and remote are genuinely in sync as expected — particularly important if using periodic sync where a failure between runs could leave things silently out of sync for a period.
Setting Up Alerting for Sync Failures
See How to Set Up Automated Backup Verification and Alerting for the general pattern — if this sync is backup-related, apply similar verification and alerting discipline, ensuring a failed sync doesn't go unnoticed.
Considering Bandwidth Impact
For large directories or frequent sync intervals, be mindful of bandwidth consumption, particularly if your VPS has metered bandwidth — large initial syncs especially can consume significant bandwidth in a short window.
Common Errors
Sync appears successful but files are missing at destination — verify bucket/path permissions are correctly configured for the credentials being used, and check for any silent errors in sync logs that might not have caused a non-zero exit code.
Continue Reading
- How to Back Up to Object Storage (S3-Compatible)
- How to Set Up Self-Hosted S3-Compatible Object Storage with MinIO
- How to Use Object Storage for Application File Uploads
Browse more articles in Object Storage, Messaging & APIs.
