How to Set Up Log Retention and Archival Policies

Vidensdatabase

Without deliberate retention policy, logs either accumulate indefinitely (wasting storage) or get deleted too aggressively (losing valuable historical data). This guide covers designing appropriate log retention and archival.

Why Deliberate Retention Policy Matters

See How to Set Up Log Sampling to Reduce Volume Without Losing Signal for the related volume-reduction concern — retention policy addresses a different question: not how much to capture, but how long to keep what you've captured, balancing storage cost against genuine future need.

Different Retention Needs for Different Log Types

Log Type	Typical Retention Consideration
Debug/verbose application logs	Short retention (days) — primarily useful for immediate troubleshooting
Security/audit logs	Longer retention, often driven by compliance requirements (see How to Set Up Audit Logging for Compliance Requirements)
Access logs	Moderate retention, useful for trend analysis and incident investigation

Setting Up Tiered Retention in Elasticsearch/ELK

PUT _ilm/policy/logs_policy
{
  "policy": {
    "phases": {
      "hot": {"min_age": "0ms", "actions": {}},
      "warm": {"min_age": "7d", "actions": {"shrink": {"number_of_shards": 1}}},
      "cold": {"min_age": "30d", "actions": {}},
      "delete": {"min_age": "90d", "actions": {"delete": {}}}
    }
  }
}

See How to Set Up Centralized Logging with the ELK Stack (Elasticsearch, Logstash, Kibana) — Index Lifecycle Management (ILM) automates moving data through progressively cheaper storage tiers before eventual deletion, based on age.

Setting Up Retention in Grafana Loki

limits_config:
  retention_period: 720h

See How to Set Up Centralized Logging with Grafana Loki (Lightweight Alternative) — Loki's retention configuration is simpler than Elasticsearch's tiered ILM, appropriate for its more streamlined logging philosophy.

Archiving to Cold Storage Rather Than Deleting

aws s3 cp old-logs.tar.gz s3://log-archive-bucket/2026/

For logs with genuine long-term value (compliance, historical analysis) but infrequent access needs, archiving to cheap object storage (see How to Back Up to Object Storage (S3-Compatible)) rather than outright deletion balances cost against retained access.

Determining Retention Periods Based on Genuine Need

Consider: how far back have you actually needed to look during past incidents? What compliance requirements apply (see specific framework guides throughout this Knowledge Base)? What's your actual storage budget? Base retention decisions on these genuine factors, not arbitrary defaults.

Balancing Retention Cost Against Debugging/Compliance Value

Longer retention has real storage cost; shorter retention risks losing data you later realize you needed — there's no universally correct answer; make this trade-off deliberately for your specific situation rather than defaulting to either extreme.

Documenting Your Retention Policy

Clearly document what's retained, for how long, and why — useful both for your own team's understanding and for any compliance audit that might ask about your data retention practices.

Reviewing and Adjusting Retention Periodically

As your storage costs, compliance requirements, and actual usage patterns evolve, periodically reassess whether your retention policy still represents the right balance — not a one-time decision to set and forget.

Common Errors

Needed log data from an incident investigation but it was already deleted — review whether your retention period genuinely matches your actual investigation needs; consider whether certain log categories warrant longer retention specifically because incident investigation sometimes requires looking back further than typical.

Continue Reading

Browse more articles in Advanced Observability & Incident Management.

Vidensdatabase

Kategorier

Kategorier