Simple static thresholds miss anomalies in metrics with natural variance/seasonality — this guide covers moving beyond basic threshold alerting to more sophisticated anomaly detection approaches.
The Limitation of Static Thresholds
See How to Set Up Anomaly Detection for Server Metrics for foundational context — a static threshold ("alert if CPU exceeds 80%") doesn't account for metrics with genuine natural variance (traffic that's naturally higher on weekdays, lower at night) — either causing false alarms during normal peak periods or missing genuine anomalies during normally-quiet periods.
Using Statistical Baseline Comparison
rate(http_requests_total[5m]) < (avg_over_time(rate(http_requests_total[5m])[7d:1h]) * 0.5)Rather than a fixed threshold, compare current values against a historical baseline (same time period across recent days/weeks) — catches genuinely anomalous deviations while accommodating normal cyclical patterns.
Implementing Seasonal-Aware Alerting
For metrics with strong daily/weekly patterns, compare against the same period from previous cycles (this Monday 2pm versus last Monday 2pm) rather than a rolling recent average that might not account for day-of-week patterns.
Using Standard Deviation-Based Detection
abs(current_value - avg_7d) > (3 * stddev_7d)Flag values that deviate significantly (multiple standard deviations) from the established baseline — a statistically grounded approach to defining "anomalous" rather than an arbitrary fixed number.
Using Machine Learning-Based Anomaly Detection
Several observability platforms include built-in ML-based anomaly detection, automatically learning normal patterns and flagging genuine deviations without manual threshold/baseline configuration — can be genuinely more sophisticated than manual statistical approaches, though less transparent/debuggable when it flags something.
Balancing Sensitivity and Noise
Overly sensitive anomaly detection generates false-positive noise, undermining trust (see How to Reduce Alert Fatigue with Smart Alerting Rules); too insensitive misses genuine issues — tune based on actual observed false-positive/false-negative experience, not a single initial guess.
Requiring Sustained Anomalies, Not Single Data Points
for: 10mA single anomalous data point is often noise; requiring the anomalous condition to persist for a meaningful duration before alerting reduces false positives from momentary blips.
Combining Anomaly Detection with Business Context
See How to Set Up Business Metrics Monitoring (Beyond Infrastructure) — a traffic spike might be anomalous by pure statistics but entirely expected given a marketing campaign or product launch; layer business context awareness where feasible rather than treating pure statistical anomaly as automatically alert-worthy.
Validating Anomaly Detection Against Known Historical Incidents
Test your anomaly detection approach against your historical incident timeline — would it have caught your past genuine incidents? This validates whether your approach is genuinely effective, not just theoretically sound.
Common Errors
Anomaly detection misses an incident that seems obvious in hindsight — review whether the specific metric/pattern was genuinely covered by your anomaly detection scope; not every possible failure mode is captured by monitoring a fixed set of metrics, revealing genuine gaps to address for next time.
Continue Reading
- How to Set Up Anomaly Detection for Server Metrics
- How to Reduce Alert Fatigue with Smart Alerting Rules
- How to Set Up Business Metrics Monitoring (Beyond Infrastructure)
Browse more articles in Advanced Observability & Incident Management.
