Static thresholds (alert if CPU > 90%) miss gradual drift and don't adapt to normal variation (e.g. predictable daily traffic patterns). Anomaly detection identifies unusual patterns relative to a metric's own historical behavior.
Why Static Thresholds Have Limits
A fixed CPU threshold might be perfectly normal during expected peak hours but genuinely concerning at 3 AM when traffic is normally minimal — static thresholds can't distinguish between these very different situations without manual, time-based tuning.
Simple Approach: Time-of-Day Aware Thresholds
Before reaching for full statistical anomaly detection, consider whether simply defining different thresholds for different times of day/week adequately solves your specific problem — often simpler and more transparent than a full anomaly detection system.
Statistical Approach: Standard Deviation-Based Detection
# Example PromQL: alert if current value is more than 3 standard deviations from the 7-day average
avg_over_time(cpu_usage[7d]) + 3 * stddev_over_time(cpu_usage[7d]) < cpu_usageFlags values that are statistically unusual relative to recent historical behavior, rather than an arbitrary fixed number.
Using Grafana's Built-In Anomaly Detection Features
Some Grafana plugins and integrations provide anomaly detection capabilities directly within your existing dashboards — check current Grafana plugin offerings, since this ecosystem evolves.
Machine Learning-Based Approaches (More Advanced)
For more sophisticated pattern recognition (seasonal trends, multi-metric correlation), dedicated anomaly detection tools using machine learning models exist — genuinely valuable at scale, but adds real operational complexity; evaluate whether your actual needs justify this over simpler statistical approaches.
Practical Example: Detecting Unusual Traffic Patterns
# Alert if current request rate deviates significantly from the same time last week
abs(rate(http_requests_total[5m]) - rate(http_requests_total[5m] offset 7d))
/ rate(http_requests_total[5m] offset 7d) > 0.5Compares current traffic to the same period one week prior, accounting for normal weekly patterns rather than flagging expected daily/weekly variation as anomalous.
Avoiding False Positives During Genuine Change
Anomaly detection based on historical patterns can generate false positives during legitimate changes (a marketing campaign driving traffic, a new feature launch) — build in a way to acknowledge "this is expected" during known events, avoiding unnecessary alert noise.
Starting Simple
Begin with straightforward statistical methods (standard deviation from a rolling average) before investing in more complex machine learning-based anomaly detection — many teams find simple statistical approaches sufficient for their actual needs.
Combining Anomaly Detection with Traditional Thresholds
Anomaly detection doesn't need to fully replace static thresholds — use hard thresholds for genuinely dangerous absolute levels (e.g. disk 95% full is bad regardless of historical pattern) alongside anomaly detection for metrics where "normal" genuinely varies by context.
Common Errors
Too many false positive anomaly alerts — the detection window or sensitivity threshold likely needs tuning; start with a more conservative (less sensitive) configuration and tighten gradually based on observed accuracy.
Continue Reading
- How to Reduce Alert Fatigue with Smart Alerting Rules
- How to Set Up Prometheus and Grafana for VPS Monitoring
- How to Define and Track SLOs and Error Budgets
Browse more articles in Advanced Observability & Incident Management.
