Alerts are only useful if you actually act on them. Poorly configured alerting either misses real problems or generates so much noise that important alerts get ignored. This guide covers building an alerting setup that works.
The Alert Fatigue Problem
If you receive dozens of alerts a day, most of which are false positives or non-actionable, you'll eventually start ignoring all of them — including the one that matters. Effective alerting is as much about what you don't alert on as what you do.
What Genuinely Deserves an Alert
- Service completely down (website, database, critical background job)
- Disk space critically low (e.g. above 90%)
- SSL certificate expiring within a few days
- Failed backup jobs
- Sustained high error rates in application logs
- Security-relevant events (multiple failed root login attempts, new admin user created)
What Usually Shouldn't Be an Immediate Alert
- Brief, momentary CPU spikes that self-resolve
- A single failed health check (require multiple consecutive failures instead)
- Informational events with no action required
Using Consecutive Failure Thresholds
Most monitoring tools support requiring multiple consecutive failed checks before alerting — this single setting eliminates the majority of false-positive noise from transient network blips or brief resource spikes.
Setting Meaningful Thresholds
| Metric | Reasonable Alert Threshold |
|---|---|
| Disk usage | 85–90%, sustained |
| Memory usage | 90%+ with active swap usage, sustained |
| CPU load | Above core count for 5+ minutes |
| SSL expiry | 14 and 3 days before expiration |
| Uptime check | 2–3 consecutive failed checks |
Routing Alerts to the Right Channel
Not every alert needs to wake you up at 3 AM. Use severity-based routing:
- Critical (site down, security incident) — SMS/phone call, always
- Warning (approaching a threshold) — email or a messaging channel, reviewed during business hours
- Informational — logged, but no active notification
Writing Actionable Alert Messages
A good alert tells you what's wrong and where to look — not just "something is wrong." Include the specific server, metric, current value, and threshold that was crossed.
Testing Your Alerts
Deliberately trigger a test condition (fill disk temporarily in a controlled way, stop a test service) periodically to confirm your alerting pipeline actually works end-to-end — a silent monitoring failure is worse than no monitoring at all, since it creates false confidence.
Reviewing and Tuning Over Time
Periodically review which alerts fired and whether they were genuinely actionable — if a specific alert consistently turns out to be a false positive, adjust its threshold or remove it rather than continuing to receive noise.
Common Mistakes
- Alerting on every single metric without prioritization
- Setting thresholds so tight that normal, expected variation triggers constant alerts
- Never testing whether the alerting pipeline itself actually works
- Sending every alert to the same urgent channel regardless of severity
Related Articles
- How to Set Up Uptime Monitoring for Your Website
- How to Install Netdata for Real-Time VPS Monitoring
- How to Plan VPS Capacity for Future Growth
