The four golden signals — latency, traffic, errors, and saturation — provide a widely-adopted framework for what to monitor for any service. This guide covers understanding and applying this framework.
Where This Framework Comes From
Popularized by Google's Site Reliability Engineering practices, the four golden signals represent a distilled, broadly applicable answer to "what should I actually monitor" — a genuinely useful starting framework rather than an exhaustive prescription.
Signal 1: Latency
How long requests take to complete — importantly, distinguish between successful request latency and failed request latency (a fast failure isn't the same as a fast success); see How to Reduce Time to First Byte (TTFB) and general percentile-based measurement principles from How to Set Up Real User Monitoring (RUM) for Your Website.
Signal 2: Traffic
The demand placed on your system — requests per second for a web service, but the appropriate traffic measure varies by service type (transactions per second for a database, concurrent connections for a streaming service).
Signal 3: Errors
The rate of failed requests — both explicit failures (5xx responses) and implicit failures (a response that technically succeeds but returns genuinely wrong/incomplete data); the latter is easy to overlook but equally important.
Signal 4: Saturation
How "full" your service is relative to its capacity — CPU/memory utilization, queue depth, connection pool usage; saturation often predicts future problems before they manifest as latency/error issues.
Applying the Framework to a Web Application
| Signal | Example Metric |
|---|---|
| Latency | p50/p95/p99 response time |
| Traffic | Requests per second |
| Errors | 5xx error rate percentage |
| Saturation | CPU utilization, connection pool usage |
Applying the Framework to a Database
| Signal | Example Metric |
|---|---|
| Latency | Query execution time |
| Traffic | Queries per second |
| Errors | Failed query rate, connection errors |
| Saturation | Connection pool utilization, disk I/O utilization |
Building Dashboards Around These Signals
See How to Set Up Grafana Dashboards for Multi-Service Observability — structuring dashboards consistently around these four signals across different services gives you a genuinely comparable, predictable view rather than ad-hoc, inconsistent per-service dashboards.
Setting Alerts Based on These Signals
See How to Set Up Effective Server Alerting (Without Alert Fatigue) — these four signals are a reasonable starting point for what genuinely warrants alerting, since they directly reflect user-impacting conditions rather than purely internal implementation details.
Recognizing This Framework's Limitations
The four golden signals are a useful starting framework, not an exhaustive prescription — some services have genuinely important signals beyond these four (business metrics, specific domain-relevant indicators); use this as a foundation, not a ceiling.
Common Errors
Monitoring dashboard exists but doesn't clearly reveal genuine problems — verify you're genuinely covering all four signals, not just the ones easiest to measure (traffic and latency are often instrumented first, while saturation is sometimes overlooked despite being equally important).
Continue Reading
- How to Set Up Grafana Dashboards for Multi-Service Observability
- How to Set Up Effective Server Alerting (Without Alert Fatigue)
- How to Define and Track SLOs and Error Budgets
Browse more articles in Advanced Observability & Incident Management.
