Observability foundations
"How do you monitor it?" follows every design question in a Senior DevOps interview. You need to explain metrics, logs and traces and how they connect; run Prometheus and Grafana on Kubernetes (kube-prometheus-stack, PromQL, recording and alerting rules, Alertmanager routing, scaling with Thanos/Mimir/AMP); ship logs with Fluent Bit to Loki/OpenSearch/CloudWatch; instrument with OpenTelemetry; use CloudWatch well on AWS; and design alerts that page for real user impact instead of noise.
Monitoring vs observability
Monitoring answers known questions ("is CPU above 80%?"). Observability is the ability to answer new questions about system behaviour from its outputs — essential for distributed systems where failures are novel. In practice you need both: dashboards and alerts for known failure modes, and rich, correlated telemetry to debug unknown ones.
The three signals (plus one)
| Signal | What | Strength | Cost profile |
|---|---|---|---|
| Metrics | Numeric time series with labels | Cheap aggregation, alerting, trends | Cost grows with cardinality (unique label combinations) |
| Logs | Timestamped event records | Detail, context, audit | Cost grows with volume |
| Traces | A request's path across services as spans | Latency breakdown, dependencies | Cost grows with request volume — sample |
| Profiles | CPU/memory profiles over time (continuous profiling: Pyroscope, Parca) | Why code is slow | Moderate |
Correlation is the goal: a latency alert (metric) → an exemplar trace ID on the histogram → the slow span → logs with the same trace_id.
Methods: what to measure
- Four golden signals (Google SRE): latency, traffic, errors, saturation.
- RED for request-driven services: Rate, Errors, Duration.
- USE for resources: Utilisation, Saturation, Errors (CPU, memory, disk, network, connection pools).
Measure latency with percentiles (p50/p95/p99) from histograms — averages hide tail latency. And measure error rates distinguishing client errors (4xx, often not your fault) from server errors (5xx).