Observability: metrics, logs, traces · Lesson 6 of 7 · 3 min read
Alerting and dashboards that work
Alerting principles
- Page on symptoms, not causes: users seeing errors/latency (SLO burn), not "CPU 85%". Cause-based alerts become tickets or dashboard signals.
- Every page must be actionable, urgent and have a runbook link and dashboard; otherwise it's a ticket or nothing.
- Use
for:durations and SLO burn-rate alerts (see the SRE topic) to avoid flapping. - Severity routing: page (wake someone), ticket (business hours), info (dashboard only).
- Track alert quality: pages per week per on-call, % actionable, time to acknowledge; delete or fix noisy alerts in a regular review.
- Alert on absence:
absent(up{job="orders"}), dead-man's-switch (a "Watchdog" alert that always fires to an external service — silence means the alerting pipeline is broken).
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Dashboards
- Top-down hierarchy: service overview (RED + SLO) → service detail → dependencies → infrastructure (USE).
- Grafana practices: variables (cluster, namespace, service), consistent units, annotations for deploys, dashboards as code (Grafonnet/JSON in Git, provisioned via sidecar ConfigMaps or Terraform), links between metrics → logs (Loki) → traces (Tempo).
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Cost control
Observability bills can rival compute. Controls: drop unused metrics at scrape (metric_relabel_configs with action: drop), reduce label cardinality, recording rules + shorter raw retention, log sampling and level hygiene, exclude health-check logs, retention tiers, trace sampling, per-team usage visibility.