Opscraft
Observability: metrics, logs, traces · Lesson 2 of 7 · 8 min read

Prometheus in depth

Architecture

  • Pull model: Prometheus scrapes /metrics HTTP endpoints at intervals (typically 15–60 s). Short-lived batch jobs push to a Pushgateway (sparingly).
  • Service discovery: Kubernetes (pods, endpoints, services, nodes), EC2, Consul, file-based; relabelling filters targets and rewrites labels.
  • TSDB: local time-series database (2-hour blocks, WAL), retention by time or size; not clustered — one Prometheus is a single node.
  • Rules: recording rules precompute expensive queries; alerting rules fire alerts to Alertmanager.
  • Exporters: node_exporter (host), kube-state-metrics (Kubernetes object state), cAdvisor via kubelet (container resources), blackbox_exporter (probing HTTP/TCP/ICMP/DNS endpoints), database/redis/kafka exporters.
Enlarge diagramapp /metricsyour codenode_exporterhostkube-state-metricsK8s objectsPrometheusscrape every 30 s ·TSDB · rulesAlertmanagerroute, group, silenceGrafanadashboardsDashed arrows: Prometheus pulls (scrapes) from the targets.
Figure 1. Prometheus pulls metrics from targets it discovers, stores them locally, evaluates rules, and sends alerts to Alertmanager.
Swipe sideways to see all of it, or tap to open it full screen.

Metric types

Type Use Example
Counter Monotonically increasing (resets on restart) http_requests_total — always use rate()/increase()
Gauge Goes up and down memory_usage_bytes, queue_depth
Histogram Bucketed observations (_bucket, _sum, _count), aggregatable across instances http_request_duration_seconds
Summary Client-side quantiles, not aggregatable across instances Legacy use

Native histograms (exponential buckets) reduce the bucket-choice problem in newer Prometheus versions.

Enlarge diagramCounteronly goes up: wrap in rate()Gaugegoes up and down: memory, queue depthHistogrambuckets: percentiles across podsSummaryclient quantiles: cannot aggregate
Figure 2. The four metric types.
Swipe sideways to see all of it, or tap to open it full screen.

Labels and cardinality

Every unique combination of metric name + label values is a separate series. Labels like user_id, request_id, full URL paths or email addresses explode cardinality and memory. Use bounded labels (method, route template, status_code class, service). Investigate with the TSDB status page or topk(10, count by (__name__)({__name__=~".+"})).

Enlarge diagramBounded labelsUnbounded labelsmethod × route × status4 × 20 × 5 = 400 seriesCheap, fast queries+ user_id (1,000,000 users)400 million seriesMemory blows up
Figure 3. Cardinality: every unique label combination is a separate series. A label like user_id turns one metric into millions.
Swipe sideways to see all of it, or tap to open it full screen.

PromQL essentials

# request rate per service over 5 minutes
sum by (service) (rate(http_requests_total[5m]))

# error ratio (5xx / all)
sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
  /
sum by (service) (rate(http_requests_total[5m]))

# p99 latency from a histogram
histogram_quantile(0.99,
  sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))

# container memory vs limit
sum by (namespace, pod) (container_memory_working_set_bytes{container!=""})
  /
sum by (namespace, pod) (kube_pod_container_resource_limits{resource="memory"})

# CPU throttling ratio
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total[5m]))
  /
sum by (pod) (rate(container_cpu_cfs_periods_total[5m]))

# pods restarting
increase(kube_pod_container_status_restarts_total[1h]) > 3

# disk will fill within 4 hours
predict_linear(node_filesystem_avail_bytes{fstype!="tmpfs"}[6h], 4*3600) < 0

# target down
up == 0

Rules of thumb: rate() for per-second averages over a window (for alerts and graphs), irate() for very spiky instantaneous rates (graphs only), increase() for counts over a window. Aggregate after rate (sum(rate(x[5m])), not rate(sum(x))). The range should be at least 4× the scrape interval.

Kubernetes setup: kube-prometheus-stack

The Helm chart installs the Prometheus Operator, Prometheus, Alertmanager, Grafana, node_exporter, kube-state-metrics and default dashboards/rules. The operator's CRDs:

Enlarge diagramPrometheusselects ServiceMonitorsby labelServiceMonitorhas that label; selectsapp=ordersService ordersport name: metricspodsserve /metricsNo data? Check each link: labels, port name, path, NetworkPolicy, then Status → Targets.
Figure 4. How a new service gets scraped with the Prometheus Operator: a ServiceMonitor selects the Service, and Prometheus must select the ServiceMonitor.
Swipe sideways to see all of it, or tap to open it full screen.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: orders
  namespace: orders
  labels: { release: kube-prometheus-stack }   # must match Prometheus serviceMonitorSelector
spec:
  selector: { matchLabels: { app.kubernetes.io/name: orders } }
  endpoints:
    - port: http-metrics
      path: /actuator/prometheus
      interval: 30s
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: orders-alerts
  namespace: orders
  labels: { release: kube-prometheus-stack }
spec:
  groups:
    - name: orders.rules
      rules:
        - record: service:http_requests:rate5m
          expr: sum by (service) (rate(http_requests_total[5m]))
        - alert: OrdersHighErrorRate
          expr: |
            sum(rate(http_requests_total{service="orders",status=~"5.."}[5m]))
              / sum(rate(http_requests_total{service="orders"}[5m])) > 0.05
          for: 10m
          labels: { severity: page, team: orders }
          annotations:
            summary: "Orders 5xx ratio above 5% for 10m"
            runbook_url: https://runbooks.acme.dev/orders/high-error-rate
            dashboard: https://grafana.acme.dev/d/orders

Most common "my metrics aren't scraped" cause: the ServiceMonitor's labels don't match the Prometheus serviceMonitorSelector (or namespace selector), or the port name doesn't match the Service port name. Check the Prometheus Targets page.

Alertmanager

Receives alerts and handles grouping (one notification for many related alerts), inhibition (suppress warnings when a critical alert for the same thing fires; suppress pod alerts when the node is down), silences (maintenance), deduplication and routing:

Enlarge diagramGroupmany alerts → onemessageInhibitnode down hides its podalertsSilencemute duringmaintenanceRoutepage vs Slack by team,severity
Figure 5. Alertmanager turns a storm of alerts into a few useful notifications.
Swipe sideways to see all of it, or tap to open it full screen.
route:
  receiver: default-slack
  group_by: [alertname, service, namespace]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page"]
      receiver: pagerduty-oncall
    - matchers: [team="payments"]
      receiver: payments-slack
inhibit_rules:
  - source_matchers: [severity="page"]
    target_matchers: [severity="warning"]
    equal: [alertname, service]
receivers:
  - name: pagerduty-oncall
    pagerduty_configs: [{ routing_key: "<from secret>" }]
  - name: default-slack
    slack_configs: [{ channel: "#alerts", send_resolved: true }]
  - name: payments-slack
    slack_configs: [{ channel: "#payments-alerts" }]

Run Alertmanager as a cluster (3 replicas gossip) for HA; run two Prometheus replicas scraping the same targets — Alertmanager deduplicates their alerts.

Scaling and long-term storage

One Prometheus per cluster is fine until you need global queries, HA deduplication and long retention:

Option How
Thanos Sidecar uploads blocks to S3, Querier fans out and deduplicates, Store gateway reads S3, Compactor downsamples
Grafana Mimir / Cortex Prometheus remote_write to a horizontally scalable multi-tenant store on object storage
Amazon Managed Service for Prometheus (AMP) Managed Mimir-like backend; remote_write with SigV4 (IRSA); pair with Amazon Managed Grafana
VictoriaMetrics Efficient alternative TSDB, remote_write compatible

Agent mode (--agent in Prometheus 3, --enable-feature=agent in 2.x; or an OTel Collector or Grafana Alloy) scrapes and remote-writes without local querying — light footprint per cluster.

Enlarge diagramcluster 1Prometheus agentcluster 2Prometheus agentcluster 3Prometheus agentMimir / Thanos / AMPdeduplicated,multi-tenantS3long retentionGrafanaglobal queries
Figure 6. Scaling Prometheus: keep a light agent per cluster and send data to a central, long-term store on object storage.
Swipe sideways to see all of it, or tap to open it full screen.