Prometheus in depth
Architecture
- Pull model: Prometheus scrapes
/metricsHTTP endpoints at intervals (typically 15–60 s). Short-lived batch jobs push to a Pushgateway (sparingly). - Service discovery: Kubernetes (pods, endpoints, services, nodes), EC2, Consul, file-based; relabelling filters targets and rewrites labels.
- TSDB: local time-series database (2-hour blocks, WAL), retention by time or size; not clustered — one Prometheus is a single node.
- Rules: recording rules precompute expensive queries; alerting rules fire alerts to Alertmanager.
- Exporters: node_exporter (host), kube-state-metrics (Kubernetes object state), cAdvisor via kubelet (container resources), blackbox_exporter (probing HTTP/TCP/ICMP/DNS endpoints), database/redis/kafka exporters.
Metric types
| Type | Use | Example |
|---|---|---|
| Counter | Monotonically increasing (resets on restart) | http_requests_total — always use rate()/increase() |
| Gauge | Goes up and down | memory_usage_bytes, queue_depth |
| Histogram | Bucketed observations (_bucket, _sum, _count), aggregatable across instances |
http_request_duration_seconds |
| Summary | Client-side quantiles, not aggregatable across instances | Legacy use |
Native histograms (exponential buckets) reduce the bucket-choice problem in newer Prometheus versions.
Labels and cardinality
Every unique combination of metric name + label values is a separate series. Labels like user_id, request_id, full URL paths or email addresses explode cardinality and memory. Use bounded labels (method, route template, status_code class, service). Investigate with the TSDB status page or topk(10, count by (__name__)({__name__=~".+"})).
PromQL essentials
# request rate per service over 5 minutes
sum by (service) (rate(http_requests_total[5m]))
# error ratio (5xx / all)
sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/
sum by (service) (rate(http_requests_total[5m]))
# p99 latency from a histogram
histogram_quantile(0.99,
sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))
# container memory vs limit
sum by (namespace, pod) (container_memory_working_set_bytes{container!=""})
/
sum by (namespace, pod) (kube_pod_container_resource_limits{resource="memory"})
# CPU throttling ratio
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total[5m]))
/
sum by (pod) (rate(container_cpu_cfs_periods_total[5m]))
# pods restarting
increase(kube_pod_container_status_restarts_total[1h]) > 3
# disk will fill within 4 hours
predict_linear(node_filesystem_avail_bytes{fstype!="tmpfs"}[6h], 4*3600) < 0
# target down
up == 0
Rules of thumb: rate() for per-second averages over a window (for alerts and graphs), irate() for very spiky instantaneous rates (graphs only), increase() for counts over a window. Aggregate after rate (sum(rate(x[5m])), not rate(sum(x))). The range should be at least 4× the scrape interval.
Kubernetes setup: kube-prometheus-stack
The Helm chart installs the Prometheus Operator, Prometheus, Alertmanager, Grafana, node_exporter, kube-state-metrics and default dashboards/rules. The operator's CRDs:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: orders
namespace: orders
labels: { release: kube-prometheus-stack } # must match Prometheus serviceMonitorSelector
spec:
selector: { matchLabels: { app.kubernetes.io/name: orders } }
endpoints:
- port: http-metrics
path: /actuator/prometheus
interval: 30s
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: orders-alerts
namespace: orders
labels: { release: kube-prometheus-stack }
spec:
groups:
- name: orders.rules
rules:
- record: service:http_requests:rate5m
expr: sum by (service) (rate(http_requests_total[5m]))
- alert: OrdersHighErrorRate
expr: |
sum(rate(http_requests_total{service="orders",status=~"5.."}[5m]))
/ sum(rate(http_requests_total{service="orders"}[5m])) > 0.05
for: 10m
labels: { severity: page, team: orders }
annotations:
summary: "Orders 5xx ratio above 5% for 10m"
runbook_url: https://runbooks.acme.dev/orders/high-error-rate
dashboard: https://grafana.acme.dev/d/orders
Most common "my metrics aren't scraped" cause: the ServiceMonitor's labels don't match the Prometheus serviceMonitorSelector (or namespace selector), or the port name doesn't match the Service port name. Check the Prometheus Targets page.
Alertmanager
Receives alerts and handles grouping (one notification for many related alerts), inhibition (suppress warnings when a critical alert for the same thing fires; suppress pod alerts when the node is down), silences (maintenance), deduplication and routing:
route:
receiver: default-slack
group_by: [alertname, service, namespace]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- matchers: [severity="page"]
receiver: pagerduty-oncall
- matchers: [team="payments"]
receiver: payments-slack
inhibit_rules:
- source_matchers: [severity="page"]
target_matchers: [severity="warning"]
equal: [alertname, service]
receivers:
- name: pagerduty-oncall
pagerduty_configs: [{ routing_key: "<from secret>" }]
- name: default-slack
slack_configs: [{ channel: "#alerts", send_resolved: true }]
- name: payments-slack
slack_configs: [{ channel: "#payments-alerts" }]
Run Alertmanager as a cluster (3 replicas gossip) for HA; run two Prometheus replicas scraping the same targets — Alertmanager deduplicates their alerts.
Scaling and long-term storage
One Prometheus per cluster is fine until you need global queries, HA deduplication and long retention:
| Option | How |
|---|---|
| Thanos | Sidecar uploads blocks to S3, Querier fans out and deduplicates, Store gateway reads S3, Compactor downsamples |
| Grafana Mimir / Cortex | Prometheus remote_write to a horizontally scalable multi-tenant store on object storage |
| Amazon Managed Service for Prometheus (AMP) | Managed Mimir-like backend; remote_write with SigV4 (IRSA); pair with Amazon Managed Grafana |
| VictoriaMetrics | Efficient alternative TSDB, remote_write compatible |
Agent mode (--agent in Prometheus 3, --enable-feature=agent in 2.x; or an OTel Collector or Grafana Alloy) scrapes and remote-writes without local querying — light footprint per cluster.