Interview questions and model answers
"How do you monitor a microservice on Kubernetes?"
Expose Prometheus metrics with RED signals and histograms, scraped by kube-prometheus-stack through a ServiceMonitor; platform metrics from node_exporter, kube-state-metrics and cAdvisor; structured JSON logs to stdout shipped by Fluent Bit to Loki or CloudWatch with trace IDs; OpenTelemetry auto-instrumentation sending traces to Tempo or X-Ray. Grafana dashboards per service, SLO-based burn-rate alerts routed by Alertmanager to the owning team with runbooks, and blackbox or synthetic checks from outside.
"Counter vs gauge vs histogram?"
A counter only increases and resets on restart, so you always apply rate or increase. A gauge goes up and down, like memory or queue depth. A histogram counts observations into buckets plus sum and count, so you can compute percentiles with histogram_quantile and aggregate across instances, which summaries can't do.
"What is high cardinality and why does it matter?"
Each unique label combination is a separate time series. Labels with unbounded values — user IDs, request IDs, raw URLs — multiply series until Prometheus runs out of memory and queries slow down. Keep labels bounded, drop or relabel bad ones, and put high-cardinality details in logs or traces instead.
"Metrics aren't showing up for a new service. Debug."
Check the Prometheus Targets page: if the target is missing, the ServiceMonitor labels don't match the Prometheus selector, the namespace isn't selected, or the selector doesn't match the Service labels; if it's down, check the port name, path, network policy and that the app actually exposes /metrics. Then query up{job="..."}.
"How do you reduce alert fatigue?"
Inventory alerts and measure actionability. Page only on user-facing symptoms using SLO burn rates; demote cause-based alerts to tickets or dashboards; add for durations, grouping and inhibition in Alertmanager; ensure every page has an owner and runbook; and review noisy alerts weekly with on-call.
"How would you scale Prometheus for 20 clusters with one year of retention?"
Keep a lightweight Prometheus or agent per cluster and remote-write to a central store — Mimir, Thanos, or Amazon Managed Prometheus on S3-backed storage — with per-cluster external labels, HA pairs deduplicated, downsampling for older data, and recording rules for heavy queries. Grafana queries the global view.
"Explain head-based vs tail-based sampling."
Head-based decides at the start of the trace, which is cheap but may drop the rare errors you care about. Tail-based waits until the trace is complete in a collector and keeps all errors and slow traces plus a percentage of the rest, at the cost of buffering in an OpenTelemetry Collector gateway tier.
"Loki vs OpenSearch for logs?"
Loki indexes only labels and stores compressed chunks in object storage, so it's much cheaper and pairs naturally with Prometheus and Grafana, but full-text search over large ranges is slower and high-cardinality labels hurt it. OpenSearch indexes content for fast, rich search and analytics at higher cost and operational effort. I'd choose based on query patterns and budget, often Loki for application logs and OpenSearch where security analytics need full-text search.