Opscraft
Kubernetes troubleshooting · Lesson 11 of 11 · 4 min read

Incident stories and interview answers

How to tell a Kubernetes incident story

Interviewers will ask "tell me about a Kubernetes incident you handled". Use this structure (STAR with technical detail):

  1. Situation — service, impact (who was affected, error rate, duration), how it was detected (alert, customer).
  2. Diagnosis — the actual commands and evidence: "kubectl get pods showed restarts; describe showed OOMKilled with exit 137; memory graphs showed a steady climb since the deploy."
  3. Mitigation — what stopped the bleeding: rollback via Git revert, scaling up, raising a limit temporarily, cordoning a bad node.
  4. Root cause — the real reason (a cache with no eviction introduced in release X).
  5. Prevention — what changed afterwards: memory alerts at 90% of limit, a load test in CI, canary analysis on memory, a runbook.

Keep it to about two minutes, use numbers, and say "I" for your actions.

Enlarge diagramSituationimpact, how itwas detectedDiagnosiscommands andevidenceMitigationwhat stopped thebleedingRoot causethe real reasonPreventionalerts, tests,runbooks12345
Figure 1. A structure for telling an incident story in an interview: about two minutes, with numbers and the commands you ran.
Swipe sideways to see all of it, or tap to open it full screen.

"A pod is in CrashLoopBackOff. Walk me through it."

kubectl describe pod for the last state, reason and exit code and the events; kubectl logs --previous for why the last run died. Exit 1 points to an app error — config, env vars, dependencies; 137 with OOMKilled to memory limits; 127 to a wrong command; liveness-probe kills show in events. Then check recent changes — image, ConfigMaps, Secrets — and fix forward or roll back through Git.

"A pod is stuck in Pending."

Read the scheduler's FailedScheduling event: insufficient CPU or memory relative to requests, untolerated taints, unmatched affinity or topology constraints, unbound PVCs or volume AZ conflicts, max pods per node, or quota. Then fix the constraint, right-size requests, or find out why Karpenter or Cluster Autoscaler didn't add capacity — limits, instance requirements, subnet tags or Spot capacity.

"Production pods are healthy but users see errors."

Work along the request path: ALB metrics — ELB vs target 5xx — and access logs, ingress controller logs, Service endpoints and readiness, recent config or ingress changes, dependency health like the database or external APIs, and traces for failing requests. Healthy pods don't mean healthy traffic — a selector change, an ingress rule or a NetworkPolicy can break the path.

"Pods restart every few hours with OOMKilled."

Check memory over time to distinguish a leak from spikes, compare usage against the limit, confirm the runtime's memory settings fit the container — for example JVM MaxRAMPercentage — fix leaks with profiling or heap dumps, right-size the limit from observed usage, and alert before the limit is hit.

"A node is NotReady."

describe node for conditions; on the node, check kubelet and containerd status and logs, disk and memory pressure, and the CNI pod. Common causes are a hung runtime, a full disk, memory exhaustion, CNI failure or the instance being impaired. On a managed node group I'd cordon and drain, replace the node, then investigate from its logs.

"Someone kubectl-edited production. What happens with Flux?"

Flux reconciles the resource back to the state in Git on its next interval, so the manual change disappears. That's intended — Git is the source of truth. If the change was needed, it goes through a PR; in an emergency we suspend the Kustomization, fix, then commit and resume.

"How do you prevent noisy neighbours and make apps highly available?"

Requests and limits enforced via LimitRanges and policy, ResourceQuotas per namespace, PriorityClasses, dedicated node pools for special workloads. For HA: multiple replicas, PodDisruptionBudgets, topology spread across zones, readiness probes, HPA, and graceful shutdown.