Pods that crash or restart
CrashLoopBackOff
The container starts and exits repeatedly; Kubernetes backs off restarts (10 s, 20 s, 40 s... up to 5 min). It's a symptom, not a cause:
kubectl logs orders-7d9f-abc -n shop --previous # why did the LAST run die?
kubectl describe pod orders-7d9f-abc -n shop # Last State: Terminated, Reason, Exit Code
kubectl get pod orders-7d9f-abc -n shop -o jsonpath='{.status.containerStatuses[0].lastState}'
Exit codes
| Exit code | Meaning | Common causes |
|---|---|---|
| 0 | Exited successfully | The main process finished — wrong command, or a one-off task run as a Deployment (use a Job) |
| 1 | Application error | Exception at startup: bad config, missing env var, can't reach a dependency, migration failure |
| 2 | Misuse of shell builtins / bad arguments | Wrong command-line flags |
| 126 | Command not executable | Missing execute permission |
| 127 | Command not found | Wrong command/entrypoint path, binary missing in slim image |
| 128+n | Killed by signal n | — |
| 137 | 128+9 SIGKILL | OOMKilled (check Reason: OOMKilled), or killed after the termination grace period, or by liveness probe failure → kubelet restart |
| 139 | 128+11 SIGSEGV | Segfault — native code bug, arch/libc mismatch |
| 143 | 128+15 SIGTERM | Graceful termination (normal during rollouts/scale-down) |
OOMKilled
Reason: OOMKilled, exit 137 — the container exceeded its memory limit (cgroup OOM), even if the node has free memory.
- Look at memory over time (
kubectl top, Prometheuscontainer_memory_working_set_bytes): a steady climb → leak; a sudden spike → a large request, batch, or cache load. - Compare to the limit. For JVMs: heap + metaspace + threads + direct buffers must fit — use
-XX:MaxRAMPercentage=75rather than fixed-Xmxlarger than the container. - Fix: right-size limits from real usage (VPA recommendations help), fix leaks (heap dumps, profiling), stream instead of loading whole files, and alert at ~90% of limit.
Node-level OOM (the kernel killing processes because the node ran out of memory, often due to pods without limits or overcommitted limits) shows up in node dmesg and as kubelet evictions.
Liveness probes killing healthy apps
Events show Liveness probe failed: ... followed by Killing container. Causes: the probe hits a slow endpoint or one that checks dependencies (a database blip restarts every pod — a cascading failure), timeout too short (timeoutSeconds: 1 default) under load or GC pauses, or the app starts slowly and liveness begins before it's up.
Fixes: liveness should be a cheap "process is alive" check, not dependency checks; add a startup probe for slow starters (liveness waits until startup succeeds); raise timeoutSeconds/failureThreshold; use initialDelaySeconds only as a blunt fallback.
Other crash causes
- Config/secrets: missing env vars, wrong DB host, secret value with a trailing newline.
- Permissions:
readOnlyRootFilesystem: truebut the app writes to/tmp(mount anemptyDir); running as non-root but files owned by root in the image; volume ownership vsrunAsUser→ setfsGroup. - Dependencies at startup: the app exits if the DB isn't reachable — make it retry with backoff instead of crashing.
- Wrong architecture:
exec format error.