Deployment strategies · Lesson 3 of 10 · 4 min read
Zero-downtime mechanics
A rolling update with the default settings can still drop requests. Zero downtime needs the whole request path to cooperate.
Getting traffic only to ready pods
- Readiness probe — removes a pod from Service endpoints until it can serve (warm caches, DB connections established). Without it, traffic hits pods that are still starting → 502/503s.
- Startup probe — for slow starters, so liveness doesn't kill them during boot.
- With the AWS Load Balancer Controller and
target-type: ip, use pod readiness gates (label the namespaceelbv2.k8s.aws/pod-readiness-gate-inject=enabled) so a pod isn't considered ready until the ALB target group health check passes — otherwise old pods may be removed before new ones are healthy in the ALB.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Graceful shutdown: the termination sequence
When a pod is deleted during a rollout, two things happen in parallel:
- The kubelet runs the
preStophook, then sends SIGTERM to the containers. - The endpoint controller removes the pod from Service endpoints; kube-proxy on every node and the ALB (via the LB controller) update — this takes seconds.
If the app exits immediately on SIGTERM, requests still being routed to it during step 2 fail. The fix:
spec:
terminationGracePeriodSeconds: 45
containers:
- name: orders
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 15"] # let endpoints/ALB deregister first
Then the app should, on SIGTERM: stop accepting new connections, finish in-flight requests, close DB connections, exit. Total time must stay under terminationGracePeriodSeconds (default 30 s), after which SIGKILL arrives.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
On the ALB, the target group deregistration delay (default 300 s) controls how long draining connections are allowed; tune it to your request durations (e.g. 30–60 s) so rollouts aren't needlessly slow.
Other pieces
- PodDisruptionBudgets protect availability during node drains and cluster upgrades (not during Deployment rollouts — those use maxUnavailable).
- Enough capacity for surge pods: cluster autoscaler/Karpenter headroom, or rollouts stall in Pending.
- Session state outside pods (Redis, JWT) — sticky sessions pin users to old pods and break when they disappear.
- Long-lived connections (WebSockets, gRPC streams): longer grace periods, server-sent "go away"/reconnect signals, clients that reconnect with backoff.
- Brief 502s during rollouts almost always mean one of: no readiness probe, no preStop delay, app not handling SIGTERM, keep-alive timeout mismatch between ALB and app, or deregistration slower than pod termination.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.