Interview questions and model answers
"Explain blue-green vs canary. Which would you use for a payments API?"
Blue-green runs two full environments and switches all traffic at once, giving instant rollback but doubling capacity and exposing everyone at once. Canary sends a small share of traffic to the new version and increases it while comparing error rate and latency against the stable version. For payments I'd use a canary with automated analysis — Flagger or Argo Rollouts — with strict thresholds on error rate, p99 latency and payment success rate, automatic abort, feature flags for new behaviour, and expand/contract database changes so rollback is always safe.
"How do you achieve zero-downtime rolling updates in Kubernetes?"
Readiness probes so only ready pods receive traffic, maxUnavailable: 0 with a surge so capacity never drops, minReadySeconds to catch flapping pods, a preStop sleep so endpoints and the ALB deregister before SIGTERM, an app that drains in-flight requests on SIGTERM within the grace period, ALB pod readiness gates with IP targets, matching keep-alive and idle timeouts, PDBs for node maintenance, and enough cluster headroom for surge pods.
"A rolling update caused a burst of 502s. What happened?"
Likely the termination race: pods received SIGTERM and exited while the ALB and kube-proxy were still routing to them — no preStop delay or no graceful shutdown. Or new pods received traffic before they were ready because there was no readiness probe or readiness gate. Or the app's keep-alive timeout was shorter than the ALB's idle timeout. I'd check which by looking at timing in ALB logs (target response codes vs ELB codes) and fix each piece.
"How do you deploy a database schema change with zero downtime?"
Expand/contract: add new schema in a backward-compatible way, deploy code that works with both old and new — dual writing and backfilling — switch reads, and only remove the old schema in a later release once no running or rollback-able version needs it. Migrations run as a separate idempotent step, avoid long locks, and I snapshot before risky changes.
"Your canary's error rate is fine but p99 latency doubled. Promote?"
No — abort. Tail latency affects real users and often signals resource contention or a slow dependency path that will get worse at 100% traffic. I'd roll back, investigate with traces comparing canary and stable, and include p99 in the automated analysis thresholds.
"How do you roll back in a GitOps setup with Flux?"
Revert the commit that changed the image tag or manifest and let Flux reconcile — that keeps Git as the source of truth and leaves an audit trail. kubectl rollout undo would be overwritten by Flux on the next reconcile. In an emergency I can flux suspend the Kustomization, roll back manually, then revert in Git and resume. With Flagger, a failed canary rolls back automatically.
"Feature flags vs canary releases?"
Canaries shift traffic between versions and catch runtime regressions — memory leaks, dependency or config problems. Flags toggle features inside one version per user segment, separating release from deploy and giving an instant kill switch without redeploying. I use both: canary for every deploy, flags for risky features — and I manage flag debt with owners and expiry.
"When would you use Recreate?"
When two versions can't run at the same time — an exclusive resource, an incompatible protocol, a migration that must complete before the new version starts — and brief downtime is acceptable, like internal tools or singleton batch workers.