Opscraft
Deployment strategies · Lesson 8 of 10 · 4 min read

Rollback, verification and multi-region rollouts

Verify every deployment

kubectl rollout status only proves pods are running. A deployment is successful when:

  • Health and smoke tests pass (key endpoints, a synthetic transaction).
  • SLO metrics — error rate, latency, saturation — stay within thresholds over a bake time.
  • Business metrics don't drop.
  • No new error signatures in logs/traces.

A pipeline that's green while production is broken is missing post-deploy verification.

Enlarge diagramPods readyrollout statusSmoke testskey endpoints, asynthetic transactionBakeSLO metrics withinlimitsBusinessorders still flowing1234
Figure 1. A deployment is done when production looks healthy, not when pods are running.
Swipe sideways to see all of it, or tap to open it full screen.

Rollback design

  • Keep previous artifacts and revisions — immutable image tags, revisionHistoryLimit, Helm release history, Git history.
  • Automate it — Argo Rollouts/Flagger abort automatically; CodeDeploy rolls back on CloudWatch alarms; ECS circuit breaker; pipeline jobs that revert the GitOps commit.
  • Make it boring — practise rollbacks; time them.
  • Decide in advance — pre-agreed rollback triggers (error rate > X% for Y minutes). During an incident the on-call/incident lead rolls back first and investigates afterwards; you don't need root cause to roll back.
  • Know when rollback isn't possible — irreversible data migrations, external side effects (emails sent, payments captured). Then roll forward with a fix, or use flags to disable the feature.

Rollback commands by tool:

kubectl rollout undo deploy/orders                  # plain Kubernetes (temporary under GitOps)
helm rollback orders 41                             # Helm release revision
git revert <commit> && git push                     # GitOps — Flux reconciles the old state
flux suspend kustomization orders                   # emergency: stop reconciliation while you act
kubectl argo rollouts abort orders                  # Argo Rollouts
aws deploy stop-deployment --deployment-id d-XXXX --auto-rollback-enabled   # CodeDeploy
Enlarge diagramKubernetesHelmGitOps / FluxArgo RolloutsCodeDeploykubectl rollout undo (temporary under GitOps)helm rollback orders 41git revert, Flux reconcileskubectl argo rollouts abortauto-rollback on alarms
Figure 2. Rollback by tool. Under GitOps, revert in Git; kubectl rollout undo is overwritten at the next reconcile.
Swipe sideways to see all of it, or tap to open it full screen.

Multi-region and large fleets

  • Deploy to one region (ideally the smallest or a canary region) first, bake, then the next — never all regions at once.
  • Within a region, roll AZ by AZ or cell by cell for very large systems (cell-based architecture limits blast radius).
  • Keep regions on compatible versions; a cross-region API or data replication format change must be backward compatible.
  • Automate the wave plan in the pipeline with gates between waves.
Enlarge diagramcanary regionsmallest, e.g.ap-south-2bakewatch SLOsregion 2AZ by AZremainingregionsgated waves
Figure 3. Rolling out across regions in waves: one region, bake, then the next. Never all at once.
Swipe sideways to see all of it, or tap to open it full screen.