Opscraft
Deployment strategies · Lesson 5 of 10 · 6 min read

Canary releases

The pattern

Send a small percentage of real traffic to the new version, compare its metrics with the stable version, and increase gradually — 1% → 5% → 25% → 50% → 100% — with automatic rollback if metrics degrade. The smallest initial blast radius of the common strategies.

Ways to split traffic

Enlarge diagram1%to v25%to v225%to v250%to v2100%to v2at every step: compare canary vs stable on errors, p99 latency, saturation → continue or roll back
Figure 1. A canary: a small share of real traffic tries the new version; metrics compared with the stable version decide whether to go further.
Swipe sideways to see all of it, or tap to open it full screen.
Enlarge diagramReplica ratioALB weighted target groupsGateway API / NGINX canaryService meshRoute 53 weights9 : 1 pods behind one Service: coarse95 / 5, exactexact weights, header targetingper request, headers, metrics + mTLSskewed by resolver caches: avoid
Figure 2. Ways to split traffic for a canary, from coarse to exact.
Swipe sideways to see all of it, or tap to open it full screen.
Method Precision Notes
Replica ratio (two Deployments behind one Service, e.g. 9 stable : 1 canary) Coarse (depends on pod counts) No extra tooling; split is per-connection, not per-request with keep-alive
NGINX Ingress canary annotations Exact weights; header/cookie targeting nginx.ingress.kubernetes.io/canary: "true", canary-weight: "5", canary-by-header: "X-Canary" (common in existing clusters; ingress-nginx was retired in March 2026)
ALB weighted target groups (AWS LB Controller actions annotation) Exact weights Traffic split at the ALB across two target groups
Gateway API HTTPRoute backendRefs weights Exact Vendor-neutral successor to Ingress
Service mesh (Istio VirtualService, Linkerd/SMI) Exact, per-request, header-based Plus rich metrics and mTLS
Route 53 weighted records Imprecise Resolver caching skews the split — avoid for canaries
CodeDeploy (Canary10Percent5Minutes, Linear10PercentEvery1Minute) Exact ECS and Lambda (alias weights)

ALB weighted example via the AWS Load Balancer Controller:

metadata:
  annotations:
    alb.ingress.kubernetes.io/actions.orders-split: >
      {"type":"forward","forwardConfig":{"targetGroups":[
        {"serviceName":"orders-stable","servicePort":"80","weight":95},
        {"serviceName":"orders-canary","servicePort":"80","weight":5}]}}
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend: { service: { name: orders-split, port: { name: use-annotation } } }

What to measure

Compare canary vs stable baseline (not vs last week) on:

Enlarge diagramError rate5xx, failed paymentsLatencyp95 / p99, not theaverageSaturationCPU, memory, pools,queuesBusinessorders per minute,conversion
Figure 3. What to compare between canary and stable. A canary with fine errors but doubled p99 latency should be stopped.
Swipe sideways to see all of it, or tap to open it full screen.
  • Error rate — 5xx ratio, failed business operations (payment failures, checkout errors).
  • Latency percentiles — p95/p99, not just averages. A canary with fine error rates but doubled p99 latency should be aborted: tail latency is what real users feel.
  • Saturation — CPU, memory, connection pools, queue depth.
  • Business/KPI signals — conversion, orders per minute — when traffic is enough to be significant.

Practical challenges: at 1% traffic, low-volume services don't generate statistically meaningful data — use longer steps, header-based canaries for internal users first, or synthetic traffic. Also make sure metrics are labelled by version so you can compare.

Automated canaries with Flagger (fits Flux)

Flagger is a Flux-family project that automates canary analysis:

Enlarge diagramNew imageFlux applies itCanary 10%Flagger shifts weightAnalysisPrometheus: errors, latencyStep up+10% each intervalPromotecopy to primaryOr roll backafter N failed checks123456
Figure 4. Automated canary analysis: the tool shifts weight in steps, checks metrics each step, and promotes or rolls back without a human.
Swipe sideways to see all of it, or tap to open it full screen.
apiVersion: flagger.app/v1beta1
kind: Canary
metadata: { name: orders, namespace: shop }
spec:
  provider: nginx                       # or istio, appmesh, gatewayapi, linkerd...
  targetRef: { apiVersion: apps/v1, kind: Deployment, name: orders }
  ingressRef: { apiVersion: networking.k8s.io/v1, kind: Ingress, name: orders }
  service: { port: 80, targetPort: 8080 }
  analysis:
    interval: 1m
    threshold: 5                        # failed checks before rollback
    maxWeight: 50
    stepWeight: 10
    metrics:
    - name: request-success-rate
      thresholdRange: { min: 99 }
      interval: 1m
    - name: request-duration           # p99 ms
      thresholdRange: { max: 500 }
      interval: 1m
    webhooks:
    - name: load-test
      url: http://flagger-loadtester.test/
      metadata: { cmd: "hey -z 1m -q 10 -c 2 http://orders-canary.shop/" }

When Flux applies a new image tag, Flagger detects the change, creates a canary, shifts weight in steps while querying Prometheus, and either promotes or rolls back automatically — no human needed.

Argo Rollouts

Replaces the Deployment with a Rollout resource:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      canaryService: orders-canary
      stableService: orders-stable
      trafficRouting:
        alb: { ingress: orders, servicePort: 80 }
      steps:
      - setWeight: 5
      - pause: { duration: 10m }
      - analysis: { templates: [{ templateName: success-rate }] }
      - setWeight: 25
      - pause: { duration: 10m }
      - setWeight: 50
      - pause: {}                        # indefinite: wait for manual promotion

An AnalysisTemplate queries Prometheus/CloudWatch/Datadog and fails the rollout (automatic abort to stable) when thresholds are breached.