Canary releases
The pattern
Send a small percentage of real traffic to the new version, compare its metrics with the stable version, and increase gradually — 1% → 5% → 25% → 50% → 100% — with automatic rollback if metrics degrade. The smallest initial blast radius of the common strategies.
Ways to split traffic
| Method | Precision | Notes |
|---|---|---|
| Replica ratio (two Deployments behind one Service, e.g. 9 stable : 1 canary) | Coarse (depends on pod counts) | No extra tooling; split is per-connection, not per-request with keep-alive |
| NGINX Ingress canary annotations | Exact weights; header/cookie targeting | nginx.ingress.kubernetes.io/canary: "true", canary-weight: "5", canary-by-header: "X-Canary" (common in existing clusters; ingress-nginx was retired in March 2026) |
| ALB weighted target groups (AWS LB Controller actions annotation) | Exact weights | Traffic split at the ALB across two target groups |
Gateway API HTTPRoute backendRefs weights |
Exact | Vendor-neutral successor to Ingress |
| Service mesh (Istio VirtualService, Linkerd/SMI) | Exact, per-request, header-based | Plus rich metrics and mTLS |
| Route 53 weighted records | Imprecise | Resolver caching skews the split — avoid for canaries |
CodeDeploy (Canary10Percent5Minutes, Linear10PercentEvery1Minute) |
Exact | ECS and Lambda (alias weights) |
ALB weighted example via the AWS Load Balancer Controller:
metadata:
annotations:
alb.ingress.kubernetes.io/actions.orders-split: >
{"type":"forward","forwardConfig":{"targetGroups":[
{"serviceName":"orders-stable","servicePort":"80","weight":95},
{"serviceName":"orders-canary","servicePort":"80","weight":5}]}}
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend: { service: { name: orders-split, port: { name: use-annotation } } }
What to measure
Compare canary vs stable baseline (not vs last week) on:
- Error rate — 5xx ratio, failed business operations (payment failures, checkout errors).
- Latency percentiles — p95/p99, not just averages. A canary with fine error rates but doubled p99 latency should be aborted: tail latency is what real users feel.
- Saturation — CPU, memory, connection pools, queue depth.
- Business/KPI signals — conversion, orders per minute — when traffic is enough to be significant.
Practical challenges: at 1% traffic, low-volume services don't generate statistically meaningful data — use longer steps, header-based canaries for internal users first, or synthetic traffic. Also make sure metrics are labelled by version so you can compare.
Automated canaries with Flagger (fits Flux)
Flagger is a Flux-family project that automates canary analysis:
apiVersion: flagger.app/v1beta1
kind: Canary
metadata: { name: orders, namespace: shop }
spec:
provider: nginx # or istio, appmesh, gatewayapi, linkerd...
targetRef: { apiVersion: apps/v1, kind: Deployment, name: orders }
ingressRef: { apiVersion: networking.k8s.io/v1, kind: Ingress, name: orders }
service: { port: 80, targetPort: 8080 }
analysis:
interval: 1m
threshold: 5 # failed checks before rollback
maxWeight: 50
stepWeight: 10
metrics:
- name: request-success-rate
thresholdRange: { min: 99 }
interval: 1m
- name: request-duration # p99 ms
thresholdRange: { max: 500 }
interval: 1m
webhooks:
- name: load-test
url: http://flagger-loadtester.test/
metadata: { cmd: "hey -z 1m -q 10 -c 2 http://orders-canary.shop/" }
When Flux applies a new image tag, Flagger detects the change, creates a canary, shifts weight in steps while querying Prometheus, and either promotes or rolls back automatically — no human needed.
Argo Rollouts
Replaces the Deployment with a Rollout resource:
apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
strategy:
canary:
canaryService: orders-canary
stableService: orders-stable
trafficRouting:
alb: { ingress: orders, servicePort: 80 }
steps:
- setWeight: 5
- pause: { duration: 10m }
- analysis: { templates: [{ templateName: success-rate }] }
- setWeight: 25
- pause: { duration: 10m }
- setWeight: 50
- pause: {} # indefinite: wait for manual promotion
An AnalysisTemplate queries Prometheus/CloudWatch/Datadog and fails the rollout (automatic abort to stable) when thresholds are breached.