HTTP & status codes · Lesson 6 of 9 · 5 min read
Troubleshooting 502, 503 and 504 on AWS and Kubernetes
502 Bad Gateway
Common causes behind an ALB or ingress:
- Keep-alive timeout race — the app's keep-alive timeout is shorter than the ALB's idle timeout (60 s default); the app closes an idle connection just as the ALB reuses it → ALB 502 ("target closed connection",
target_status_code -). Fix: app/server keep-alive timeout greater than the ALB idle timeout (e.g. 65–75 s; Node.jsserver.keepAliveTimeoutdefault is only 5 s; gunicorn--keep-alive). - App crash or restart mid-request (OOMKilled pods, CrashLoopBackOff) — connections reset.
- Pods terminating during rollouts without a preStop delay/graceful shutdown → resets.
- Wrong port or protocol — target group says HTTP:8080 but the app listens on 3000, or the ALB speaks HTTP to an HTTPS-only backend (or HTTPS to HTTP).
- Malformed responses — headers too large for the proxy buffer (nginx "upstream sent too big header" → raise
proxy_buffer_size), invalid HTTP. - Lambda targets returning invalid responses or erroring.
503 Service Unavailable
- No healthy targets in the target group (ALB returns 503 itself): health check path/port wrong, expected code mismatch (app redirects
/→/loginwith 302 but check expects 200; app returns 301 to HTTPS on the health path), security group doesn't allow the ALB, app not ready, all pods crashing. - Kubernetes Service has no ready endpoints — ingress-nginx returns 503: selector doesn't match pod labels, readiness probes failing, wrong
targetPort. - During deploys only — all targets draining or not ready at once: rolling update with
maxUnavailabletoo high, missing readiness probes, not enough capacity, ALB readiness gates missing. - Overload / throttling — app or upstream deliberately shedding load, circuit breakers open, API Gateway/Lambda concurrency limits.
- Maintenance mode — intentional, with
Retry-After.
504 Gateway Timeout
- Requests longer than the proxy's timeout: ALB idle timeout (60 s) → 504 after exactly 60 s is the giveaway; nginx
proxy_read_timeout(60 s default); API Gateway REST integration timeout 29 s; CloudFront origin response timeout (30 s default). - Slow upstream — slow DB queries, a downstream API hanging, thread pool exhaustion, GC pauses.
- Network path blocked to the target (SG/NACL change) — the proxy waits for a connection that never completes.
Fixes: find why the request is slow (traces, DB slow query logs); for legitimately long work (reports, exports), make it asynchronous — return 202 Accepted with a job ID and poll/notify — rather than raising timeouts endlessly; when raising timeouts, align every layer (client < CDN < LB < proxy < app, or vice versa deliberately) so the error appears where you expect.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Reading ALB access logs
https 2026-10-25T10:15:02.123456Z app/shop-alb/50dc6c495c0c9188 49.37.x.x:51544 10.0.12.34:8080
0.001 60.002 -1 504 - 512 0 "GET https://shop.example.com:443/reports/annual HTTP/1.1" ...
Key fields: request_processing_time, target_processing_time, response_processing_time (a value of -1 means the LB couldn't get a complete response — target timeout or connection error), elb_status_code (504 here), target_status_code (- = the target never responded). Here: the target took 60 s → ALB idle timeout → 504.
A response playbook: "5xx spiking after a deploy"
- Scope: which codes, which layer (ELB vs target metrics), which paths, since when — correlate with the deploy time.
- Health of the new version: pods crashing or not Ready?
kubectl get pods, events, restarts, OOMKilled; target group healthy host count. - Logs and traces for the failing requests (request IDs).
- Mitigate: if errors persist and correlate with the release, roll back first (revert the GitOps commit /
kubectl rollout undo/ abort the canary), then investigate. - Prevent: canary analysis on error rate, readiness gates, post-deploy SLO checks.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.