Opscraft
HTTP & status codes · Lesson 8 of 9 · 3 min read

Monitoring, SLOs, retries and API design

What to alert on

  • 5xx error rate as a percentage of requests — not raw counts (100 errors per minute is a disaster at 200 rpm and noise at 2 million).
  • Latency percentiles (p95/p99) — tail latency often degrades before errors appear.
  • Saturation — CPU, memory, connection pools, queue depth.
  • Per layer: ELB 5xx vs target 5xx; per route for critical endpoints (checkout, login).
  • Burn-rate alerts on SLOs (e.g. 99.9% of requests succeed over 30 days) rather than static thresholds.

Retry policies

  • Retry only idempotent requests (GET, PUT, DELETE) — or POSTs with idempotency keys.
  • Retry on transient failures: 502, 503, 504, 429 (respect Retry-After), connection resets/timeouts. Don't retry 4xx like 400/401/403/404 — they'll fail again.
  • Use exponential backoff with jitter, a small retry limit, and an overall deadline; budget retries across layers (retries at client, mesh and app multiply load: 3 × 3 × 3 = 27×).
  • Circuit breakers stop hammering a failing dependency and fail fast.
Enlarge diagramclient3 attemptsmesh sidecar× 3 eachapp× 3 eachstrugglingservice27 callsRetry at one layer, with backoff, jitter and a deadline; add circuit breakers to fail fast.
Figure 1. Retries multiply. Three layers that each retry three times turn one failing request into 27 calls on a service that is already struggling.
Swipe sideways to see all of it, or tap to open it full screen.

Status codes in API design (what good looks like)

  • 201 + Location for creates; 202 for async work; 204 for no-content success.
  • 400/422 for validation with a structured error body (e.g. RFC 9457 problem details: type, title, detail, instance).
  • 401 vs 403 used correctly; 404 for missing; 409 for conflicts; 429 with Retry-After for limits.
  • Never 200 with an error body; never 500 for client mistakes (it pollutes your error rate and pages on-call for client bugs).