Opscraft
Observability: metrics, logs, traces · Lesson 4 of 7 · 3 min read

Distributed tracing and OpenTelemetry

Concepts

  • Trace = tree of spans for one request; a span has name, start/end, attributes, status, events, parent.
  • Context propagation — the W3C traceparent header carries trace and span IDs between services (and through queues via message attributes).
  • Sampling: head-based (decide at the start, e.g. 10%) is cheap but may miss rare errors; tail-based (decide after seeing the whole trace, keep all errors and slow traces) needs a collector tier that buffers traces.
Enlarge diagramGET /checkout · api-gatewayorders: create orderpostgres: INSERTpayments: chargebank API callnotify: send email0 ms1,200 ms
Figure 1. A trace is a tree of spans for one request. The widest span shows where the time went.
Swipe sideways to see all of it, or tap to open it full screen.

OpenTelemetry (OTel)

The CNCF standard for generating and shipping telemetry:

  • APIs/SDKs per language, plus auto-instrumentation (Java agent, Python, Node, .NET) for HTTP, DB, messaging libraries — in Kubernetes, the OTel Operator injects agents via an Instrumentation resource and a pod annotation.
  • OTLP protocol for traces, metrics and logs.
  • Collector — receivers → processors (batch, memory_limiter, k8sattributes, tail_sampling, redaction) → exporters (Tempo, Jaeger, X-Ray, Prometheus remote_write, Loki, vendors). Deploy as agent (DaemonSet/sidecar) and gateway (Deployment).
receivers:
  otlp: { protocols: { grpc: {}, http: {} } }
processors:
  memory_limiter: { check_interval: 1s, limit_percentage: 80 }
  k8sattributes: {}
  batch: {}
  tail_sampling:
    policies:
      - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
      - { name: slow, type: latency, latency: { threshold_ms: 1000 } }
      - { name: sample, type: probabilistic, probabilistic: { sampling_percentage: 10 } }
exporters:
  otlp/tempo: { endpoint: tempo-distributor:4317, tls: { insecure: true } }
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, tail_sampling, batch]
      exporters: [otlp/tempo]

Backends: Grafana Tempo, Jaeger, AWS X-Ray (via ADOT — AWS Distro for OpenTelemetry), commercial APMs (Datadog, New Relic, Dynatrace, Elastic APM). Vendor-neutral instrumentation with OTel avoids lock-in.

Enlarge diagramOTEL COLLECTORappsOTel SDK or autoagentreceiversOTLPprocessorsbatch, k8sattributes, tail samplingexportersto each backendTempo / X-RayPrometheusLokiOTLP
Figure 2. The OpenTelemetry Collector: receive telemetry from apps, process it, and export it to whichever backends you use.
Swipe sideways to see all of it, or tap to open it full screen.