Observability: metrics, logs, traces · Lesson 3 of 7 · 3 min read
Logging in depth
Principles
- Log to stdout/stderr in containers; the platform collects.
- Structured logs (JSON) with consistent fields: timestamp, level, service, environment,
trace_id/span_id, request ID, user-safe identifiers — not free-text only. - Never log secrets or PII (tokens, passwords, card numbers); mask at source, and add redaction in the pipeline as a safety net.
- Log levels with discipline; avoid DEBUG floods in prod; sample high-volume success logs.
- Retention by value: hot searchable days, cheap archive (S3) for compliance.
Kubernetes log pipeline
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Fluent Bit (lightweight C) is the usual node agent; Fluentd or Vector/OTel Collector for heavier aggregation. Key settings: Kubernetes filter (pod labels, namespace), multiline parsing (Java stack traces), memory/filesystem buffering and retry for backpressure, throttling noisy namespaces.
Backends compared
| Loki | OpenSearch/Elasticsearch | CloudWatch Logs | |
|---|---|---|---|
| Indexing | Only labels indexed; content scanned with LogQL at query time | Full-text inverted index | Managed; Logs Insights query language |
| Cost | Low (object storage, small index) | Higher (index storage/compute) | Pay per GB ingested + stored + queried |
| Strength | Cheap, Grafana-native, Prometheus-like labels | Powerful search/analytics | Zero ops, native AWS integration, subscription filters, metric filters |
| Pitfall | High-cardinality labels kill it | Cluster ops, shard sizing | Costs at volume; set retention |
LogQL example:
sum by (status) (count_over_time({namespace="orders", app="orders-api"} | json | status >= 500 [5m]))
CloudWatch Logs Insights example:
fields @timestamp, @message
| filter level = "ERROR" and service = "orders"
| stats count(*) by bin(5m)