Operating serverless
Observability
| Metric | What it tells you |
|---|---|
Errors, Invocations |
Error rate per function |
Throttles |
Hitting concurrency limits |
Duration (p50/p99) |
Latency, and how close you are to the timeout |
ConcurrentExecutions |
Headroom against the account and reserved limits |
IteratorAge (streams) |
How far behind the function is; rising means it can't keep up or is stuck on a bad record |
AsyncEventAge, DeadLetterErrors |
Async backlog and failures to reach the DLQ |
SQS ApproximateAgeOfOldestMessage |
Queue backlog |
Every invocation writes START, END and REPORT lines to CloudWatch Logs, with duration, billed duration, memory used and Init Duration on cold starts. Log in structured JSON with the request ID, use Powertools for AWS Lambda for logging, metrics and tracing, and enable X-Ray or OpenTelemetry (ADOT) for traces across API Gateway, Lambda and downstream calls.
Orchestration with Step Functions
Chaining functions by having one call another directly is fragile: the caller pays for waiting, and errors are hard to handle. Step Functions state machines coordinate steps instead, with retries, back-off, catch blocks, parallel branches, maps over items, waits and human approval steps, all visible as a diagram.
- Standard workflows: up to a year long, exactly-once step execution, priced per state transition. For business processes and long jobs.
- Express workflows: up to 5 minutes, at-least-once, priced by duration. For high-volume event processing.
- Direct service integrations call DynamoDB, SQS, ECS, Glue and others without a Lambda in between.
EventBridge is the other glue: rules route events between services and accounts, EventBridge Scheduler replaces cron jobs, and Pipes connect a source to a target with filtering and enrichment.
Cost and failure traps
- Recursive loops: a function writes to the S3 bucket or SQS queue that triggers it. Lambda detects and stops many such loops, but design triggers carefully (separate input and output buckets or prefixes).
- Retries multiplying cost and side effects: async retries plus SQS redelivery plus client retries. Make handlers idempotent and set retry counts deliberately.
- Over-sized memory or timeouts on functions that run millions of times; use Power Tuning and Compute Optimizer recommendations.
- Chatty logging: CloudWatch Logs ingestion can cost more than the function. Set log retention and levels.
- Steady high traffic: at a constant few hundred requests per second, containers on Fargate or EKS are usually cheaper. Compare monthly cost at expected load, not just at zero.
- NAT gateway data charges for VPC functions calling AWS APIs; use VPC endpoints.
Choosing Lambda, Fargate or EKS
- Lambda: event-driven tasks, glue between services, APIs with spiky or low traffic, scheduled jobs, file processing. Work under 15 minutes.
- Fargate (ECS or EKS): containerised services without managing nodes, long-running processes, steady traffic, more control over runtime and networking.
- EKS on EC2: many services with a shared platform, GPUs or special instance types, daemons, service meshes, maximum cost efficiency at scale.
Many real systems mix them: the main services on EKS, with Lambda handling S3 events, scheduled maintenance, alarms-to-Slack and other glue.