Docker Swarm · Lesson 8 of 10 · 3 min read
Operating a swarm
Backup and restore
The cluster state (services, networks, secrets, configs, node membership) is the Raft data in /var/lib/docker/swarm on managers.
- Backup: on a non-leader manager, stop Docker (to get a consistent copy), archive
/var/lib/docker/swarm, start Docker. With autolock, also store the unlock key safely. - Restore: on a new node, restore the directory, start Docker, run
docker swarm init --force-new-cluster, then re-add managers and workers. - Application data in volumes needs its own backups — Swarm backups don't include it.
Storage for stateful services
Local volumes live on one node, so a rescheduled task on another node gets an empty volume. Options: pin the service to a node with a constraint (losing failover), use a shared or cloud volume plugin (EFS via NFS, Portworx, REX-Ray), or keep state in managed services (RDS, S3) — usually the right answer on AWS.
Upgrading Docker across the cluster
Drain a node, upgrade Docker Engine, reactivate it, move on. Upgrade managers one at a time so quorum is never lost. Keep engine versions close; mixed versions are supported briefly during upgrades.
Monitoring and logging
- Metrics: Prometheus with node-exporter and cAdvisor as global services; Docker's metrics endpoint (
"metrics-addr"in daemon.json) for engine metrics. - Logs:
docker service logsfor quick looks; a log driver or a global Fluent Bit service shipping to CloudWatch/Loki/OpenSearch for retention. - Alert on
docker service lsreplicas below desired, manager reachability, and node status.