Opscraft
Docker Swarm · Lesson 8 of 10 · 3 min read

Operating a swarm

Backup and restore

The cluster state (services, networks, secrets, configs, node membership) is the Raft data in /var/lib/docker/swarm on managers.

  • Backup: on a non-leader manager, stop Docker (to get a consistent copy), archive /var/lib/docker/swarm, start Docker. With autolock, also store the unlock key safely.
  • Restore: on a new node, restore the directory, start Docker, run docker swarm init --force-new-cluster, then re-add managers and workers.
  • Application data in volumes needs its own backups — Swarm backups don't include it.

Storage for stateful services

Local volumes live on one node, so a rescheduled task on another node gets an empty volume. Options: pin the service to a node with a constraint (losing failover), use a shared or cloud volume plugin (EFS via NFS, Portworx, REX-Ray), or keep state in managed services (RDS, S3) — usually the right answer on AWS.

Upgrading Docker across the cluster

Drain a node, upgrade Docker Engine, reactivate it, move on. Upgrade managers one at a time so quorum is never lost. Keep engine versions close; mixed versions are supported briefly during upgrades.

Monitoring and logging

  • Metrics: Prometheus with node-exporter and cAdvisor as global services; Docker's metrics endpoint ("metrics-addr" in daemon.json) for engine metrics.
  • Logs: docker service logs for quick looks; a log driver or a global Fluent Bit service shipping to CloudWatch/Loki/OpenSearch for retention.
  • Alert on docker service ls replicas below desired, manager reachability, and node status.