Opscraft
Docker Swarm · Lesson 2 of 10 · 5 min read

Architecture: managers, workers and Raft

Node roles

  • Managers maintain cluster state, schedule tasks, serve the Swarm API (docker service ..., docker node ...) and replicate state among themselves using Raft. One manager is the leader; others forward writes to it. Managers also run workloads by default.
  • Workers only run tasks assigned by managers. They can't run cluster-management commands ("This node is not a swarm manager").

A node can be promoted or demoted at any time:

Enlarge diagramMANAGERS · RAFT CONSENSUSWORKERSmanager 1leadermanager 2followermanager 3followerworker 1runs tasksworker 2runs tasksworker 3runs tasksworker 4runs tasks
Figure 1. A swarm: three managers keep the cluster state in agreement using Raft; workers just run the tasks they are given.
Swipe sideways to see all of it, or tap to open it full screen.
docker node promote node-4
docker node demote node-2
docker node ls                    # MANAGER STATUS: Leader / Reachable / Unavailable

Raft and quorum

Raft needs a majority (quorum) of managers to agree on any change. Quorum = ⌊N/2⌋ + 1.

Managers (N) Quorum Managers you can lose
1 1 0
2 2 0 (worse than 1 — two points of failure, no tolerance)
3 2 1
4 3 1
5 3 2
7 4 3

Rules of thumb:

Enlarge diagram1 managercan lose 02 managerscan lose 0 (worse than 1!)3 managerscan lose 15 managerscan lose 27 managerscan lose 3green = needed for the majorityamber = can fail safely
Figure 2. Raft needs a majority of managers. With 3 managers you can lose 1; with 5 you can lose 2. Even numbers add cost without adding safety.
Swipe sideways to see all of it, or tap to open it full screen.
  • Use an odd number — 3 for most clusters, 5 for large or critical ones. An even number adds cost and failure points without adding tolerance.
  • More than 7 managers slows consensus; Docker recommends no more.
  • Spread managers across availability zones so losing one AZ can't take out quorum (3 managers in 3 AZs).
  • Managers' Raft state lives in /var/lib/docker/swarm — keep it on reliable, low-latency disk.

If quorum is lost: running tasks keep running, and the routing mesh keeps working, but you can't change anything — no deploys, no scaling, no rescheduling of failed tasks. Recover by restoring managers or, if they're gone for good, running on a surviving manager:

docker swarm init --force-new-cluster --advertise-addr <this-node-ip>

This creates a single-manager cluster from that node's copy of the state; then promote new managers back to 3 or 5.

Enlarge diagramStill worksFrozenRunning tasks keep runningThe routing mesh keeps serving trafficNo deploys or scalingFailed tasks are not rescheduledFix: restore managers, or --force-new-cluster ona survivor
Figure 3. What happens when quorum is lost: running work keeps running, but nothing can change until managers are restored.
Swipe sideways to see all of it, or tap to open it full screen.

Built-in security (PKI)

docker swarm init creates a root CA. Every node gets a certificate, and all manager–node communication uses mutual TLS. Certificates rotate automatically (default every 90 days; docker swarm update --cert-expiry). The CA can be rotated with docker swarm ca --rotate.

Join tokens contain the CA fingerprint plus a secret; there's one for workers and one for managers:

docker swarm init --advertise-addr 10.0.1.10
docker swarm join-token worker           # print the join command
docker swarm join-token --rotate manager # invalidate a leaked token
docker swarm join --token SWMTKN-1-... 10.0.1.10:2377
docker swarm leave [--force]
docker node rm node-5

Autolock (docker swarm init --autolock or update --autolock=true) encrypts the Raft logs' keys at rest; after a manager restarts you must run docker swarm unlock with the unlock key. It protects secrets if someone steals a manager's disk.

Ports to open between nodes

Port Protocol Purpose
2377 TCP Cluster management (to managers)
7946 TCP + UDP Node-to-node gossip (membership, network control plane)
4789 UDP Overlay network data (VXLAN)
IP protocol 50 ESP Only if overlay encryption is enabled

In AWS security groups, a classic mistake is opening 7946 TCP but forgetting UDP, or forgetting 4789/UDP — containers on different nodes then can't talk.