Opscraft
Kubernetes in depth · Lesson 3 of 11 · 9 min read

Node components and the pod lifecycle

kubelet

The agent on every node. It:

Enlarge diagramkubeletone per nodeAPI serverwatch my pods, reportstatusCRI → containerdpull images, run containersCNIgive the pod an IPCSIattach and mount volumesprobes · cgroup limits · eviction
Figure 1. The kubelet on every node does the real work of running pods, through plug-in interfaces for containers, networking and storage.
Swipe sideways to see all of it, or tap to open it full screen.
  • Registers the node and sends heartbeats (Lease objects in kube-node-lease plus NodeStatus updates).
  • Watches for pods bound to its node, and drives the container runtime via the CRI (Container Runtime Interface) to create the pod sandbox, pull images, and start containers.
  • Mounts volumes (with CSI drivers), sets up the network via the CNI plugin.
  • Runs probes (liveness, readiness, startup) and restarts containers per restartPolicy.
  • Reports pod status back to the API server.
  • Enforces resource limits via cgroups, and performs node-pressure eviction when memory, disk (nodefs/imagefs) or PIDs run low (e.g. memory.available<100Mi).
  • Runs static pods from manifests in /etc/kubernetes/manifests (that's how kubeadm runs the control plane itself), visible in the API as mirror pods.
  • Garbage-collects dead containers and unused images.

The kubelet's PLEG (pod lifecycle event generator) relists containers periodically; "PLEG is not healthy" means the runtime is too slow to respond — often overloaded nodes or a stuck container runtime — and leads to NotReady.

Container runtime

Enlarge diagramkubeletcontainerdvia CRI (gRPC)shimone per podruncstarts theprocessnamespaces +cgroupsthe Linux kernel
Figure 2. From kubelet to a running process. Each pod first gets a tiny "pause" sandbox that owns its network; the app containers join it.
Swipe sideways to see all of it, or tap to open it full screen.

Each pod first gets a sandbox (the "pause" container) that holds the pod's network namespace; app containers join it — that's why containers in a pod share an IP and can talk over localhost. Debug the runtime directly with crictl ps, crictl logs, crictl inspectp, crictl images.

Enlarge diagramPOD ORDERS-7D9F · IP 10.0.12.34pauseholds the networknamespaceapp containerlistens on :8080sidecarcalls localhost:8080
Figure 3. Why containers in a pod share an IP: they all join the network namespace held by the pause container, so they talk over localhost.
Swipe sideways to see all of it, or tap to open it full screen.

kube-proxy

Implements Services on each node by programming packet rules from EndpointSlices:

  • iptables mode (default): NAT rules — a Service's ClusterIP:port DNATs to a randomly chosen backend pod (probability rules). Rule count grows with Services × endpoints; updates get slower at large scale.
  • IPVS mode: kernel load balancer with hash tables — scales better and offers algorithms (rr, least-connection, source hash).
  • nftables mode (newer releases) as the iptables successor.
  • Replaced entirely by eBPF in Cilium's kube-proxy-free mode.

kube-proxy doesn't proxy traffic itself in these modes — it only writes kernel rules.

Enlarge diagramEndpointSlicesready pod IPs per Servicekube-proxyon every node, watchingiptables / IPVS rulesin the node's kernelwatchwrite
Figure 4. kube-proxy does not carry traffic. It watches which pods back each Service and writes kernel rules on every node.
Swipe sideways to see all of it, or tap to open it full screen.

CNI plugin

Gives every pod an IP and connectivity. The Kubernetes network model requires: every pod gets its own IP; pods can reach all other pods without NAT; nodes can reach all pods. How it's implemented varies:

CNI Approach
Amazon VPC CNI Pods get real VPC IPs from ENIs — no overlay (EKS default)
Calico BGP routing or VXLAN/IP-in-IP overlay; strong NetworkPolicy support
Cilium eBPF dataplane, kube-proxy replacement, L7 policies, observability (Hubble)
Flannel Simple VXLAN overlay, no NetworkPolicy

What happens when you create a Deployment

  1. kubectl apply sends the Deployment to the API server: authn → authz → admission → validation → etcd.
  2. The Deployment controller (watching Deployments) creates a ReplicaSet for the pod template (named with a hash of the template).
  3. The ReplicaSet controller creates N Pod objects (no node yet).
  4. The scheduler filters and scores nodes, then binds each pod to a node.
  5. The kubelet on that node sees the pod: pulls images (via containerd), sets up the sandbox and networking (CNI assigns an IP), mounts volumes, runs init containers in order, then starts app containers.
  6. Probes run; when readiness passes, the pod condition Ready=True.
  7. The EndpointSlice controller adds the pod IP to matching Services' EndpointSlices; kube-proxy on every node updates rules; ingress/ALB controllers register targets; CoreDNS answers for the Service.
Enlarge diagramkubectlAPI server+ etcdcontrollersschedulerkubeletnode-3apply Deploymentcreate RS + Podsbind each pod to a nodesees a pod for node-3pull image, CNI IP,start containersstatus: Readyupdate endpoints
Figure 5. What happens after kubectl apply for a Deployment: each component reacts to the previous one's change through the API server.
Swipe sideways to see all of it, or tap to open it full screen.

Pod phases, conditions and container states

  • Phases: Pending (accepted, not yet running — scheduling, image pulls), Running (bound, at least one container running), Succeeded, Failed, Unknown (node unreachable).
  • Conditions: PodScheduled, Initialized, ContainersReady, Ready (and readiness gates).
  • Container states: Waiting (reasons: ContainerCreating, ImagePullBackOff, CrashLoopBackOff, CreateContainerConfigError), Running, Terminated (reason, exit code).
  • restartPolicy: Always (Deployments), OnFailure or Never (Jobs). Restarts back off exponentially up to 5 minutes — that's CrashLoopBackOff.
Enlarge diagramPendingscheduling, image pullRunningat least one containerupSucceededall exited 0 (Jobs)Faileda container exited non-zeroWaiting reasons to know: ImagePullBackOff · CrashLoopBackOff · CreateContainerConfigError
Figure 6. A pod's life. Most problems show up as a pod stuck in Pending or a container stuck Waiting with a reason.
Swipe sideways to see all of it, or tap to open it full screen.

Init containers and sidecars

  • Init containers run to completion, in order, before app containers start: wait for a dependency, run migrations, fetch config, fix volume permissions. If one fails, the pod restarts them per policy.
  • Sidecars run alongside the app: log shippers, proxies (Envoy), secret refreshers. Since Kubernetes 1.28 (stable in 1.33) there are native sidecars: an init container with restartPolicy: Always starts before app containers, keeps running, and is stopped after them — fixing old problems like Jobs never completing because a sidecar kept running.
Enlarge diagraminit 1wait for DBinit 2run migrationssidecarlog shipper: keepsrunningappstarts last
Figure 7. Start order inside a pod: init containers run one after another to completion; native sidecars start next and keep running; then the app starts.
Swipe sideways to see all of it, or tap to open it full screen.

Pod termination

On deletion: the pod gets a deletion timestamp and is removed from endpoints; the kubelet runs preStop hooks, sends SIGTERM, waits up to terminationGracePeriodSeconds (default 30), then SIGKILL. Endpoint removal and SIGTERM happen concurrently, so apps need a short preStop delay to avoid dropped requests (covered in the deployment-strategies topic).

Enlarge diagramDeletedeletionTimestamp setRemoved from endpointsin parallel with the next steppreStop hooke.g. sleep 5SIGTERMapp drains connectionsGrace period30 s by defaultSIGKILLif still running123456
Figure 8. Pod shutdown. Removal from the Service and SIGTERM happen at the same time, so apps should keep serving a few seconds after SIGTERM.
Swipe sideways to see all of it, or tap to open it full screen.