Kubernetes in depth · Lesson 8 of 11 · 5 min read
Scheduling and resource management
Requests, limits and QoS
- Requests — what the scheduler reserves; the basis for HPA utilisation.
- Limits — the cgroup ceiling. CPU over the limit → throttled. Memory over the limit → OOMKilled (exit 137).
| QoS class | Rule | Eviction priority under node pressure |
|---|---|---|
| Guaranteed | Every container has requests = limits for CPU and memory | Last |
| Burstable | At least one request/limit set, not Guaranteed | Middle — those using more than their requests first |
| BestEffort | No requests or limits | First |
Common senior positions: always set memory requests and limits (memory isn't compressible); set CPU requests but consider no CPU limits for latency-sensitive services to avoid throttling (requests still guarantee fair share); Guaranteed QoS for critical components.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Node pressure and eviction
The kubelet evicts pods when node resources cross thresholds (memory.available, nodefs.available, imagefs.available, pid.available), ranking by QoS and usage over requests. The node gets conditions like MemoryPressure/DiskPressure and taints so new pods avoid it. Separately, the kernel OOM killer can kill processes when memory runs out faster than kubelet reacts. Reserve resources for system daemons with kubeReserved/systemReserved.
Steering placement
spec:
nodeSelector: { karpenter.sh/capacity-type: on-demand }
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- { key: kubernetes.io/arch, operator: In, values: [arm64] }
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector: { matchLabels: { app: orders } }
topologyKey: kubernetes.io/hostname
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector: { matchLabels: { app: orders } }
tolerations:
- { key: dedicated, operator: Equal, value: payments, effect: NoSchedule }
priorityClassName: business-critical
- nodeSelector / node affinity — pull pods toward nodes (required = hard, preferred = soft). "IgnoredDuringExecution" means running pods aren't moved if labels change.
- Pod affinity / anti-affinity — co-locate with or separate from other pods (per node, per zone).
- Taints and tolerations — taints repel pods from nodes unless pods tolerate them. Effects:
NoSchedule,PreferNoSchedule,NoExecute(also evicts running pods without a toleration;tolerationSecondsdelays it). A toleration allows but doesn't attract — combine with affinity to dedicate nodes. Kubernetes itself taints nodesnot-ready/unreachable(NoExecute), and pods tolerate those for 300 s by default — which is why pods on a dead node are rescheduled after about 5 minutes. - Topology spread constraints — spread replicas evenly across zones/nodes (
maxSkew), the modern way to get AZ resilience. - PriorityClass and preemption — higher-priority pods can evict lower-priority ones when the cluster is full; system pods use
system-cluster-critical/system-node-critical.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Guardrails per namespace
- ResourceQuota — caps totals per namespace:
requests.cpu,limits.memory,pods,persistentvolumeclaims,services.loadbalancers, object counts. With a CPU/memory quota, every pod must declare requests/limits (or get defaults). - LimitRange — default and min/max requests/limits per container in a namespace.
- PodDisruptionBudget —
minAvailable/maxUnavailablefor voluntary disruptions (drains, upgrades, Karpenter consolidation). A PDB withmaxUnavailable: 0(or minAvailable equal to replicas) blocks drains forever — a common upgrade blocker.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.