Opscraft
Kubernetes troubleshooting · Lesson 8 of 11 · 3 min read

Node problems

A node is NotReady

kubectl get nodes
kubectl describe node ip-10-20-1-15.ap-south-1.compute.internal   # Conditions, events, allocated resources

Conditions: Ready=False/Unknown, MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable.

On the node (SSM Session Manager or kubectl debug node/...):

systemctl status kubelet containerd
journalctl -u kubelet --since "30 min ago" | tail -100
df -h /var/lib/containerd /var/log ; free -m
crictl ps ; crictl pods

Common causes: kubelet stopped or can't reach the API server (network, security groups, certificate/credential issues), container runtime hung ("PLEG is not healthy"), disk full (images, logs) → DiskPressure, memory exhaustion from pods without limits, CNI problems (aws-node crashing → NetworkUnavailable, pods stuck ContainerCreating), the EC2 instance itself impaired (check EC2 status checks), or a pod spike overloading kubelet or exhausting IPs.

When a node is NotReady, its pods are marked for eviction after the not-ready/unreachable toleration expires (300 s default) and recreated elsewhere (if capacity allows). Usually the right fix on managed groups is replace the node (cordon, drain, terminate) and investigate from logs.

Enlarge diagramheartbeats stopkubelet ornetwork downNotReadytaint added300 s laterpods evictedrecreatedelsewhereif capacity allows
Figure 1. What happens when a node goes NotReady.
Swipe sideways to see all of it, or tap to open it full screen.

Drains that never finish

kubectl drain blocks when evicting a pod would violate a PodDisruptionBudget — e.g. minAvailable: 1 on a single-replica Deployment, or maxUnavailable: 0. It also refuses unmanaged pods (no controller) without --force and pods with emptyDir without --delete-emptydir-data. Fix the PDB (or scale up the app), then drain:

kubectl drain NODE --ignore-daemonsets --delete-emptydir-data --timeout=10m
kubectl get pdb -A

This is also why node group upgrades and Karpenter consolidation "hang".

Enlarge diagramkubectl drainnode-3orders pod1 replica onlyPDBminAvailable: 1evictrefusedFix: run at least 2 replicas, or use maxUnavailable: 1.
Figure 2. Why a drain hangs: evicting the only pod would break its PodDisruptionBudget, so the eviction is refused again and again.
Swipe sideways to see all of it, or tap to open it full screen.