Node problems
A node is NotReady
kubectl get nodes
kubectl describe node ip-10-20-1-15.ap-south-1.compute.internal # Conditions, events, allocated resources
Conditions: Ready=False/Unknown, MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable.
On the node (SSM Session Manager or kubectl debug node/...):
systemctl status kubelet containerd
journalctl -u kubelet --since "30 min ago" | tail -100
df -h /var/lib/containerd /var/log ; free -m
crictl ps ; crictl pods
Common causes: kubelet stopped or can't reach the API server (network, security groups, certificate/credential issues), container runtime hung ("PLEG is not healthy"), disk full (images, logs) → DiskPressure, memory exhaustion from pods without limits, CNI problems (aws-node crashing → NetworkUnavailable, pods stuck ContainerCreating), the EC2 instance itself impaired (check EC2 status checks), or a pod spike overloading kubelet or exhausting IPs.
When a node is NotReady, its pods are marked for eviction after the not-ready/unreachable toleration expires (300 s default) and recreated elsewhere (if capacity allows). Usually the right fix on managed groups is replace the node (cordon, drain, terminate) and investigate from logs.
Drains that never finish
kubectl drain blocks when evicting a pod would violate a PodDisruptionBudget — e.g. minAvailable: 1 on a single-replica Deployment, or maxUnavailable: 0. It also refuses unmanaged pods (no controller) without --force and pods with emptyDir without --delete-emptydir-data. Fix the PDB (or scale up the app), then drain:
kubectl drain NODE --ignore-daemonsets --delete-emptydir-data --timeout=10m
kubectl get pdb -A
This is also why node group upgrades and Karpenter consolidation "hang".