Pods that won't start
Pending — not scheduled
kubectl describe pod shows a FailedScheduling event from default-scheduler. Decode the message:
| Message fragment | Meaning | Fix |
|---|---|---|
Insufficient cpu / Insufficient memory |
No node has enough unrequested capacity for the pod's requests | Lower requests to realistic values, scale the node group, let Karpenter/Cluster Autoscaler add nodes (check why they didn't) |
node(s) had untolerated taint {dedicated: gpu} |
Taints repel the pod | Add a toleration or schedule elsewhere |
node(s) didn't match Pod's node affinity/selector |
nodeSelector/affinity can't be satisfied | Fix labels or affinity; check instance types available |
node(s) didn't match pod anti-affinity rules / didn't satisfy existing pods anti-affinity |
Required anti-affinity with too few nodes | Use preferred anti-affinity or topology spread |
doesn't satisfy spread constraint |
topologySpreadConstraints with DoNotSchedule |
More capacity in the needed zone, or ScheduleAnyway |
pod has unbound immediate PersistentVolumeClaims |
PVC not bound | See storage lesson |
volume node affinity conflict |
The pod's EBS volume is in an AZ with no suitable node | Capacity in that AZ; WaitForFirstConsumer |
Too many pods |
Node's max pods reached (VPC CNI IP limits) | Prefix delegation, bigger instances, more nodes |
exceeded quota (event on the ReplicaSet, not pod) |
ResourceQuota denies creation | Raise quota or reduce requests — check kubectl describe rs / resourcequota |
0/6 nodes are available: 6 node(s) were unschedulable |
Nodes cordoned | Uncordon or add nodes |
Autoscaler didn't add nodes? Cluster Autoscaler: node group max size reached, no instance type in the group fits the pod, the pod's constraints (taints, affinity, zone, arch) can't be met by that group, ASG launch failures (insufficient capacity, quotas) — check CA logs. Karpenter: NodePool limits reached, requirements (instance families, zones, capacity-type) exclude what's needed, EC2NodeClass issues (subnet/SG discovery tags, AMI), IAM, or Spot capacity — check kubectl get nodeclaims and Karpenter controller logs.
ContainerCreating — scheduled, not started
Stuck here means the kubelet can't finish setup. describe events tell you which part:
FailedCreatePodSandBox...failed to assign an IP address— VPC CNI IP exhaustion in the subnet/node (see EKS topic: prefix delegation, secondary CIDR, bigger subnets);aws-nodepod unhealthy.FailedMount/FailedAttachVolume— a volume can't attach or mount: EBS attached elsewhere (Multi-Attach error), wrong AZ, CSI driver missing or lacking IAM permissions, EFS mount target/security group missing.MountVolume.SetUp failed ... configmap "x" not found/ secret not found — a referenced ConfigMap/Secret doesn't exist (yet). With External Secrets, the ExternalSecret may be failing to sync.
Image pull errors
ErrImagePull is the failure; ImagePullBackOff is Kubernetes waiting (exponential backoff up to 5 minutes) before retrying. Read the event message:
| Event message | Cause |
|---|---|
manifest unknown / not found |
Wrong image name or tag doesn't exist (typo, image not pushed yet, pipeline pushed a different tag) |
unauthorized / no basic auth credentials / 403 |
Private registry without credentials: missing imagePullSecrets, node role lacks ecr:GetAuthorizationToken/BatchGetImage, cross-account ECR without a repository policy |
toomanyrequests |
Docker Hub rate limits — mirror images to ECR or use an ECR pull-through cache |
i/o timeout / dial tcp ... connect: connection refused |
No network path to the registry — private subnets without NAT or VPC endpoints for ECR (api, dkr) and S3 |
no matching manifest for linux/arm64 |
Architecture mismatch (Graviton nodes, amd64-only image) — build multi-arch |
InvalidImageName |
Malformed reference (uppercase letters, bad characters) |
kubectl describe pod orders-7d9f-abc -n shop | sed -n '/Events/,$p'
aws ecr describe-images --repository-name orders --image-ids imageTag=3f9c2a1
CreateContainerConfigError
The container spec references something missing: a ConfigMap or Secret key that doesn't exist (couldn't find key DB_PASSWORD in Secret shop/orders-db), an invalid runAsUser with runAsNonRoot (image runs as root but the pod requires non-root: container has runAsNonRoot and image will run as root), or a missing ServiceAccount.
Init containers
Init:0/2, Init:Error, Init:CrashLoopBackOff — an init container is failing or waiting (a "wait for DB" loop that never succeeds). Logs: kubectl logs pod -c <init-container-name>.