Opscraft
Kubernetes troubleshooting · Lesson 2 of 11 · 6 min read

Pods that won't start

Enlarge diagramPendingContainerCreatingImagePullBackOffCreateContainerConfigErrorCrashLoopBackOffRunning 0/1scheduling: describe → FailedSchedulingvolumes, CNI IPs, missing ConfigMapimage name, tag, registry accessmissing Secret or ConfigMap keylogs --previous, exit codereadiness probe failing
Figure 1. Start from the STATUS column. Each value points to a different layer and a different first command.
Swipe sideways to see all of it, or tap to open it full screen.

Pending — not scheduled

kubectl describe pod shows a FailedScheduling event from default-scheduler. Decode the message:

Enlarge diagram3 Insufficient memoryrequests too big, or add nodes2 untolerated taintadd a toleration or go elsewhere1 unschedulablenode is cordoned0/6 nodes are available: 3 Insufficient memory, 2 untolerated taint, 1 unschedulableScheduling uses requests, not real usage. Also check quota events on the ReplicaSet.
Figure 2. Reading a FailedScheduling message. The numbers add up to every node and say why each was rejected.
Swipe sideways to see all of it, or tap to open it full screen.
Message fragment Meaning Fix
Insufficient cpu / Insufficient memory No node has enough unrequested capacity for the pod's requests Lower requests to realistic values, scale the node group, let Karpenter/Cluster Autoscaler add nodes (check why they didn't)
node(s) had untolerated taint {dedicated: gpu} Taints repel the pod Add a toleration or schedule elsewhere
node(s) didn't match Pod's node affinity/selector nodeSelector/affinity can't be satisfied Fix labels or affinity; check instance types available
node(s) didn't match pod anti-affinity rules / didn't satisfy existing pods anti-affinity Required anti-affinity with too few nodes Use preferred anti-affinity or topology spread
doesn't satisfy spread constraint topologySpreadConstraints with DoNotSchedule More capacity in the needed zone, or ScheduleAnyway
pod has unbound immediate PersistentVolumeClaims PVC not bound See storage lesson
volume node affinity conflict The pod's EBS volume is in an AZ with no suitable node Capacity in that AZ; WaitForFirstConsumer
Too many pods Node's max pods reached (VPC CNI IP limits) Prefix delegation, bigger instances, more nodes
exceeded quota (event on the ReplicaSet, not pod) ResourceQuota denies creation Raise quota or reduce requests — check kubectl describe rs / resourcequota
0/6 nodes are available: 6 node(s) were unschedulable Nodes cordoned Uncordon or add nodes

Autoscaler didn't add nodes? Cluster Autoscaler: node group max size reached, no instance type in the group fits the pod, the pod's constraints (taints, affinity, zone, arch) can't be met by that group, ASG launch failures (insufficient capacity, quotas) — check CA logs. Karpenter: NodePool limits reached, requirements (instance families, zones, capacity-type) exclude what's needed, EC2NodeClass issues (subnet/SG discovery tags, AMI), IAM, or Spot capacity — check kubectl get nodeclaims and Karpenter controller logs.

ContainerCreating — scheduled, not started

Stuck here means the kubelet can't finish setup. describe events tell you which part:

  • FailedCreatePodSandBox ... failed to assign an IP address — VPC CNI IP exhaustion in the subnet/node (see EKS topic: prefix delegation, secondary CIDR, bigger subnets); aws-node pod unhealthy.
  • FailedMount / FailedAttachVolume — a volume can't attach or mount: EBS attached elsewhere (Multi-Attach error), wrong AZ, CSI driver missing or lacking IAM permissions, EFS mount target/security group missing.
  • MountVolume.SetUp failed ... configmap "x" not found / secret not found — a referenced ConfigMap/Secret doesn't exist (yet). With External Secrets, the ExternalSecret may be failing to sync.

Image pull errors

ErrImagePull is the failure; ImagePullBackOff is Kubernetes waiting (exponential backoff up to 5 minutes) before retrying. Read the event message:

Enlarge diagrammanifest unknowntag does not existunauthorized / 403node role or pull secrettoomanyrequestsDocker Hub limit: mirror to ECRi/o timeoutno NAT or ECR endpointsno matching manifestarm64 vs amd64 imageInvalidImageNamebad characters
Figure 3. Image pull errors: the event message names the cause.
Swipe sideways to see all of it, or tap to open it full screen.
Event message Cause
manifest unknown / not found Wrong image name or tag doesn't exist (typo, image not pushed yet, pipeline pushed a different tag)
unauthorized / no basic auth credentials / 403 Private registry without credentials: missing imagePullSecrets, node role lacks ecr:GetAuthorizationToken/BatchGetImage, cross-account ECR without a repository policy
toomanyrequests Docker Hub rate limits — mirror images to ECR or use an ECR pull-through cache
i/o timeout / dial tcp ... connect: connection refused No network path to the registry — private subnets without NAT or VPC endpoints for ECR (api, dkr) and S3
no matching manifest for linux/arm64 Architecture mismatch (Graviton nodes, amd64-only image) — build multi-arch
InvalidImageName Malformed reference (uppercase letters, bad characters)
kubectl describe pod orders-7d9f-abc -n shop | sed -n '/Events/,$p'
aws ecr describe-images --repository-name orders --image-ids imageTag=3f9c2a1

CreateContainerConfigError

The container spec references something missing: a ConfigMap or Secret key that doesn't exist (couldn't find key DB_PASSWORD in Secret shop/orders-db), an invalid runAsUser with runAsNonRoot (image runs as root but the pod requires non-root: container has runAsNonRoot and image will run as root), or a missing ServiceAccount.

Init containers

Init:0/2, Init:Error, Init:CrashLoopBackOff — an init container is failing or waiting (a "wait for DB" loop that never succeeds). Logs: kubectl logs pod -c <init-container-name>.