Opscraft
Amazon EKS · Lesson 10 of 11 · 3 min read

Troubleshooting EKS-specific problems

Enlarge diagramAccess entrynode role allowed in?Network pathsubnet → API endpoint,security groupsEndpointsno NAT? need ECR, S3,STS endpointsBootstrapAMI and nodeadmconfig1234
Figure 1. Nodes that fail to join the cluster: the usual suspects, in order.
Swipe sideways to see all of it, or tap to open it full screen.
Symptom Likely cause / fix
Nodes fail to join ("Instances failed to join the cluster") Node IAM role missing access entry (or aws-auth mapping); subnets can't reach the API endpoint (private endpoint + no route, security groups); missing VPC endpoints for STS/ECR/EC2 in private-only clusters; wrong AMI/bootstrap user data; check /var/log/cloud-init-output.log and kubelet logs on the instance
Pods stuck ContainerCreating: failed to assign an IP address Subnet IP exhaustion — prefix delegation, secondary CIDR, bigger subnets
ImagePullBackOff from ECR in private subnets No NAT and no ECR api/dkr + S3 gateway endpoints; node role missing ECR read policy; cross-account repository policy
ALB not created for an Ingress LB controller logs; IngressClass alb; subnet tags; IAM permissions; annotation typos
ALB targets unhealthy Health check path/port, readiness, security groups between ALB and pods/nodes
Pod gets AccessDenied calling AWS ServiceAccount annotation/association missing, trust policy sub doesn't match system:serviceaccount:<ns>:<sa>, pod not using that ServiceAccount, SDK too old for web identity/Pod Identity, permissions policy too narrow
Pods using the node role instead of IRSA Credential chain falling back to IMDS — IRSA env vars missing (pod created before annotation → restart pods)
kubectl Unauthorized / You must be logged in Wrong AWS profile/role, no access entry for the principal, expired SSO session; aws sts get-caller-identity
DNS timeouts CoreDNS scaling, ndots, VPC resolver per-ENI limits → NodeLocal DNSCache
Karpenter not launching nodes NodePool limits, requirements too narrow, EC2NodeClass discovery tags, IAM, Spot capacity — kubectl get nodeclaims, controller logs
Upgrade broke workloads Removed API versions in manifests/charts, add-on incompatibility