DNS in AWS VPCs and Kubernetes
The VPC resolver
Every VPC has the Route 53 Resolver at the VPC CIDR base +2 (e.g. 10.0.0.2) and at 169.254.169.253. It resolves private hosted zones, EC2 internal names, endpoint private DNS, forwarding rules, and public DNS. Limits matter at scale: each ENI can send up to 1,024 packets per second to it; beyond that, queries are dropped — visible as intermittent DNS timeouts on busy instances and Kubernetes nodes (where all pods on the node share that ENI's budget through CoreDNS).
CoreDNS in Kubernetes
Pods' /etc/resolv.conf (set by the kubelet for dnsPolicy: ClusterFirst):
nameserver 172.20.0.10 ← kube-dns Service (CoreDNS)
search shop.svc.cluster.local svc.cluster.local cluster.local ap-south-1.compute.internal
options ndots:5
The CoreDNS configuration (coredns ConfigMap):
.:53 {
errors
health
ready
kubernetes cluster.local in-addr.arpa ip6.arpa { # answer Service/pod names from the API
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153 # metrics
forward . /etc/resolv.conf # everything else → node's resolver (VPC resolver)
cache 30 # cache up to 30 s
loop
reload
loadbalance
}
corp.example.com:53 { # added: forward an on-prem domain
forward . 10.200.0.10 10.200.0.11
}
- Names under
cluster.localcome from the Kubernetes API (Services, headless Service pods). - A short name like
ordersresolves via the search list toorders.shop.svc.cluster.local— only in the pod's own namespace; other namespaces needorders.paymentsor the FQDN.
The ndots:5 multiplier
With ndots:5, a lookup of api.github.com (2 dots < 5) tries the search domains first:
Each of those is done for both A and AAAA — up to 10 queries for one external name. Multiply by every outbound call in a busy cluster and CoreDNS and the VPC resolver's per-ENI limits get hammered.
Mitigations:
- Use FQDNs with a trailing dot for external names in config (
api.github.com.). - Lower
ndotsper pod:
spec:
dnsConfig:
options:
- { name: ndots, value: "2" }
- Deploy NodeLocal DNSCache — a caching DNS agent on every node (DaemonSet on a link-local IP) that absorbs repeated lookups, reduces conntrack pressure, and avoids the UDP conntrack race behind random 5-second delays.
- Scale CoreDNS (cluster-proportional autoscaler or HPA), give it adequate CPU, spread replicas across nodes/AZs.
Debugging DNS in EKS
- From a debug pod:
kubectl run -it --rm dns --image=nicolaka/netshoot -- bash, thendig orders.shop.svc.cluster.local,dig api.github.com,cat /etc/resolv.conf. - CoreDNS health:
kubectl -n kube-system get pods -l k8s-app=kube-dns, logs (enable thelogplugin briefly), Prometheus metrics (coredns_dns_request_duration_seconds,coredns_forward_*, SERVFAIL counts), CPU throttling. - Upstream: does the VPC resolver answer from the node (
dig @169.254.169.253)? Any per-ENI throttling (linklocal_allowance_exceededinethtool -S)? - Conntrack:
conntrack -Sinsert failures on nodes → 5-second race. - NetworkPolicies blocking egress to CoreDNS on UDP/TCP 53.