A troubleshooting method
Bottom-up, with evidence at each layer
- Define the failure precisely: who (one client, all clients?), what (DNS, connect, TLS, HTTP status?), since when, intermittent or constant, what changed.
- DNS — does the name resolve, to the expected IP, from the failing client?
dig,getent hosts. - Routing — is there a route?
ip route get, VPC route tables, Reachability Analyzer. - Filtering — SGs, NACLs, host firewalls, NetworkPolicies, proxies. Flow Logs show REJECTs.
- Transport — does the TCP handshake complete?
nc -zv,tcpdumpon both ends. - TLS —
openssl s_client -connect host:443 -servername host. - Application —
curl -v, status codes, app logs, LB access logs.
Reading failure signatures
| Symptom | Most likely meaning |
|---|---|
Connection refused |
Host reached; nothing listening on that port (RST) — service down, wrong port, bound to 127.0.0.1 |
Connection timed out |
Packets dropped — SG/NACL/firewall, missing route, wrong IP, host down |
No route to host |
Routing failure or an ICMP reject from a firewall |
Connection reset by peer |
Remote sent RST mid-connection — app crashed, idle timeout on a proxy/NAT, protocol mismatch |
Name or service not known / NXDOMAIN |
DNS — wrong name, missing record, wrong resolver |
| Works by IP, not by hostname | DNS (resolv.conf, resolver, the record, split-horizon) |
| Small requests work, large hang | MTU / PMTUD |
| Intermittent timeouts only under load | NAT port exhaustion, conntrack table full, LB limits, DNS throttling (VPC resolver per-ENI limits), ephemeral port exhaustion |
| TLS errors from one environment only | Missing CA bundle, TLS-intercepting proxy, clock skew, SNI missing |
| Works from laptop, not server (or reverse) | Different DNS, routes, egress firewall/SG, proxy settings, TLS trust store |
| Some users slow, others fine | Geography/latency, a bad CDN edge or ISP path, DNS-based routing differences |
Tools
dig +short api.example.com ; dig @10.0.0.2 api.internal # DNS
ip -br a ; ip route ; ip route get 10.30.1.15 # local config and routing
ss -tulnp # what's listening
nc -zv db.internal 5432 # TCP connect test
curl -v --connect-timeout 5 https://api.internal/health # full HTTP + TLS
openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null
traceroute -T -p 443 api.example.com ; mtr -rwc 50 api.example.com # path, loss per hop
tcpdump -i any -nn host 10.30.1.15 and port 5432 # packets
iperf3 -s / iperf3 -c host # throughput between two hosts
Proving a firewall drop: run tcpdump on both ends — if SYNs leave the client but never arrive at the server, something in between drops them; if they arrive and SYN-ACKs leave but never reach the client, suspect the return path (NACL outbound rule, asymmetric routing).
Scenario walkthroughs
"Intermittent timeouts to an external API only at peak." Suspect NAT gateway port allocation errors (CloudWatch ErrorPortAllocation), conntrack limits on nodes, or DNS throttling. Fix: connection pooling/keep-alive, more NAT IPs, spreading traffic, NodeLocal DNSCache.
"Users far away see slow pages." Measure where time goes (curl -w timings). Fix with CloudFront for static and cacheable content, regional deployments with Route 53 latency routing, HTTP/2 or HTTP/3, fewer round trips.
"NAT gateway bill is huge." Use Flow Logs/Cost Explorer to find top talkers; add S3/DynamoDB gateway endpoints and ECR/Logs interface endpoints; avoid cross-AZ NAT traffic (one NAT per AZ, routes per AZ); cache images closer (pull-through cache).
"Packet loss suspected between two EC2 instances." mtr and ping -c 1000 for loss, iperf3 for throughput, check instance-type network limits (bandwidth, PPS — ethtool -S shows bw_in_allowance_exceeded, pps_allowance_exceeded on ENA), and placement.