Troubleshooting runbook: "the server is slow"
Think in a method, not commands
Seniors are expected to troubleshoot systematically. Two methods to name:
- USE (Brendan Gregg): for every resource — CPU, memory, disk, network — check Utilisation, Saturation (queueing), and Errors.
- Change first: what changed? Deploys, config, traffic, cron jobs, AWS events. Most incidents follow a change.
The first 60 seconds
uptime # load averages vs nproc — rising or falling?
dmesg -T | tail -30 # OOM kills, disk errors, conntrack full, segfaults
vmstat 1 5 # r (run queue), b (blocked), si/so (swap), us/sy/wa/st
mpstat -P ALL 1 3 # one hot CPU? (single-threaded bottleneck)
pidstat 1 5 # which processes use CPU
iostat -xz 1 3 # per-disk %util, await (latency), queue size
free -h # available memory, swap use
sar -n DEV 1 3 # network throughput per interface
sar -n TCP,ETCP 1 3 # connection rates, retransmits
top # overall picture
How to read the key vmstat columns: r greater than CPU count → CPU saturation; b > 0 consistently → processes blocked on I/O; si/so non-zero → swapping; wa high → I/O wait; st high → steal.
iostat -x: %util near 100% with high await (ms per I/O) means the disk is saturated. On EBS, compare IOPS/throughput to the volume's limits (gp3 baseline 3,000 IOPS / 125 MB/s unless provisioned higher) and to the instance's EBS bandwidth limit.
Going deeper on one process
top -H -p <pid> # hot threads
strace -fp <pid> -c # syscall summary: what is it doing?
strace -fp <pid> -e trace=network
lsof -p <pid> # open files and sockets
ls -l /proc/<pid>/fd | wc -l # FD count
cat /proc/<pid>/status # memory, threads, state
perf top -p <pid> # CPU profiling (if perf is installed)
Scenario walkthroughs
Load is 20 on an 8-core box, CPU 90% user. Find the process (pidstat), then the threads (top -H), then what it's doing (profile, thread dump for Java). Was there a traffic spike (check request rate) or a code change (a regex, a loop)? Mitigate by scaling out or rolling back, then fix.
Load is 20, CPU mostly idle, wa high. Disk-bound. iostat -x to find the device, iotop to find the process. Common: a backup or log compression job, a database without enough IOPS, EBS burst balance exhausted on gp2. Mitigate: move the job, raise IOPS (gp3 lets you provision independently of size).
App suddenly can't accept connections. Check ss -s and the listen backlog (ss -ltn shows Recv-Q filling), FD limits, ephemeral port exhaustion on outbound calls ("Cannot assign requested address"), conntrack table full.
Server unreachable over SSH but app is up. sshd crashed, disk full (sshd can't write), SG changed, or memory pressure. On AWS use EC2 Serial Console or SSM Session Manager, and the "Get system log" output.
An EKS node goes NotReady. SSH/SSM in, check systemctl status kubelet containerd, journalctl -u kubelet, disk pressure (df -h /var/lib/containerd), memory pressure, and the CNI (aws-node pod logs).
Making the fix stick
Every troubleshooting answer should end with prevention: an alert that would have caught it earlier (disk at 80%, memory available low, OOM kill events), a limit or quota, log rotation, a runbook entry, and an automated remediation where safe.