Opscraft
Linux · Lesson 8 of 10 · 5 min read

Troubleshooting runbook: "the server is slow"

Think in a method, not commands

Seniors are expected to troubleshoot systematically. Two methods to name:

  • USE (Brendan Gregg): for every resource — CPU, memory, disk, network — check Utilisation, Saturation (queueing), and Errors.
  • Change first: what changed? Deploys, config, traffic, cron jobs, AWS events. Most incidents follow a change.
Enlarge diagramUtilisation · how busySaturation · queueingErrorsCPUbusy %mpstat -P ALL 1run queue > CPUsvmstat 1 (r column)raremachine checks in dmesgMemoryused vs availablefree -hswappingvmstat si / soOOM killsdmesg | grep -i oomDisk% busyiostat -xz 1 %utilwait time, queueiostat awaitI/O errorsdmesg -TNetworkthroughputsar -n DEV 1drops, retransmitssar -n TCP,ETCP 1interface errorsip -s link
Figure 1. The USE method: for every resource, check Utilisation, Saturation and Errors. Commands to run for each.
Swipe sideways to see all of it, or tap to open it full screen.

The first 60 seconds

uptime                       # load averages vs nproc — rising or falling?
dmesg -T | tail -30          # OOM kills, disk errors, conntrack full, segfaults
vmstat 1 5                   # r (run queue), b (blocked), si/so (swap), us/sy/wa/st
mpstat -P ALL 1 3            # one hot CPU? (single-threaded bottleneck)
pidstat 1 5                  # which processes use CPU
iostat -xz 1 3               # per-disk %util, await (latency), queue size
free -h                      # available memory, swap use
sar -n DEV 1 3               # network throughput per interface
sar -n TCP,ETCP 1 3          # connection rates, retransmits
top                          # overall picture

How to read the key vmstat columns: r greater than CPU count → CPU saturation; b > 0 consistently → processes blocked on I/O; si/so non-zero → swapping; wa high → I/O wait; st high → steal.

Enlarge diagramStart hereWhat changed? Then runuptime, vmstat 1 and topus high: the app is busypidstat, top -H, profile or thread dumpwa high or b > 0: disk-boundiostat -x, iotop; check EBS IOPS limitsst high: CPU stolenburstable T-instance out of creditssi / so non-zero: swappingfree -h; OOM kills in dmesgall quiet: look outsidenetwork, slow dependencies, DNS
Figure 2. "The server is slow": what vmstat and top tell you, and where to look next.
Swipe sideways to see all of it, or tap to open it full screen.

iostat -x: %util near 100% with high await (ms per I/O) means the disk is saturated. On EBS, compare IOPS/throughput to the volume's limits (gp3 baseline 3,000 IOPS / 125 MB/s unless provisioned higher) and to the instance's EBS bandwidth limit.

Going deeper on one process

top -H -p <pid>              # hot threads
strace -fp <pid> -c          # syscall summary: what is it doing?
strace -fp <pid> -e trace=network
lsof -p <pid>                # open files and sockets
ls -l /proc/<pid>/fd | wc -l # FD count
cat /proc/<pid>/status       # memory, threads, state
perf top -p <pid>            # CPU profiling (if perf is installed)

Scenario walkthroughs

Load is 20 on an 8-core box, CPU 90% user. Find the process (pidstat), then the threads (top -H), then what it's doing (profile, thread dump for Java). Was there a traffic spike (check request rate) or a code change (a regex, a loop)? Mitigate by scaling out or rolling back, then fix.

Load is 20, CPU mostly idle, wa high. Disk-bound. iostat -x to find the device, iotop to find the process. Common: a backup or log compression job, a database without enough IOPS, EBS burst balance exhausted on gp2. Mitigate: move the job, raise IOPS (gp3 lets you provision independently of size).

App suddenly can't accept connections. Check ss -s and the listen backlog (ss -ltn shows Recv-Q filling), FD limits, ephemeral port exhaustion on outbound calls ("Cannot assign requested address"), conntrack table full.

Server unreachable over SSH but app is up. sshd crashed, disk full (sshd can't write), SG changed, or memory pressure. On AWS use EC2 Serial Console or SSM Session Manager, and the "Get system log" output.

An EKS node goes NotReady. SSH/SSM in, check systemctl status kubelet containerd, journalctl -u kubelet, disk pressure (df -h /var/lib/containerd), memory pressure, and the CNI (aws-node pod logs).

Making the fix stick

Every troubleshooting answer should end with prevention: an alert that would have caught it earlier (disk at 80%, memory available low, OOM kill events), a limit or quota, log rotation, a runbook entry, and an automated remediation where safe.