Storage, filesystems and disks
Block devices to mounted filesystems
The layers, bottom to top:
- Block device — a disk:
/dev/nvme1n1(EBS on Nitro instances shows up as NVMe),/dev/xvdfon older types. - Partitions (optional) —
/dev/nvme1n1p1, managed withfdisk/parted/growpart. - LVM (optional) — physical volumes → volume groups → logical volumes, letting you resize and span disks.
- Filesystem —
mkfs.ext4ormkfs.xfsformats the device. - Mount — attach it into the tree:
mount /dev/nvme1n1 /data.
lsblk -f # devices, partitions, filesystems, mountpoints
blkid # UUIDs
mount | column -t
findmnt /data
Persistent mounts go in /etc/fstab — always by UUID, because NVMe device names can change between boots:
UUID=3f1c... /data xfs defaults,nofail 0 2
nofail means a missing volume won't drop the server into emergency mode. Test with mount -a before rebooting.
ext4 vs XFS: both are solid journaling filesystems. XFS (default on RHEL/Amazon Linux) handles large files and parallel I/O well and can grow but not shrink. ext4 can shrink offline. Grow commands differ: resize2fs for ext4, xfs_growfs for XFS (run on the mountpoint).
Growing an EBS volume with no downtime
A very common real task:
# 1. Modify the volume size in AWS (console/CLI/Terraform). Wait for "optimizing".
lsblk # disk shows the new size, partition doesn't yet
sudo growpart /dev/nvme0n1 1 # 2. grow partition 1 to fill the disk
sudo xfs_growfs -d / # 3a. XFS: grow the filesystem (by mountpoint)
sudo resize2fs /dev/nvme0n1p1 # 3b. ext4: grow the filesystem (by device)
df -h /
You can't shrink an EBS volume, and AWS limits how often you can change one (since January 2026, up to four modifications in any 24 hours; before that, once every 6 hours).
Space vs inodes vs open files
df -h # bytes used/free per filesystem
df -i # inodes used/free
du -sh /var/* # where the bytes are
du -xh / --max-depth=1 2>/dev/null | sort -rh | head
Three classic "disk full" puzzles:
dfsays full butduadds up to much less. A process still has a deleted file open — usually a log someonerm'd instead of truncating. The space isn't freed until the FD closes. Find it withlsof +L1(open files with link count 0), then truncate it in place with: > /proc/<pid>/fd/<fd>or restart the process. Lesson: truncate active logs with: > app.logortruncate -s 0, neverrm; and use logrotate withcopytruncateor a reload signal.- "No space left on device" but
df -hshows free space. Inodes are exhausted (df -iat 100%). Find directories with huge file counts:find / -xdev -type d -size +1Morfor d in /var/*; do echo "$(find $d | wc -l) $d"; done | sort -n. - The disk filled overnight. Usually logs, Docker images/containers (
/var/lib/docker), journald, core dumps, or a runaway temp directory. Fix now (clean up), then fix permanently (rotation, retention, alerts at 80%).
logrotate
/var/log/orders/*.log {
daily
rotate 14
compress
delaycompress
missingok
notifempty
copytruncate # or: postrotate systemctl reload orders endscript
}
Swap
Swap is disk used as overflow memory. A little swap absorbs rarely used pages; heavy swapping ("thrashing") makes everything crawl. Kubernetes historically required swap off on nodes so memory limits behave predictably. vm.swappiness (0–100) tunes how eagerly the kernel swaps.