Opscraft
Linux · Lesson 5 of 10 · 5 min read

Storage, filesystems and disks

Block devices to mounted filesystems

The layers, bottom to top:

  1. Block device — a disk: /dev/nvme1n1 (EBS on Nitro instances shows up as NVMe), /dev/xvdf on older types.
  2. Partitions (optional) — /dev/nvme1n1p1, managed with fdisk/parted/growpart.
  3. LVM (optional) — physical volumes → volume groups → logical volumes, letting you resize and span disks.
  4. Filesystem — mkfs.ext4 or mkfs.xfs formats the device.
  5. Mount — attach it into the tree: mount /dev/nvme1n1 /data.
Enlarge diagramMount point/data: where the files appear in the treemount /dev/nvme1n1 /dataadd it to /etc/fstab by UUID to survive rebootsFilesystemext4 or XFS: turns blocks into files and foldersmkfs.xfs /dev/nvme1n1formats the device (erases it)LVM (optional)logical volumes that can span and resize diskspvcreate · vgcreate · lvcreatePartition (optional)/dev/nvme1n1p1: a slice of the diskfdisk · parted · growpartBlock device/dev/nvme1n1: the raw disk, e.g. an EBS volumelsblk -fshows every layer at once
Figure 1. From a raw disk to a folder you can use: the layers, bottom to top. Partitions and LVM are optional.
Swipe sideways to see all of it, or tap to open it full screen.
lsblk -f                 # devices, partitions, filesystems, mountpoints
blkid                    # UUIDs
mount | column -t
findmnt /data

Persistent mounts go in /etc/fstab — always by UUID, because NVMe device names can change between boots:

UUID=3f1c...  /data  xfs  defaults,nofail  0  2

nofail means a missing volume won't drop the server into emergency mode. Test with mount -a before rebooting.

ext4 vs XFS: both are solid journaling filesystems. XFS (default on RHEL/Amazon Linux) handles large files and parallel I/O well and can grow but not shrink. ext4 can shrink offline. Grow commands differ: resize2fs for ext4, xfs_growfs for XFS (run on the mountpoint).

Growing an EBS volume with no downtime

A very common real task:

# 1. Modify the volume size in AWS (console/CLI/Terraform). Wait for "optimizing".
lsblk                               # disk shows the new size, partition doesn't yet
sudo growpart /dev/nvme0n1 1        # 2. grow partition 1 to fill the disk
sudo xfs_growfs -d /                # 3a. XFS: grow the filesystem (by mountpoint)
sudo resize2fs /dev/nvme0n1p1       # 3b. ext4: grow the filesystem (by device)
df -h /
Enlarge diagramBeforeeverything is 20 GB1. Grow the EBS volumemodify-volume --size 402. Grow the partitiongrowpart /dev/nvme0n1 13. Grow the filesystemxfs_growfs -d /(resize2fs on ext4)No reboot and no unmount: the application keeps running the whole time. You cannot shrink an EBS volume.disk20 GBpartition20 GBfilesystem20 GBdisk40 GBpartition20 GBfilesystem20 GBdisk40 GBpartition40 GBfilesystem20 GBdisk40 GBpartition40 GBfilesystem40 GB
Figure 2. Growing an EBS volume is three steps, one per layer: the disk, the partition, then the filesystem.
Swipe sideways to see all of it, or tap to open it full screen.

You can't shrink an EBS volume, and AWS limits how often you can change one (since January 2026, up to four modifications in any 24 hours; before that, once every 6 hours).

Space vs inodes vs open files

df -h            # bytes used/free per filesystem
df -i            # inodes used/free
du -sh /var/*    # where the bytes are
du -xh / --max-depth=1 2>/dev/null | sort -rh | head

Three classic "disk full" puzzles:

  1. df says full but du adds up to much less. A process still has a deleted file open — usually a log someone rm'd instead of truncating. The space isn't freed until the FD closes. Find it with lsof +L1 (open files with link count 0), then truncate it in place with : > /proc/<pid>/fd/<fd> or restart the process. Lesson: truncate active logs with : > app.log or truncate -s 0, never rm; and use logrotate with copytruncate or a reload signal.
  2. "No space left on device" but df -h shows free space. Inodes are exhausted (df -i at 100%). Find directories with huge file counts: find / -xdev -type d -size +1M or for d in /var/*; do echo "$(find $d | wc -l) $d"; done | sort -n.
  3. The disk filled overnight. Usually logs, Docker images/containers (/var/lib/docker), journald, core dumps, or a runaway temp directory. Fix now (clean up), then fix permanently (rotation, retention, alerts at 80%).
Enlarge diagram/var/log/app.logname removed with rmjava (PID 4321)still has it open as FD 730 GB of datalink count 0, but blocksstill in useduwalks names: cannot see itdfasks the filesystem: counts itFixFind it with lsof +L1, then restart the process or empty it with : > /proc/4321/fd/7. To empty a live log, truncate it(: > app.log) instead of deleting it.no name nowFD 7
Figure 3. Why df can say "full" while du finds far less: a process still holds a deleted file open.
Swipe sideways to see all of it, or tap to open it full screen.

logrotate

/var/log/orders/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    copytruncate      # or: postrotate systemctl reload orders endscript
}

Swap

Swap is disk used as overflow memory. A little swap absorbs rarely used pages; heavy swapping ("thrashing") makes everything crawl. Kubernetes historically required swap off on nodes so memory limits behave predictably. vm.swappiness (0–100) tunes how eagerly the kernel swaps.