Opscraft
Linux · Lesson 3 of 10 · 9 min read

Processes, signals and systemd

How processes are born

A process is created with fork() (copy the parent) followed by exec() (replace the copy's program with a new one). Every process has a PID, a parent PID (PPID), a UID, an environment, a working directory, open FDs, and resource limits. ps -ef --forest or pstree -p shows the family tree.

Enlarge diagrambashPID 100Linux kernelchildPID 200fork()new process: a copy of bashexecve("/usr/bin/ls")now it is ls:prints the file listexit(0)wait() returns 0 → saved in $?
Figure 1. Running ls from bash: the shell copies itself with fork(), the copy becomes ls with exec(), and the shell waits for the exit status.
Swipe sideways to see all of it, or tap to open it full screen.

Threads are execution units inside one process that share its memory and FDs. Java and Go apps run many threads; ps -eLf or top -H shows them. A thread dump (jstack) is how you find a Java thread stuck on a lock.

Process states

In ps / top, the STAT column shows:

State Meaning
R Running or runnable (waiting for CPU)
S Interruptible sleep — waiting for an event (most processes)
D Uninterruptible sleep — usually waiting on disk or NFS I/O; can't be killed until the I/O returns
Z Zombie — exited, but parent hasn't collected the exit status
T Stopped (Ctrl-Z or SIGSTOP) or being traced

Many processes in D state is a strong signal of storage problems (a hung NFS mount, an EBS volume at its throughput limit).

Enlarge diagramS · sleepingwaiting for an eventnew processfork()R · runningon a CPU or queued for oneZ · zombieexited, status not yetcollectedT · stoppedpaused (Ctrl-Z)D · uninterruptiblewaiting on disk or NFSgonePID freedwaits / wakesdisk I/OCtrl-Z / SIGCONTexitsparent calls wait()
Figure 2. The life of a process and the letters you see in the STAT column of ps and top.
Swipe sideways to see all of it, or tap to open it full screen.

Zombies take no memory, only a PID-table slot. You can't kill them — they're already dead. The parent must call wait(); if the parent is buggy, kill or restart the parent, and the zombies get re-parented to PID 1, which reaps them.

Orphans are live processes whose parent died; they're adopted by PID 1 (or the nearest "subreaper").

This is exactly why containers need a proper PID 1: if your app is PID 1 and never reaps children, zombies accumulate. docker run --init (tini) or init: true in Compose fixes it.

Signals

Signal Number Default Notes
SIGHUP 1 Terminate Often "reload config" for daemons (nginx, sshd)
SIGINT 2 Terminate Ctrl-C
SIGKILL 9 Kill Cannot be caught, blocked or ignored; no cleanup
SIGTERM 15 Terminate The polite default of kill; apps should catch it and shut down gracefully
SIGSTOP / SIGCONT 19 / 18 Pause / resume SIGSTOP can't be caught
SIGCHLD 17 Ignore Sent to a parent when a child exits

Graceful shutdown everywhere — systemd, Docker, Kubernetes — follows the same pattern: send SIGTERM, wait a grace period, then SIGKILL. Kubernetes waits terminationGracePeriodSeconds (30 s default). An app that ignores SIGTERM always gets killed hard after the timeout, dropping in-flight requests.

Enlarge diagramYour appsystemd · Docker ·Kubernetesgrace period: 10 s in Docker, 30 s in Kubernetes by defaultApp handles SIGTERMSIGTERM: please stopstops taking new work,finishes in-flight requestsexits cleanly, code 0App ignores SIGTERMSIGTERM: please stopSIGKILL: cannot be caughtkilled mid-request: exit code 137 = 128 + 9
Figure 3. Every graceful shutdown (systemd, Docker, Kubernetes) follows this pattern. Exit code 137 means the process was killed with SIGKILL.
Swipe sideways to see all of it, or tap to open it full screen.

Exit codes follow a convention: 0 success, 1–125 app errors, 126 not executable, 127 command not found, and 128 + signal number when killed by a signal — so 137 = 128 + 9 (SIGKILL, often the OOM killer) and 143 = 128 + 15 (SIGTERM).

kill -TERM 4321       # ask nicely
kill -9 4321          # last resort
pkill -f 'java.*orders'
kill -HUP $(cat /run/nginx.pid)   # reload config

Priorities

The scheduler gives CPU time based on nice values from −20 (highest priority) to 19 (lowest). nice -n 10 ./backup.sh starts a job at lower priority; renice -n 5 -p PID changes a running one. Only root can raise priority (lower the number). For disk I/O priority, ionice -c3 (idle class) keeps a backup from hurting the app.

systemd

systemd is PID 1 and the service manager. Services are units defined in files:

# /etc/systemd/system/orders.service
[Unit]
Description=Orders API
After=network-online.target
Wants=network-online.target

[Service]
User=orders
WorkingDirectory=/opt/orders
EnvironmentFile=/etc/orders/env
ExecStart=/opt/orders/bin/orders --port 8080
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
MemoryMax=1G

[Install]
WantedBy=multi-user.target
systemctl daemon-reload            # after editing unit files
systemctl enable --now orders      # start now and at boot
systemctl status orders            # state, PID, last log lines, exit code
systemctl restart orders
systemctl list-units --failed
systemctl cat orders               # effective unit including drop-ins
systemctl edit orders              # create an override drop-in safely

Key ideas:

  • enable creates the boot-time symlink; start runs it now. They're independent.
  • Restart=on-failure gives you supervision without extra tools.
  • LimitNOFILE sets the FD limit for the service — ulimit in your shell does not apply to systemd services.
  • MemoryMax, CPUQuota use cgroups, the same mechanism containers use.
  • Other unit types: .timer (scheduling), .socket, .mount, .target (groups of units).
Enlarge diagramsystemctl start orderssystemctl enable orderssystemctl enable --now ordersstart now and at every bootRuns the service right nowDoes nothing for the next rebootCreates a symlink in multi-user.target.wants/Starts it automatically at every bootDoes not start it now
Figure 4. "start" and "enable" are independent. Use systemctl enable --now to do both.
Swipe sideways to see all of it, or tap to open it full screen.

journald stores logs from all units:

journalctl -u orders -f                 # follow one service
journalctl -u orders --since "1 hour ago" -p err
journalctl -b -1                        # previous boot (needs persistent journal)
journalctl -k                           # kernel ring buffer, like dmesg
journalctl --disk-usage
journalctl --vacuum-size=500M           # trim

Scheduling: cron and timers

cron uses five fields — minute hour day-of-month month day-of-week:

# m   h  dom mon dow  command
*/15  *  *   *   *    /opt/scripts/sync.sh          # every 15 minutes
0     2  *   *   1-5  /opt/scripts/backup.sh        # 02:00 Mon–Fri
30    3  1   *   *    /opt/scripts/monthly.sh       # 03:30 on the 1st
Enlarge diagram302**1-5backup.shthe command to runRuns at 02:30, Monday to Fridayminute0-59hour0-23day of month1-31month1-12day of week0-7, Sun = 0* = every value*/15 = every 15th1-5 = a range1,15 = a list
Figure 5. The five time fields of a cron line, read left to right.
Swipe sideways to see all of it, or tap to open it full screen.

User crontabs via crontab -e; system ones in /etc/crontab and /etc/cron.d/ (which include a user field).

Watch out: cron runs with a minimal environment — a short PATH, no profile, no ~/.bashrc. Scripts that work in your terminal fail in cron. Use absolute paths, set variables explicitly, and redirect output (>> /var/log/job.log 2>&1) so failures aren't silent.

systemd timers are the modern alternative: logging in journald, Persistent=true to run missed jobs after downtime, randomised delays, and dependencies.

To keep a process alive after logging out: nohup cmd & (ignores SIGHUP), tmux/screen, or — properly — a systemd unit.