Opscraft
Linux · Lesson 9 of 10 · 4 min read

Security hardening and fleet operations

Hardening a Linux server

A checklist you can talk through:

  1. Patch — automatic security updates (unattended-upgrades, dnf-automatic) or, better, immutable rebuilds from a patched AMI on a schedule.
  2. Minimal install — remove unused packages and services; fewer listening ports (ss -tlnp should be short).
  3. SSH — key-only auth (PasswordAuthentication no), no root login (PermitRootLogin no), AllowUsers/AllowGroups, restrict source IPs in the security group. Better still, no inbound SSH at all: use SSM Session Manager (IAM-authenticated, logged to CloudWatch/S3, no open port 22, no keys to rotate).
  4. Least privilege — services run as dedicated non-root users; narrow sudo rules; no shared accounts.
  5. MAC — keep SELinux/AppArmor enforcing.
  6. Audit and logging — auditd rules for sensitive files, logs shipped off-box (CloudWatch agent, Fluent Bit) so attackers can't erase them.
  7. Filesystem — noexec,nosuid,nodev on /tmp where possible; correct permissions on secrets (600, owned by the service user).
  8. Kernel — sysctl hardening (net.ipv4.conf.all.rp_filter=1, disable IP forwarding where not needed, kernel.randomize_va_space=2).
  9. Benchmarks — measure against CIS benchmarks; AWS Inspector scans EC2 for CVEs.
  10. Secrets — never in shell history, scripts or user data in plain text; fetch from Secrets Manager/SSM Parameter Store at runtime.
Enlarge diagramSSH through a bastionSSM Session ManagerPort 22 open to some networkSSH keys to hand out, rotate and revokeBastion server to patch and monitorSession history stays on the boxNo inbound ports at allIAM decides who can connectEvery session logged to CloudWatch or S3Works for private instances through the SSMagent
Figure 1. Two ways to get a shell on a server. Session Manager removes the open port and the keys.
Swipe sideways to see all of it, or tap to open it full screen.

Package management

Family Low-level High-level Examples
Debian/Ubuntu dpkg apt apt update && apt install -y nginx, apt list --upgradable
RHEL/Amazon Linux/Fedora rpm dnf (yum on older) dnf install -y nginx, rpm -qf /usr/bin/curl (which package owns a file)
Alpine — apk apk add --no-cache curl

Pin versions for reproducibility in images; use internal mirrors/proxies (Artifactory, Nexus) in locked-down environments.

Managing many servers without Ansible

Not every team uses a configuration management tool such as Ansible. If you haven't used one, have a strong answer ready for "how would you push a config change to 50 servers?":

  • Immutable infrastructure — bake the change into the AMI (Packer or EC2 Image Builder) and roll the Auto Scaling group with an instance refresh. No drift, easy rollback.
  • Launch templates and user data for bootstrap-time configuration.
  • SSM Run Command / State Manager — run a document across instances selected by tags, with rate control and error thresholds, output logged to S3. State Manager reapplies periodically to fix drift.
  • SSM Patch Manager for OS patching windows.
  • For Kubernetes nodes, you rarely touch nodes at all — replace them (Karpenter drift / node group updates).
Enlarge diagramMake the changea patch or config update, in codeBake a new AMIPacker or EC2 Image BuilderUpdate the launch templatepoint it at the new AMIInstance refreshAuto Scaling replaces a fewservers at a timeHealth checks passnew servers take trafficOld servers terminatedno drift; rollback = previous AMI123456
Figure 2. Immutable servers: never edit a running server. Bake a new image and replace the servers a few at a time.
Swipe sideways to see all of it, or tap to open it full screen.

Then add honestly: "Ansible does the same push-based configuration over SSH with idempotent modules; the concepts — inventory, idempotency, roles — map directly, and I'd ramp up quickly."

Time, logs and certificates

  • Keep clocks synced (chronyd; on AWS the Amazon Time Sync Service at 169.254.169.123). Clock skew breaks TLS validation, AWS request signing ("Signature expired"), JWTs and distributed logs.
  • Centralise logs and include host, service and request IDs.
  • Trust store: CA certificates live in /etc/ssl/certs / /etc/pki/ca-trust; add a corporate CA with update-ca-certificates (Debian) or update-ca-trust (RHEL).