Opscraft
Python & Bash automation · Lesson 1 of 4 · 8 min read

Production-grade Bash

Most Senior DevOps interviews in Bangalore include a scripting round: "parse this log and find the top 10 IPs", "write a script that cleans up unused EBS volumes", "call an API with retries", or a live Bash/Python exercise. The Linux topic covers shell fundamentals; this topic goes further into production-grade Bash, Python for automation (boto3, the Kubernetes client, HTTP APIs), packaging and testing your tools, and worked solutions to the classic coding-round problems.

A robust script skeleton

#!/usr/bin/env bash
# rotate-logs.sh — compress and prune application logs
set -Eeuo pipefail
IFS=$'\n\t'

readonly SCRIPT_NAME=$(basename "$0")
LOG_DIR="/var/log/app"
RETENTION_DAYS=14
DRY_RUN=false

log()  { printf '%s [%s] %s\n' "$(date -u +%FT%TZ)" "$1" "${*:2}" >&2; }
die()  { log ERROR "$*"; exit 1; }
usage() {
  cat <<EOF
Usage: $SCRIPT_NAME [-d dir] [-r days] [-n]
  -d  log directory (default: $LOG_DIR)
  -r  retention in days (default: $RETENTION_DAYS)
  -n  dry run
EOF
}

while getopts ":d:r:nh" opt; do
  case $opt in
    d) LOG_DIR=$OPTARG ;;
    r) RETENTION_DAYS=$OPTARG ;;
    n) DRY_RUN=true ;;
    h) usage; exit 0 ;;
    :) die "option -$OPTARG requires an argument" ;;
    \?) usage; exit 2 ;;
  esac
done

[[ -d $LOG_DIR ]] || die "no such directory: $LOG_DIR"
[[ $RETENTION_DAYS =~ ^[0-9]+$ ]] || die "retention must be a number"

tmpdir=$(mktemp -d)
cleanup() { rm -rf "$tmpdir"; }
trap cleanup EXIT
trap 'log ERROR "failed at line $LINENO: $BASH_COMMAND"' ERR

run() { if $DRY_RUN; then log INFO "DRY-RUN: $*"; else "$@"; fi; }

find "$LOG_DIR" -type f -name '*.log' -mtime +0 -print0 |
  while IFS= read -r -d '' f; do run gzip -9 "$f"; done

find "$LOG_DIR" -type f -name '*.gz' -mtime "+$RETENTION_DAYS" -print0 |
  while IFS= read -r -d '' f; do run rm -f -- "$f"; done

log INFO "done"
Enlarge diagram1. Shebang and strict modeset -Eeuo pipefailstops on errors, unset variables and failed pipes2. Settings and helperslog(), die(), usage()messages go to stderr; stdout stays clean for data3. Parse optionsgetoptsstandard flags such as -d, -r, -n (dry run)4. Validate input[[ -d $DIR ]] || diefail early with a clear message5. Cleanup trapstrap cleanup EXITtemp files and locks are removed even on failure6. The work, dry-run awarerun gzip "$f"safe to test before it changes anything
Figure 1. The parts of a production-grade Bash script, top to bottom, and what each one protects you from.
Swipe sideways to see all of it, or tap to open it full screen.

What each piece buys you:

Element Why
set -e Exit on unhandled command failure (know its gaps: not in if conditions, && chains, or command substitutions in some contexts)
set -u Error on unset variables (catches typos like $LOG_DRI)
set -o pipefail A pipeline fails if any stage fails, not just the last
set -E ERR trap inherited by functions
trap ... EXIT Cleanup always runs (temp files, locks)
getopts Standard option parsing
-print0 / read -d '' Filenames with spaces/newlines handled safely
-- before filenames Prevents names starting with - being treated as options
Logging to stderr stdout stays clean for data piped to other tools
Dry-run mode Safe testing of destructive scripts

Run ShellCheck on every script (in CI and editor); format with shfmt; test with bats for important scripts.

Enlarge diagramWithout strict modeWith set -Eeuo pipefail and trapscurl bad-url | jq . exits 0 because jqsucceededA typo like $LOG_DRI expands to nothingThe script carries on after a failed commandTemp files are left behind on errorsAny failing stage fails the pipelineUnset variables stop the scriptUnhandled failures stop it immediatelyEXIT trap always cleans up; ERR trap logs the line
Figure 2. What strict mode changes. Without pipefail, a failed download piped into jq still "succeeds".
Swipe sideways to see all of it, or tap to open it full screen.

Idioms worth knowing

# defaults and required variables
: "${AWS_REGION:=ap-south-1}"
: "${CLUSTER:?CLUSTER must be set}"

# arrays (keep arguments intact)
args=(--region "$AWS_REGION" --output json)
aws ec2 describe-volumes "${args[@]}"

# associative arrays (bash 4+)
declare -A count
while read -r ip _; do (( count[$ip]++ )) || true; done < access.log
for ip in "${!count[@]}"; do echo "${count[$ip]} $ip"; done | sort -rn | head

# retry with exponential backoff
retry() {
  local max=$1; shift; local n=1 delay=2
  until "$@"; do
    (( n >= max )) && { log ERROR "failed after $n attempts: $*"; return 1; }
    log WARN "attempt $n failed, retrying in ${delay}s"; sleep "$delay"
    (( n++ )); (( delay *= 2 ))
  done
}
retry 5 curl -fsS https://api.example.com/health

# lock to prevent concurrent runs (e.g. from cron)
exec 9>/var/lock/rotate-logs.lock
flock -n 9 || die "another instance is running"

# parallelism with a limit
printf '%s\n' "${hosts[@]}" | xargs -P 8 -I{} ssh -o BatchMode=yes {} 'uptime'

# wait for a condition with timeout
timeout 300 bash -c 'until kubectl rollout status deploy/orders -n orders; do sleep 5; done'

# here-doc to a file with sudo
sudo tee /etc/sysctl.d/99-net.conf >/dev/null <<'EOF'
net.core.somaxconn = 4096
EOF
Enlarge diagramAdd a maximum number of attempts, and only retry operations that are safe to repeat. Adding a little random"jitter" stops many clients retrying in lockstep.1failswait 2 s2failswait 4 s3failswait 8 s4failswait 16 s5succeedsattempt
Figure 3. Retry with exponential backoff: each wait doubles, so a struggling service gets room to recover instead of being hammered.
Swipe sideways to see all of it, or tap to open it full screen.

jq for JSON (AWS CLI, kubectl, APIs)

# instance IDs and Name tags of running instances
aws ec2 describe-instances --filters Name=instance-state-name,Values=running |
  jq -r '.Reservations[].Instances[] |
         [.InstanceId, (.Tags // [] | map(select(.Key=="Name"))[0].Value // "-")] | @tsv'

# pods not Running, with namespace
kubectl get pods -A -o json |
  jq -r '.items[] | select(.status.phase != "Running") |
         "\(.metadata.namespace)/\(.metadata.name) \(.status.phase)"'

# images running in the cluster, counted
kubectl get pods -A -o json |
  jq -r '.items[].spec.containers[].image' | sort | uniq -c | sort -rn

# update a JSON file
jq '.replicas = 3' config.json > config.json.tmp && mv config.json.tmp config.json
Enlarge diagramkubectl get pods -o json{"items": [ pod, pod, … ]}.items[]split the list: one pod at a timeselect(.status.phase != "Running")keep only the pods with a problem"\(.metadata.namespace)/\(.metadata.name)"turn each one into a line of text
Figure 4. How a jq filter works: each stage takes JSON in and passes a smaller piece on, like a pipe in the shell.
Swipe sideways to see all of it, or tap to open it full screen.

The AWS CLI also has --query (JMESPath): aws ec2 describe-volumes --query 'Volumes[?State==\available`].VolumeId' --output text`.

When to switch from Bash to Python

Bash is ideal for gluing commands, short operational scripts and container entrypoints. Switch to Python when you need: data structures beyond arrays, JSON/YAML manipulation, API clients with pagination and retries, error handling and testing, concurrency, or anything over ~100 lines. Interviewers like hearing that judgement.

Enlarge diagramBash is great forSwitch to Python forGluing commands togetherShort operational scriptsContainer entrypointsAnything under ~100 linesJSON, YAML and real data structuresAPI clients with pagination and retriesError handling, tests and concurrencyTools that will grow
Figure 5. Choosing the right tool. Interviewers like hearing this judgement.
Swipe sideways to see all of it, or tap to open it full screen.