Shell scripting and text processing
Streams, redirection and pipes
cmd > out.log # stdout to file (truncate)
cmd >> out.log # append
cmd 2> err.log # stderr only
cmd > all.log 2>&1 # both: redirect stdout first, THEN point stderr at it
cmd &> all.log # bash shorthand for the same
cmd 2>/dev/null # discard errors
cmd1 | cmd2 # stdout of cmd1 → stdin of cmd2
cmd | tee out.log # see it AND save it
Order matters: cmd 2>&1 > file sends stderr to the terminal, because stderr was duplicated before stdout moved.
Exit codes and safe scripts
Every command returns an exit code; $? holds the last one. && runs the next command only on success, || only on failure.
Start production scripts with strict mode:
#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'
-e— exit on any failing command.-u— error on unset variables (catches typos likerm -rf "$BUILDDIR/"with an empty variable).-o pipefail— a pipeline fails if any command fails, not just the last. Without it,curl bad-url | jq ."succeeds" becausejqsucceeded.
Clean up with a trap:
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
Variables and quoting
name="api"
echo "$name" # always quote expansions
echo "${name}-svc" # braces when adjacent text follows
echo "${ENV:-dev}" # default if unset/empty
echo "${#name}" # length
files=(a.yaml b.yaml) # arrays
for f in "${files[@]}"; do kubectl apply -f "$f"; done
out=$(git rev-parse --short HEAD) # command substitution
export AWS_REGION=ap-south-1 # visible to child processes
Unquoted variables undergo word splitting and globbing — the source of many bugs with spaces in filenames.
Loops, conditions and functions
if [[ -f /etc/os-release ]]; then source /etc/os-release; fi
[[ "$ENV" == "prod" ]] && echo "careful"
for host in web1 web2 web3; do ssh "$host" uptime; done
while read -r line; do echo "got $line"; done < hosts.txt
retry() { # retry a command with backoff
local n=0 max=5
until "$@"; do
n=$((n+1)); (( n >= max )) && return 1
sleep $((2**n))
done
}
retry curl -fsS https://internal/health
Test operators: -f file exists, -d directory, -z empty string, -n non-empty, -eq/-lt/-gt numeric.
The text toolkit
grep -rn "ERROR" /var/log/app/ # recursive, with line numbers
grep -v DEBUG app.log # invert match
grep -E 'timeout|refused' app.log # extended regex
grep -c 500 access.log # count matching lines
find /var/log -name '*.log' -mtime +7 -size +100M
find . -type f -name '*.tmp' -delete
find . -name '*.yaml' -print0 | xargs -0 -n1 kubeval # safe with spaces
cut -d: -f1 /etc/passwd # field 1, colon-delimited
awk '{print $1}' access.log # first whitespace field
awk -F, '$3 > 100 {sum += $3} END {print sum}' data.csv
sed -n '100,120p' big.log # print lines 100–120
sed -i 's/old-host/new-host/g' app.conf # in-place replace
sort | uniq -c | sort -rn | head # frequency count
jq -r '.items[].metadata.name' pods.json
Classic interview one-liners:
# Top 10 client IPs in an nginx access log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
# Count of each HTTP status code
awk '{print $9}' access.log | sort | uniq -c | sort -rn
# 5xx errors per minute
awk '$9 ~ /^5/ {print substr($4,2,17)}' access.log | uniq -c
# Largest 10 directories under /var
du -xh /var 2>/dev/null | sort -rh | head -10
uniq only collapses adjacent duplicates — always sort first. find matches names and attributes; grep matches contents.
Watch out:
sed 's/a/b/' file > fileproduces an empty file — the shell truncates the output file before sed reads it. Usesed -i(or write to a temp file andmv).