Mastering Linux SRE Runbooks: A Production Troubleshooting Deep Dive

Scenario

Your organization runs a production Kubernetes cluster on Linux-based nodes. On-call SREs receive alerts for increased latency in application responses and occasional HTTP 503 errors. The cluster uses multiple workers, but one node consistently shows high load averages. The business impact is noticeable during peak hours.

Symptoms

• Application response time p95 spikes from 300ms to over 2s.
• kubectl top nodes shows high CPU/memory usage on one node.
• Users report intermittent timeouts.
• Node load average (1m) exceeds the number of CPU cores.
• dmesg reveals OOM (Out-Of-Memory) events or disk I/O warnings.

Diagnosis

Start from a high-level view and narrow down. Use these Linux commands to gather evidence.

1. Check Overall System Load

uptime gives load averages over 1, 5, and 15 minutes. A 1-minute load significantly higher than 15-minute suggests a recent spike.

top -b -n 1 shows processes sorted by CPU usage. Pay attention to wa (I/O wait) and st (steal time).

2. Analyze CPU and Memory

vmstat 1 5 reports process, memory, paging, block IO, traps, and CPU activity. Look at r (runnable processes), b (blocked processes), si, so (swap in/out), us, sy, id, wa.

free -m shows memory usage. Check for low available memory and high swap usage.

ps aux --sort=-%mem | head -20 identifies the top memory consumers.

3. Inspect Disk I/O

iostat -x 1 reports disk utilization, I/O wait, and queue size. High %util or high await indicates disk bottleneck.

df -h checks filesystem space; full disks cause application failures.

4. Review System Logs

dmesg -T | tail -30 shows kernel messages, including OOM killer events, hardware errors, or filesystem issues.

journalctl -u kubelet --since "1 hour ago" highlights kubelet problems (e.g., failing to mount volumes, node status flapping).

5. Check Kubernetes Node Health

kubectl describe node <node-name> reveals conditions like MemoryPressure, DiskPressure, PIDPressure. Also shows container runtime issues.

kubectl get events --sort-by='.lastTimestamp' surfaces recent cluster events.

Risk Controls

• All the above commands are read-only and safe for production.
• Do not run stress or sysbench on production nodes unless explicitly approved.
• Avoid kill -9 on critical processes; use systemctl restart only after confirming the process is not the cause.
• Before making any configuration changes, save current settings (e.g., sysctl -a > /tmp/sysctl.bak).

Rollback

• If you adjust kernel parameters (e.g., sysctl -w vm.swappiness=10), restore them with sysctl -p /etc/sysctl.conf or reboot.
• If you modify Kubernetes node labels or taints, reverse them immediately using kubectl label node <name> <key>- or kubectl taint node <name> <key>-.
• If you restart kubelet, monitor it; if it fails, use systemctl status kubelet to see the error and revert any changed files.

Verification

After applying fixes, re-run key commands:

• uptime should show declining load averages.
• vmstat should show low wa and plenty of idle CPU.
• Application p95 latency should return to baseline.
• kubectl get nodes shows all nodes in Ready status.

When to Submit an OpsGlobal Ticket

Contact OpsGlobal when: - The issue persists after you have performed the basic steps. - You need 24/7 coverage for ongoing observation. - Root cause is beyond your team's Linux or Kubernetes expertise. - You need an independent second opinion on performance-affecting changes.

Our SRE team can take over troubleshooting, apply hotfixes, and provide a detailed postmortem.