Linux SRE Runbook: Taming a Production Server Under High Load

Linux SRE Runbook: Taming a Production Server Under High Load

Scenario

It's 2 AM and your pager goes off. A critical application server is showing a load average of 30 on a 16-core machine. Customer requests are timing out. This runbook is your guide to systematically diagnose and resolve such issues without escalating the incident.

Symptoms

• Load average consistently exceeds the number of CPU cores (check with nproc).
• SSH session feels sluggish; commands take seconds to return.
• Application latency spikes, and health checks start failing.
• uptime shows high 1-, 5-, and 15-minute averages.

Diagnosis Steps

1. Quick Snapshot
   Run uptime and cat /proc/loadavg to confirm the load. Also note the CPU count (nproc). This sets the baseline.

2. Find the Hottest Processes
   Use top -b -n 1 | head -30 for a snapshot. Or interactive htop if installed. For a cleaner listing, use: bash ps -eo pid,ppid,user,cmd,%cpu,%mem --sort=-%cpu | head Look for processes consuming the most CPU. If nothing obvious, check I/O and memory next.

3. CPU vs I/O Bound
   Run vmstat 1 5. The us (user), sy (system), and wa (I/O wait) columns are key. If wa is consistently high (e.g., >30%), the issue is disk I/O. If us/sy dominate, it's CPU-bound.

4. Drill into Disk I/O
   For I/O-bound systems, use iostat -x 1 to see per-device utilization and queue length. iotop (if available) shows per-process I/O. Look for processes like log daemons, database checkpoints, or backups hammering the disks.

5. Check Memory Pressure
   Use free -m and check for swap usage. A system that is swapping heavily can appear to have high load. Look at dmesg for oom-killer messages. High memory pressure can also cause OS-level thrashing.

6. Review System Logs
   Suddenly, check for errors in the last few minutes: bash journalctl -p err -n 100 --since "10 min ago" Look for disk errors, systemd service failures, or repeated crashes.

7. Trace the Process
   If a specific process is stuck in a loop, attach strace carefully. For example, trace system calls for 10 seconds and summarize: bash timeout 10 strace -c -p $PID This can reveal endless futex calls (lock contention) or file I/O. For deeper kernel profiling, perf top (as root) may show hot functions.

Risk Controls

• Do not kill processes blindly. Always check with ps -o user,pid,cmd -p $PID to see what it is and who owns it.
• Use kill -TERM (graceful) before kill -KILL.
• If the process is not critical, reduce its priority instead: renice +20 -p $PID.
• For I/O-heavy processes, use ionice -c3 -p $PID to set idle class.
• Never modify a configuration file without copying it first. Use cp file file.bak.
• Test any systemd unit changes with systemd-analyze verify /path/to/service.

Rollback Plan

• If you changed a service config, validate it and then restart: systemctl restart my-service.
• If you removed a suspected cron job, restore the crontab from backup.
• If you installed a temporary tracing package, uninstall it after the incident.
• Always document what you changed and when, so you can revert if needed.

Verification

• Re-run uptime and confirm load average drops to a healthy range (e.g., <= 1.5x CPU count for your workload).
• Hit the service with a few test requests: curl -w "@curl-format.txt" -o /dev/null -s https://your-api/health to measure latency.
• Watch vmstat and iostat for 5-10 minutes to ensure the system is stable.
• Check logs for any new errors: journalctl -p err --since "5 min ago".

When to Submit an OpsGlobal Ticket

OpsGlobal provides 24/7 remote SRE support. Submit a ticket when: - The load remains high after you've addressed the obvious process or I/O issue. - You suspect a kernel bug, driver issue, or filesystem corruption that requires deep expertise. - The root cause is not clear, and the business is losing money while you search. - You need a second opinion on a complex configuration or performance tuning task.

Our SREs can analyze kernel crash dumps, use advanced tracing tools, and help you build a long-term fix. You'll also get a post-incident report to improve your monitoring.

Remember, a systematic approach saves time. Don't panic, follow this runbook, and know when to ask for help.