Linux SRE Production Troubleshooting Runbook: High System Load

Scenario

A production web server experiences overall slowness, with users reporting page timeouts. Monitoring shows system load average consistently above 80% of CPU cores.

Symptoms

• Application request latency increases significantly
• SSH login feels sluggish
• top or uptime shows high load average
• May be accompanied by high CPU usage, high I/O wait, or memory swapping

Diagnosis Steps

1. Quick system check: bash uptime top -bn1 | head -n 20 free -h df -h
2. Analyze CPU usage: bash mpstat -P ALL 1 5 # Focus on %usr, %sys, %iowait
3. Identify high-CPU processes: bash ps aux --sort=-%cpu | head -10 pidstat -u 1 5
4. Check disk I/O (if %iowait high): bash iostat -x 1 5 # Look at await, %util
5. Trace system calls (carefully): bash strace -p <PID> -c -S time 2>&1 | head -20 # run only 5 seconds
6. Kernel logs: bash dmesg --level=err,warn | tail -20

Risk Controls

• Before running diagnostics on production, prefer a replica or low-traffic node
• Limit strace duration (use -S or timeout) to avoid hanging processes
• Avoid kill -9 in production; use kill -TERM or restart services
• Log all commands executed and their output

Rollback Steps

If a mistake is made or recovery needed: - If a critical process was killed, restart the service immediately: systemctl restart <service-name> - If system parameters (e.g., vm.swappiness) were changed, revert and reload: sysctl -p - If temporary tools were installed, remove and clean: yum remove -y <tool> or apt-get purge

Verification

• Confirm load has dropped: uptime and top show normalized load average
• Verify application response: test an API endpoint with curl for timing
• Check that monitoring alerts clear

When to Submit an OpsGlobal Ticket

• Root cause is unclear (e.g., suspected hardware failure)
• Load spikes persist uncontrollably
• Deep debugging (e.g., perf analysis) or kernel tuning needed
• Capacity expansion or architectural changes required