Linux SRE Runbook: Production Troubleshooting for High Load Incidents

Scenario A production web server (RHEL 8) experiences high load average (e.g., 50+ on a 16-core machine), causing slow response times and intermittent 503 errors. The on-call SRE needs to identify the culprit process and restore normal operation without causing downtime.

Symptoms - Load average > number of CPU cores - uptime shows load above threshold - top shows a process consuming >90% CPU or memory - Application health checks fail intermittently - New connections to the server time out

Diagnosis 1. SSH into the server (using bastion host). 2. Run uptime to confirm load. 3. Run top -o %CPU -n 1 to identify the top CPU consumer. 4. Use ps aux --sort=-%cpu | head for detailed process info. 5. If process is unknown, check with systemctl status <service> or ls -l /proc/<PID>/exe. 6. Use strace -p <PID> -c -S time 2>&1 | head -20 to see syscall profile (if strace is available, but avoid long runs in production). 7. Check logs: journalctl -u <service> --since "5 min ago". 8. Verify if the process is a legitimate application or malicious.

Commands

# Check load
uptime
# Top processes
top -b -o %CPU -n 1 | head -20
# Detailed process info
ps -p <PID> -o pid,ppid,user,%cpu,%mem,cmd,etime
# Check open files
lsof -p <PID>
# If process is stuck, send SIGTERM
kill -TERM <PID>
# If unresponsive, kill -KILL (last resort)
kill -KILL <PID>
# Restart service
systemctl restart <service>

Risk Controls - Never kill a process without identifying it first. - Use SIGTERM (15) before SIGKILL (9). - If using strace, limit to short duration (e.g., -c for summary). - Ensure you have a rollback plan (e.g., backup configuration, deployment artifacts). - Avoid running commands that might exacerbate load (e.g., find / without restrictions).

Rollback If the killed process was critical: 1. Restore from backup: cp /backup/config /etc/service/config 2. Restart service: systemctl restart service 3. Verify with systemctl status service and application health endpoint.

Verification - Run uptime again; load should drop. - Check top to ensure process is terminated. - Access application health endpoint (e.g., curl http://localhost:8080/health). - Monitor for 5 minutes: watch -n 1 'uptime; ps aux --sort=-%cpu | head -5'.

When to Submit an OpsGlobal Ticket - If the root cause is unclear or the issue recurs. - If you suspect a memory leak or kernel bug. - If the process is part of a complex distributed system and requires cross-team investigation. - When you need to escalate to senior SRE or infrastructure team.