Production Runbook: Troubleshooting High CPU Load on Linux Servers

Scenario

A production Linux server becomes unresponsive, monitoring alerts show CPU usage persistently above 90%.

Symptoms

• SSH connections are slow or time out
• Application response times increase
• top or uptime shows high load averages

Diagnosis

1. SSH into the server using a management IP or out-of-band management (e.g., iDRAC).
2. Run top -b -n1, press P to sort by CPU, identify the top consumer.
3. Use ps -eo pid,ppid,cmd,%cpu,%mem --sort=-%cpu | head for confirmation.
4. Examine the process: ls -l /proc/<PID>/exe and strace -p <PID> -c -S time 2>&1 to trace syscalls.
5. Use perf top if installed for performance profiling.
6. Check system logs: journalctl -u <service> --since "5 minutes ago".

Commands

• Gather process info: ps -p <PID> -o pid,ppid,user,start,etime,%cpu,%mem,cmd
• Analyze threads: top -H -p <PID> or ps -L -p <PID>
• Memory map: pmap -x <PID>
• Stack trace: pstack <PID> or gdb -batch -ex "thread apply all bt" -p <PID>
• Adjust priority with nice: renice +10 <PID> (lower CPU contention)

Risk Controls

• Confirm the process is not critical before killing it.
• Use kill -STOP <PID> to pause the process instead of terminating.
• If the process is orphaned or zombie, check the parent; restart the parent service if needed.
• For non-core services, use systemctl restart <service> instead of kill.
• Avoid running perf or strace during peak hours as they add overhead.

Rollback

• If a process is killed by mistake, immediately restart the service: systemctl start <service> or manual command.
• If priority adjustment causes other issues, restore defaults with renice 0 <PID>.
• If failover affected cluster health, switch to standby node.

Verification

• Run top and confirm CPU usage is normal (e.g., <50%).
• Check application health endpoint: curl -I http://localhost:8080/health.
• Monitor logs for new errors: tail -f /var/log/syslog | grep -i error.
• Verify response times with load testing tools.

When to Submit an OpsGlobal Ticket

• Unable to determine the process origin or ownership.
• The process repeatedly restarts or cannot be killed (e.g., stuck in D state).
• Root cause analysis requires code review or kernel-level investigation.
• Patches or configuration changes need change management approval.