Linux SRE Production Troubleshooting Runbook

Scenario

A production web server (running Nginx and PHP-FPM) for an e-commerce platform suddenly experiences high CPU load. User response time increases from 200ms to over 5 seconds, and some requests time out.

Symptoms

• top shows CPU usage >95% sustained, with user space at 80%.
• uptime load average exceeds 2x the number of CPU cores (16 cores).
• Nginx logs show many 504 Gateway Timeout errors.
• Application monitoring indicates PHP-FPM processes reached max limit.

Diagnosis Steps

1. Identify resource bottleneck: Run top, sort by CPU (P), observe multiple PHP-FPM processes consuming high CPU.
2. Trace system calls: Use strace -p <PID> -c -S time to identify time-consuming syscalls; found heavy epoll_wait and recvfrom.
3. Performance sampling: perf top -p <PID> reveals hot functions in php:-- like execute_ex and Zend engine functions.
4. Check slow logs: Examine PHP-FPM slow log (slowlog) reveals multiple requests taking >30 seconds, all calling an external API.
5. Network analysis: netstat -anp | grep :80 shows many TIME_WAIT connections but not hitting limits.

Commands Used

# Monitor CPU usage
watch -n 1 'ps aux --sort=-%cpu | head -20'

# Trace process syscalls
strace -fp $(pgrep -d',' php-fpm) -c -S time 2>&1 | head -30

# Profile hot spots
perf top -p $(pgrep -d',' php-fpm)

# Check slow log
tail -f /var/log/php-fpm/slow.log

# Count network connections
ss -tan | awk '{print $1}' | sort | uniq -c

Risk Controls

• Before diagnosis, ensure monitoring alerts and rollback capability exist (e.g., blue-green deployment or quick restart script).
• Run strace during low traffic to avoid overhead.
• Use timeout for dangerous commands: timeout 10 strace ....
• Avoid kill -9 on production; prefer kill -TERM or systemd stop.

Rollback Operations

1. If caused by new code, rollback to previous stable version: git revert <commit> && systemctl restart php-fpm nginx.
2. If external API timeout, temporarily reduce Nginx proxy_read_timeout from 30s to 5s and enable circuit breaker.
3. If PHP-FPM processes are out of control, kill all children and restart: bash systemctl stop php-fpm killall -9 php-fpm # only if normal stop fails systemctl start php-fpm

Verification

• Use ab or wrk to simulate requests: wrk -t4 -c100 -d30s http://localhost/, observe response time and error rate.
• Check CPU usage returns to normal (<60%).
• Confirm no 504 errors in logs.
• Verify user-reported latency is back to baseline.

When to Submit an OpsGlobal Ticket

• Issue persists >30 minutes and root cause is unclear.
• Requires cross-team coordination (e.g., database or network team).
• High-risk operations (e.g., kernel tuning) needed with low confidence.
• Data loss or complete service outage (submit emergency ticket immediately).
• Need post-mortem assistance or second opinion to improve runbook.