Linux SRE Runbook: Production Troubleshooting in Practice

Scenario

A microservice application runs on a 4-core, 8GB Linux server with Nginx and Node.js. Suddenly, monitoring alerts show a spike in HTTP 502 errors, and users cannot access the service.

Symptoms

• Load average >10 (normal <2)
• I/O wait >30%
• Slow SSH response
• dmesg showing multiple "hung_task_timeout_secs" errors

Diagnosis Steps

1. Check system load bash uptime top -bn1 | head -5 Output: load average: 12.5, 10.3, 8.7.

2. Identify high resource processes bash ps aux --sort=-%cpu | head -10 ps aux --sort=-%mem | head -10 Nginx worker processes consuming 300% CPU (likely infinite loop).

3. Analyze I/O bottleneck bash iostat -x 1 3 %util near 100%, await >100ms — disk performance issue.

4. Check disk space and inodes bash df -h / df -i / Disk usage 85%, inodes normal, but log files growing rapidly.

5. Review system logs bash tail -100 /var/log/messages | grep -i error Found: "EXT4-fs error: journal has aborted".

6. Check filesystem consistency bash touch /test && sync && rm /test # test write If fails, schedule fsck during maintenance window; be cautious.

Risk Controls

• Avoid fsck during peak hours: risk of data loss.
• Attempt read-only remount first: bash mount -o remount,ro /dev/sda1 /
• Backup critical data: use dd or rsync after confirming.

Rollback Steps

If recent changes caused the issue (e.g., config update, kernel patch): 1. Rollback application: bash systemctl restart nginx # or load old config 2. Rollback kernel (requires reboot): bash grub2-set-default "CentOS Linux (3.10.0-1160.el7.x86_64) 7 (Core)" reboot 3. If filesystem corrupted, restore from backup or use LVM snapshot.

Verification

• Confirm load normal: uptime
• Confirm I/O reduction: iostat -x 1 2
• Confirm service health: curl -I http://localhost/health
• Check monitoring dashboard: error rate returns to 0.

When to Submit an OpsGlobal Ticket

• Troubleshooting steps fail
• Need fsck but cannot schedule downtime
• Suspected hardware failure (e.g., bad sectors)
• Kernel debugging or performance tuning required
• Need long-term root cause analysis or architecture optimization

Include with ticket: - dmesg and /var/log/messages logs - top and iostat output - Change timeline

This standardized runbook enables SREs to quickly diagnose and recover production issues, and seamlessly escalate to OpsGlobal for expert support.