Production Troubleshooting for Linux SRE: A Practical Runbook

Scenario

A critical e-commerce application server begins to experience increased response latency. Users report page timeouts. The server runs CentOS 7 with 64 CPU cores and 256 GB RAM, hosting Nginx and a Java application.

Symptoms

• SSH login is sluggish; commands take 2-3 seconds to output.
• uptime shows load average: 55.23, 48.19, 32.54 (above 80% of the 64-core CPU).
• top reveals several Java processes consuming up to 3000% CPU (multi-threaded).
• iostat -x 1 shows disk %util consistently at 99%, await > 100ms.

Diagnosis Steps

Step 1: Overall Resource Check

# Check load, memory, and I/O
top -b -n1 | head -20
free -h
vmstat 1 5
iostat -x 1 3

Findings: High user CPU, low system CPU; sufficient memory; disk I/O is anomalous.

Step 2: Identify I/O Source

# Identify processes with heavy disk I/O
iotop -oP
# Check file system status
df -h
dstat --top-io --top-bio 1 5

The Java process is writing massive logs, and the /var/log partition is at 95% usage.

Step 3: Analyze Log Writing

# Check log file sizes
ls -lh /var/log/app/*.log
# Use strace to trace write syscalls (CAUTION: not on production unless necessary)
strace -p <PID> -e trace=write -c -S time 2>&1 | head -20

Confirmation: The logging framework is configured for synchronous writes, generating huge logs on each GC.

Risk Controls

• Do not kill -9 critical processes.
• Notify the team before changing log levels (audit logs may be lost).
• Use ionice to reduce I/O priority: ionice -c 2 -n 7 -p <PID>.
• Rotate logs safely (stop writes first or use copytruncate): logrotate -f /etc/logrotate.d/app.

Solution and Rollback

Short-term: Compress and rotate logs

# Manual rotation
mv /var/log/app/current.log /var/log/app/current.log.$(date +%Y%m%d%H%M%S)
kill -HUP <PID>  # Notify process to reopen file
# Configure logrotate for daily compression, keep 7 days

Long-term: Change logging to async

Modify log4j2.xml Appender:

<RollingFile name="RollingFile" fileName="/var/log/app/app.log" filePattern="/var/log/app/app-%d{yyyy-MM-dd}-%i.log.gz">
  <PatternLayout pattern="%d{ISO8601} [%t] %-5p %c - %m%n"/>
  <Policies>
    <TimeBasedTriggeringPolicy/>
    <SizeBasedTriggeringPolicy size="100 MB"/>
  </Policies>
  <DefaultRolloverStrategy max="7"/>
  <AsyncLogger>
    <AppenderRef ref="RollingFile"/>
  </AsyncLogger>
</RollingFile>

Test in staging before applying to production.

Rollback

If the change fails, restore the original log configuration and restart:

cp /etc/app/log4j2.xml.bak /etc/app/log4j2.xml
systemctl restart app

Verification

• Run iostat -x 1 again; %util should be <30%, await <10ms.
• Stress test Nginx with ab or wrk; response time should normalize.
• Monitor load average below 70% of CPU cores for 5 consecutive minutes.

When to Submit an OpsGlobal Ticket

• Root cause is unclear after following the above steps.
• Kernel parameters (e.g., vm.dirty_ratio) need tuning but you lack production access.
• Suspected hardware failure (e.g., bad sectors) requiring replacement.
• Troubleshooting exceeds 2 hours without progress.

Submit via OpsGlobal console with: - Time window and impact scope. - Executed diagnostic commands and outputs (sanitized). - OS version, kernel parameters, and relevant config backups.