Linux SRE Runbook: Production Troubleshooting for Disk Space Exhaustion

Scenario

A production Linux server runs out of disk space, causing application write failures and service disruption. Monitoring alerts show disk usage above 95%. Affected services may include databases, web servers, log collectors, or application runtimes.

Symptoms

• Application logs: write error: No space left on device
• df -h shows 100% utilization on one or more partitions
• System logs (e.g., /var/log/messages) report space exhaustion
• Services fail to start or respond slowly
• Possible inode exhaustion: df -i shows 100% IUse%

Diagnosis

1. Quick disk check bash df -h df -i # check inode usage Identify the affected partition and free space.

2. Find large directories/files bash du -sh /* 2>/dev/null | sort -rh | head -10 du -sh /var/* | sort -rh | head -5 Prioritize /var (logs), /tmp, /home, and application data mounts.

3. Check deleted but held files bash lsof +L1 # list all deleted files still held open These take space but don't appear in du; truncate or restart the holding process.

4. Inspect log rotation bash ls -lh /var/log/*.log systemctl status logrotate cat /etc/logrotate.conf Ensure logrotate is running and properly configured. Manually rotate: logrotate -f /etc/logrotate.conf

5. Find files >1GB bash find / -type f -size +1G -exec ls -lh {} \; 2>/dev/null Review results; decide deletion/compression.

Risk Controls

• Do not rm -rf unknown files without checking dependencies.
• For logs, avoid deleting critical logs (e.g., database binlogs).
• Use truncate -s 0 <file> to empty a file instead of deleting it (preserves file handles).
• Check lsof before removing files held by processes.
• Take a filesystem snapshot (if cloud) or backup critical directories.

Remediation

1. Immediate space recovery - Truncate a large log: truncate -s 0 /var/log/syslog - Clean old temp files: find /tmp -type f -atime +7 -delete - Compress old logs with gzip.

2. Permanent fix - Adjust logrotate retention and compression. - Set up automated cleanup scripts. - Increase disk size or mount additional storage.

3. Handle deleted-but-held files - If the holding process is expendable, restart it: systemctl restart <service> - Otherwise, use > /proc/<pid>/fd/<N> to truncate the file descriptor (dangerous).

Rollback

• If a file is deleted in error, restore from backup (requires pre-existing backup).
• If truncation breaks a service, restart the service (log files are re‑created automatically).
• For truncation causing process crash, remove the file and restart the process.
• Disk expansion is often irreversible; rollback by unmounting new volumes and restarting services.

Verification

• Run df -h to confirm space is under threshold.
• Check application logs for absence of space errors.
• Perform functional test (e.g., write a test file).
• Verify monitoring alert clears.
• Ensure logrotate works: generate new log and check rotation.

When to Submit an OpsGlobal Ticket

• You cannot identify the root cause of space consumption.
• Critical files (e.g., database files) need recovery after deletion.
• Suspected filesystem corruption (dmesg shows I/O errors).
• Long‑term capacity planning or architecture changes required.
• Your team lacks Linux system administration expertise.

OpsGlobal engineers will remotely assist with deep analysis, root cause identification, remediation, and preventive recommendations.