Scenario

Your team receives an alert: a production Kubernetes node has been reporting a high load average for 15 minutes, and disk usage has crossed 90%. User-facing services are slow, and some pods are restarting. You need to investigate immediately without making things worse.

Symptoms

Typical signals include: - Node load average exceeds the number of CPU cores by 2x or more. - df -h shows /var/lib/docker or /var/lib/containerd above 90%. - Pods are in CrashLoopBackOff or ImagePullBackOff. - kubectl get nodes shows NotReady after a few minutes.

Diagnosis

Start with top-level visibility:

uptime
kubectl get nodes -o wide
kubectl describe node NODE_NAME

Check overall load and CPU:

top -bn1 | head -20
vmstat 1 5

If you see high wa (I/O wait), inspect disks:

iostat -x 1 5
df -h

Find large files or directories causing disk pressure:

du -x -h --max-depth=2 /var/lib | sort -h | tail -20

Look for orphaned logs or container files:

find /var/log -type f -size +100M -exec ls -lh {} \;

Check system messages and Kubelet service:

dmesg -T | tail -50
journalctl -u kubelet --since "15 minutes ago"

If containers are failing, inspect them with CRI:

crictl ps -a
crictl logs CONTAINER_ID --tail 100

Risk Controls

Never delete files unless you are absolutely sure they are disposable. Run read-only commands first. Capture the output for later analysis. Do not restart the node unless you have already tried to identify the root cause. If disk is full, do not blindly remove application data – use du and find to locate logs and temporary files. Use touch and truncate to rotate logs in production only when the service is not writing critical data.

Rollback

If the issue is due to a recent deployment, roll back:

kubectl rollout undo deployment/YOUR_DEPLOYMENT -n YOUR_NAMESPACE

If a log file grew unexpectedly, move it aside:

mv /var/log/example.log /var/log/example.log.old
systemctl restart rsyslog   # only if rsyslog is the logger

If container runtime storage is full, clean orphaned images and volumes safely:

crictl rmi --prune
crictl rm --prune   # use with caution

Only after careful verification, restart the Kubelet:

sudo systemctl restart kubelet

Verification

After each action, check the load and disk levels:

uptime
df -h
kubectl get nodes -o wide
kubectl get pods -o wide | grep YOUR_APP

Also watch the event channel:

kubectl get events --sort-by=.metadata.creationTimestamp

If the node stabilises, keep monitoring for at least 30 minutes before closing the incident.

When to Submit an OpsGlobal Ticket

Submit a ticket if: - You cannot identify the root cause within 30 minutes. - The load remains high after you have cleaned up obvious culprits. - You need to recover data from a failed mount or corrupted filesystem. - Your on-call engineer is overwhelmed and needs second pair of eyes. - You require a proactive performance review of your Kubernetes nodes.

OpsGlobal provides production-grade support for Linux and Kubernetes. We step in when your runbook stops being enough.