Kubernetes Incident Response: Surviving Node Disk Pressure Without Losing Your Sleep

Scenario

Your on-call phone buzzes at 3 AM: a P1 alert triggers because the checkout service error rate has spiked above 2%. As you log into the cluster, you see multiple replicas terminating, some with CrashLoopBackOff, and one worker node is reporting DiskPressure. This is a common yet dangerous incident—if handled poorly, it can result in cascading evictions and data loss.

Symptoms

• Users see 503/504 errors.
• Prometheus alert HighErrorRate is firing.
• kubectl get pods -n checkout shows many CrashLoopBackOff or Evicted pods.
• kubectl get nodes shows a node with DiskPressure condition.
• Kubelet events contain The node was low on resource: disk.

Diagnosis

1. Start by inspecting recent events cluster-wide:

bash kubectl get events --all-namespaces --sort-by=.lastTimestamp

1. Check node conditions:

bash kubectl describe node worker-3 | grep -A5 Conditions

If DiskPressure=True, the node is running out of disk space or inodes.

1. Access the node (or use kubectl debug node/worker-3 -it --image=busybox) and check disk usage:

bash df -h df -i

1. Find which Pods are consuming disk: inspect the evicted pod's events:

bash kubectl describe pod checkout-xyz -n checkout | grep -A3 Events

1. Look at container logs for excessive or unrotated writes:

bash kubectl logs checkout-xyz --previous -n checkout | tail -100

1. Check kubelet logs on the node to confirm eviction decisions:

bash journalctl -u kubelet --since "30 min ago" | grep -i evict

Risk Controls and Safety Notes

• Never delete PersistentVolumes (PV) or PersistentVolumeClaims (PVC) unless you have verified a backup. Pod eviction is recoverable; data deletion is not.
• Do not use kubectl delete node to force-remove a node; this makes workloads unschedulable and can trigger Cloud Controller Manager issues.
• If using kubectl drain, avoid --ignore-daemonsets=false unless you understand the consequences for critical daemonsets.
• Do not randomly kill Pods; use kubectl delete pod only when the workload is managed by a Deployment/StatefulSet, and even then use kubectl rollout restart if possible.

Rollback

If you scaled a Deployment as a temporary mitigation, revert it to the desired replica count:

kubectl scale deployment checkout --replicas=3

If you modified resource limits or requests, use kubectl rollout undo deployment/checkout to roll back to the previous revision.

Since we haven't changed infrastructure, most rollback actions involve reverting any ephemeral tuning you applied.

Verification

• Confirm Pods are running and ready: kubectl get pods -n checkout should show all Running and Ready.
• Check node condition: kubectl describe node worker-3 | grep -A3 Conditions should show DiskPressure=False.
• Verify application error rate drops in Grafana or your monitoring dashboard.
• Inspect recent logs: kubectl logs -n checkout deployment/checkout --tail=20.

When to Submit an OpsGlobal Ticket

You should escalate to OpsGlobal immediately if:

• You suspect control plane issues (etcd unhealthy, API server timeouts, etcdserver: request timed out errors).
• Multiple nodes are entering NotReady simultaneously.
• The disk pressure persists for more than 30 minutes despite mitigation.
• You see signs of persistent data corruption or potential PVC/PV loss.
• You are uncertain about the root cause and need a second pair of expert eyes.

OpsGlobal provides 24/7 production-grade support to help you stabilize critical clusters and prevent repeat incidents.