Kubernetes Incident Response: A Hands-On Guide to Cluster Reliability

Scenario

A worker node goes down due to hardware failure, causing its Pods to be evicted and services to degrade.

Symptoms

• Elevated error rates and increased latency
• Some Pods stuck in Pending state (no available nodes)
• Monitoring alerts (e.g., Node Down, Pod CrashLoopBackOff)

Diagnosis

1. Check node status: kubectl get nodes -o wide to find NotReady nodes.
2. Describe the node: kubectl describe node <node> to see conditions (Ready=False) and events.
3. Identify affected Pods: kubectl get pods --all-namespaces --field-selector spec.nodeName=<node>.
4. Check cluster events: kubectl get events --all-namespaces --sort-by='.lastTimestamp' for eviction records.
5. Use metrics-server or Prometheus to confirm resource pressure.

Commands

# Cordon the node to prevent new schedules
kubectl cordon <node>

# Drain the node (safely migrate Pods)
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

# If node is completely dead, force-delete Pods
kubectl delete pod <pod> --grace-period=0 --force

# Add a new node or bring the old one back
kubectl uncordon <node>

Risk Controls

• Confirm PodDisruptionBudget (PDB) is configured before draining to avoid total service outage.
• Use --ignore-daemonsets to skip DaemonSet pods.
• Be cautious with --delete-emptydir-data if Pods use emptyDir volumes.
• Avoid draining multiple nodes simultaneously.

Rollback

• If drain fails (e.g., PDB prevents it), cancel with kubectl uncordon <node> and reassess.
• For failures caused by updates, use kubectl rollout undo deployment/<name>.
• Recover data from PersistentVolume backups or snapshots if needed.

Verification

• After recovery, kubectl get nodes confirms Ready.
• kubectl get pods --all-namespaces -o wide shows Pods running on correct nodes.
• Test service endpoints for normal functionality.
• Monitor metrics (error rate, latency) return to baseline.

When to Submit an OpsGlobal Ticket

• Incident involves storage, networking, or persistent data.
• Advanced debugging is required (e.g., kernel, CSI plugin issues).
• Your team lacks 24x7 coverage or adequate SLA.
• Root cause is unclear and requires expert analysis.