Kubernetes Incident Response: Node Failure and Cluster Reliability

Scenario

A node goes down due to hardware failure or kernel panic, causing pods on that node to become unreachable and affecting service availability.

Symptoms

• kubectl get nodes shows a NotReady status for the node.
• Affected pods are in Pending or Unknown state.
• Alerts fire (e.g., NodeDown, PodCrashLoopBackOff).
• Users report service degradation or outages.

Diagnosis

1. Identify the unhealthy node: kubectl get nodes.
2. Inspect node conditions: kubectl describe node <node-name> — look for Ready: False and the underlying reason (e.g., KubeletNotReady or NodeStatusUnknown).
3. Check recent cluster events: kubectl get events --sort-by='.lastTimestamp'.
4. If possible, access the node (via SSH or out-of-band) and examine kubelet logs: journalctl -u kubelet -n 100.
5. Verify control plane health (etcd, scheduler, etc.).

Commands

Cordon the node (prevent new pods)

kubectl cordon <node-name>

Drain the node (evict pods)

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
Risk Notes:
- Drain will not recreate bare pods (those without a controller).
- Draining a StatefulSet with replicas=1 may cause data loss.
- Using --delete-emptydir-data deletes temporary data; ensure it is safe.

Uncordon after recovery

kubectl uncordon <node-name>

Risk Controls

• Configure PodDisruptionBudgets (PDBs) for critical workloads to limit disruptions during drainage.
• Use multiple replicas and anti-affinity for stateful applications.
• Regularly test drain procedures and backup etcd.
• Before draining, verify that any attached PVCs (especially RWO) can be reattached elsewhere.

Rollback

• If draining causes service issues, immediately run kubectl uncordon <node-name> and interrupt the drain (Ctrl+C).
• If the node is permanently dead, delete the node object: kubectl delete node <node-name> — pod rescheduling will happen automatically.

Verification

• After node recovery, confirm node is Ready: kubectl get nodes.
• Ensure all pods are running on healthy nodes: kubectl get pods -o wide.
• Verify service endpoints and latency return to normal.

When to Submit an OpsGlobal Ticket

• Multiple nodes are failing or nodes keep flipping states.
• Control plane components are impacted (e.g., etcd unavailable).
• Risk of persistent data loss exists.
• Pods cannot be rescheduled after drain due to resource constraints or configuration issues.
• A root cause analysis is needed for recurring incidents.