Linux SRE Production Troubleshooting Runbook: Kubernetes Node NotReady

Scenario

A node in a production Kubernetes cluster becomes NotReady, causing workloads to be evicted or unschedulable.

Symptoms

• Node status shows NotReady via kubectl get nodes
• kubelet logs errors: journalctl -u kubelet -f shows timeouts or out-of-resource
• High CPU/memory usage on the node (check with top, free -h, df -h)
• Pods on the node are in Pending or Unknown state

Diagnosis

1. Check node status: kubectl get nodes -o wide
2. Describe node: kubectl describe node <node-name> for events and conditions
3. Examine kubelet logs: journalctl -u kubelet --since "10 min ago"
4. System resources: top, htop, free -m, df -h, iostat -x
5. Container runtime: crictl ps or docker ps
6. Network plugin: kubectl get pods -n kube-system to check CNI pods

Key Commands

# Node health
kubectl get nodes
kubectl describe node <node-name>
kubectl get pods --all-namespaces -o wide | grep <node-name>

# System resources
top -c -o %CPU
free -h
vmstat 1 5

# kubelet logs
journalctl -u kubelet --no-pager -n 100

# Restart kubelet (risky)
systemctl restart kubelet

# Drain node (caution)
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Risk Controls

• Identify if the node is a master – never drain or restart kubelet on master nodes without explicit approval, as it may destabilize the control plane.
• Maintain out-of-band access (IPMI, BMC) in case SSH is lost.
• Ensure recent backups or snapshots of critical data.
• Before draining, assess workload impact and avoid peak hours.

Rollback Steps

If the issue stems from a recent change (e.g., config update): 1. Roll back kubelet config: systemctl stop kubelet; cp /etc/kubernetes/kubelet.conf.bak /etc/kubernetes/kubelet.conf; systemctl start kubelet 2. If restarting kubelet doesn't help, consider resource cleanup (remove stale logs, temp files) or scaling up resources. 3. As last resort, reboot the node: reboot (data loss risk).

Verification

• Node status becomes Ready: kubectl get nodes
• Evicted pods reschedule: kubectl get pods -o wide
• kubelet logs show no further errors
• Application monitoring recovers

When to Submit an OpsGlobal Ticket

• Node remains NotReady after all self-service steps (including kubelet restart and node reboot)
• Hardware issues suspected (disk failure, memory errors)
• Data corruption or security incident
• Need for infrastructure team to replace or scale node