Linux SRE Runbook: Production Troubleshooting for Kubernetes Node Network Issues

Scenario

A production Kubernetes worker node running Linux becomes unreachable from the cluster, causing pods to lose connectivity. The node status shows NotReady and application monitoring alerts fire.

Symptoms

• kubectl get nodes shows NotReady status for the node
• kubectl describe node <node> reports KubeletNotReady condition
• Pods on that node stuck in ContainerCreating or CrashLoopBackOff
• Pod IPs unreachable from other nodes (ping/curl timeout)
• kubelet logs contain repeated network errors like network plugin is not ready or failed to get pod status

Diagnosis Steps

1. Check kubelet status bash systemctl status kubelet journalctl -u kubelet -n 100 --no-pager Look for errors such as cni config uninitialized or no such file or directory.

2. Check container runtime (containerd/cri-o) bash crictl info crictl pods Ensure runtime is healthy and CNI plugins are registered.

3. Inspect CNI configuration bash cat /etc/cni/net.d/*.conf ls -la /opt/cni/bin/ Verify config file syntax and presence of plugin binaries.

4. Examine network interfaces and routing bash ip a ip route bridge fdb show Confirm bridge cni0 exists and physical interface is up.

5. Check iptables rules bash iptables -L -n -t nat iptables -L -n -t filter Look for unexpected DROP rules or stale entries.

6. System logs bash dmesg -T | tail -20 grep -i error /var/log/syslog | tail -30 Check for NIC driver errors, MTU mismatches, or kernel warnings.

7. Basic connectivity test bash ping <gateway> curl -I http://<pod-ip>:<port> If gateway fails, issue is likely physical or L2; if gateway works but pod IP fails, CNI layer is the culprit.

Risk Controls

• Never restart kubelet or network services during peak traffic unless the node is already failing. Use kubectl cordon and drain beforehand.
• Backup all config files before modifying /etc/cni/ or /etc/kubernetes/.
• Test changes on a non‑production node first whenever possible.
• Avoid mass node operations — troubleshoot one node at a time to limit blast radius.

Rollback Plan

• Restore CNI configuration from backup and restart kubelet: bash cp /etc/cni/net.d/10-flannel.conflist.bak /etc/cni/net.d/10-flannel.conflist systemctl restart kubelet
• If kubelet restart causes further issues, drain and reset the node: bash systemctl stop kubelet crictl ps -aq | xargs crictl rm systemctl start kubelet
• Revert any sysctl changes: bash sysctl -p /etc/sysctl.conf.bak

Verification

• Node status becomes Ready: bash kubectl get nodes
• All pods transition to Running: bash kubectl get pods -o wide --all-namespaces | grep <node>
• Pod-to-pod connectivity works: bash kubectl exec -it <pod> -- curl http://<other-pod-ip>:<port>
• Application health checks pass (e.g., HTTP 200).

When to Submit an OpsGlobal Ticket

Submit immediately if: - Basic diagnostics fail to identify the root cause after 15 minutes. - Kernel panics, hardware errors (NIC/PCIe), or driver issues are detected. - Node is completely unreachable via SSH (requires out-of-band assistance). - You need help with CNI upgrade or kernel parameter tuning.

OpsGlobal provides 24/7 remote SRE support – upload logs and get expert help fast.