Mastering Linux SRE Runbooks: A Practical Guide to Production Troubleshooting

As an SRE, you know that production incidents are inevitable. The difference between a minor blip and a major outage often comes down to how quickly and systematically you can diagnose and resolve the issue. A well-structured runbook is your first line of defense. In this post, we’ll walk through a real-world scenario involving a Kubernetes node that becomes unresponsive due to a kubelet hang, using a runbook-driven approach. You’ll learn practical commands, risk controls, rollback strategies, and when to escalate to OpsGlobal.

Scenario

Imagine a typical Tuesday morning. Your monitoring stack (Prometheus) sends an alert: NodeNotReady for node-prod-03. This node runs a mix of critical workloads, including a payment API and a database replica. The API error rate is climbing, and some pods are stuck in Terminating. You need to act fast, but also methodically.

Symptoms

• kubectl get nodes shows node-prod-03 as NotReady.
• Pods on that node are in Pending or Terminating state.
• kubectl describe node node-prod-03 shows KubeletNotReady condition with a reason like PLEG is not healthy or Docker Daemon not running.
• System load on the node is exceptionally high (e.g., load average > 100).
• The node may be unresponsive to SSH.

Diagnosis

Start with read-only commands to gather data without changing state. Never jump to restarts without understanding the root cause.

1. Check Node Status and Conditions

kubectl get nodes -o wide
kubectl describe node node-prod-03

Look for the Conditions section. Common reasons for NotReady: - KubeletNotReady with PLEG is not healthy (often caused by container runtime hangs). - MemoryPressure, DiskPressure, PIDPressure, or NetworkUnavailable.

2. Access the Node

If SSH is responsive, log in. If not, rely on out-of-band management (e.g., iDRAC, IPMI) or cloud console.

ssh sre@node-prod-03

3. Inspect Kubelet Logs

Kubelet logs often contain the smoking gun.

journalctl -u kubelet -n 200 --no-pager

Look for recurring errors like PLEG is not healthy, Failed to connect to containerd, or Unhealthy probes.

4. Check System Resources

Run top, vmstat, free -h, df -h to assess CPU, memory, swap, and disk.

top -bn1 | head -20
vmstat 1 5
free -m
df -h

High CPU load could be due to a runaway process, OOM kernel panic, or an infrastructural issue (e.g., noisy neighbors on a shared hypervisor).

5. Kernel Messages

dmesg can reveal OOM kills, disk I/O errors, or hardware issues.

dmesg --time-format iso | tail -100

6. Container Runtime Health

Since Kubernetes 1.24+, the default runtime is containerd. Check its status.

systemctl status containerd
journalctl -u containerd -n 100 --no-pager

7. Check for Zombie Processes

A zombie process storm can break kubelet. Inspect:

ps aux | awk '$8=="Z" {print}'

Risk Controls

Before making any changes, assess the blast radius:

• Cordon the node to prevent new workloads from scheduling: kubectl cordon node-prod-03.
• If workloads are critical, consider draining the node, but only after understanding the issue. Draining can cause downtime if the services are not rescheduled properly.
• Avoid restarting kubelet unless you have a clear reason. Restarting kubelet may not fix an underlying system issue and could cause a thundering herd of pod restarts.
• Use systemctl to restart services only when necessary, and always check for dependent components.
• Never kill processes without verifying their PID and parent. Use ps -ef to confirm.

Rollback

If you make a change and it worsens, you need a plan to revert.

• If you restarted kubelet and the node comes back but then immediately goes unstable, you may need to roll back to a previous kernel or system package. That is beyond runbook scope, but you should document the exact steps.
• If you drained the node, use kubectl uncordon node-prod-03 to allow scheduling again.
• If you changed a sysctl or configuration file, back up the original file (e.g., cp /etc/sysctl.conf /etc/sysctl.conf.bak) and restore it.

Always test rollback in a staging environment when possible. In an emergency, you can rely on your configuration management system (e.g., Ansible, Terraform) to reapply known-good state.

Verification

After remediation, verify that the node is healthy and the workloads are back to normal.

1. Check node status: bash kubectl get nodes
2. Check node conditions: bash kubectl describe node node-prod-03 | grep Conditions -A5
3. Verify pods are running: bash kubectl get pods --field-selector spec.nodeName=node-prod-03 -o wide
4. Monitor system load and kubelet logs for a few minutes: bash uptime journalctl -u kubelet -f
5. Confirm the API error rate is back to baseline via your monitoring dashboard.

When to Submit an OpsGlobal Ticket

You can handle many incidents with a runbook, but some situations require a deeper dive. Submit a ticket to OpsGlobal if:

• You cannot determine the root cause despite following this guide.
• The node continues to go NotReady after multiple restarts.
• You see signs of hardware failure (e.g., smartctl errors, memory ECC errors).
• You need assistance with a kernel-level issue or vendor-specific support.
• The incident has a business impact and you need a second pair of eyes while you continue working.

At OpsGlobal, we provide 24/7 remote DevOps and SRE support. Our engineers are experienced in Linux and Kubernetes troubleshooting. We can help you build more robust runbooks, diagnose complex incidents, and get your services back online faster.

This runbook is a living document. Update it based on what you learn in each incident. The goal is to reduce MTTR and increase confidence when the pager goes off.