Kubernetes Incident Response: Diagnosing and Resolving Cluster Reliability Issues

Scenario

A production Kubernetes cluster experiences multiple pods repeatedly crashing and entering CrashLoopBackOff, causing service degradation.

Symptoms

• kubectl get pods shows many CrashLoopBackOff states
• kubectl describe pod reveals Last State as Error or OOMKilled
• Nodes show NotReady or high memory/CPU pressure (kubectl top nodes)

Diagnosis

1. Check node health: kubectl get nodes -o wide to see status, journalctl -u kubelet for kubelet logs.
2. View events: kubectl get events --sort-by=.metadata.creationTimestamp to pinpoint timeline.
3. Analyze pod logs: kubectl logs <pod> --previous for last crash logs.
4. Check resource quotas: kubectl describe quota to verify exceeded limits.

Commands

# List crashing pods across all namespaces
kubectl get pods --all-namespaces | grep -E 'CrashLoop|Error|OOM'

# Get detailed events for a specific pod
kubectl describe pod my-app-5d8c9b7f6-abcde -n production

# Check node resources
kubectl top nodes

# View pod resource usage on a node
kubectl describe node worker-node-1 | grep -A 5 'Non-terminated Pods'

Risk Controls

• Before modifying resources, cordon the node: kubectl cordon <node> to prevent new pods from scheduling.
• Safely drain the node: kubectl drain <node> --ignore-daemonsets --delete-local-data
• Confirm sufficient node capacity before increasing resource requests/limits.

Rollback

If caused by a recent configuration change: - Roll back deployment: kubectl rollout undo deployment/my-app - Roll back to a specific revision: kubectl rollout undo deployment/my-app --to-revision=2 - Verify rollback: kubectl rollout status deployment/my-app

Verification

• Check pod status: kubectl get pods -o wide to confirm Running
• Check service endpoints: kubectl get endpoints <service> shows correct IPs
• Application health check: Send HTTP request to verify correct response
• Monitor metrics: Check cluster monitoring (e.g., Prometheus) for recovered CPU/memory

When to Submit an OpsGlobal Ticket

• Nodes completely unreachable and cannot be drained
• Persistent volume failure causing data unavailability
• Control plane component issues (etcd, apiserver)
• Urgent patching needed but lacks internal expertise
• Complex network issues unresolved within 24 hours