Mastering Kubernetes Incident Response: A Practical Guide for Cluster Reliability

Scenario

A critical deployment experiences resource contention during high traffic, causing increased latency and partial request timeouts.

Symptoms

• Pod restart count spikes (kubectl get pods shows CrashLoopBackOff).
• Application returns 503 errors.
• Node CPU/memory usage exceeds 80%.
• User reports slow response times.

Diagnosis

1. Check node resources: kubectl top nodes shows utilization.
2. Inspect Pod status: kubectl get pods -o wide identifies abnormal Pod distribution.
3. Deep dive into Pod: kubectl describe pod <pod-name> for events and resource limits.
4. Log analysis: kubectl logs <pod-name> --tail=100 to find errors.
5. Cluster events: kubectl get events --sort-by='.lastTimestamp' to pinpoint triggers.

Commands Example

# View node resources
kubectl top nodes

# List non-running Pods across namespaces
kubectl get pods --all-namespaces | grep -v Running

# Describe deployment
kubectl describe deployment <deployment-name>

# Tail logs for a label
kubectl logs --tail=50 -l app=<app-label>

Risk Controls

• Set ResourceQuotas to prevent single team from exhausting cluster resources.
• Use PriorityClass to ensure critical services are scheduled first.
• Configure PodDisruptionBudget to avoid full outage during rolling updates.
• Enable HorizontalPodAutoscaler for critical deployments.

Rollback Procedure

# Roll back to previous revision
kubectl rollout undo deployment/<deployment-name>

# Roll back to specific revision
kubectl rollout undo deployment/<deployment-name> --to-revision=3

# Verify rollback status
kubectl rollout status deployment/<deployment-name>

Verification

• Monitor metrics: request latency (P99), error rate, pod restart count.
• Canary deployment: deploy new version to a small subset and observe.
• Load testing: simulate traffic using hey or wrk to ensure performance.

When to Submit an OpsGlobal Ticket

• Root cause unclear and issue persists beyond 30 minutes.
• Need cluster-level audit, e.g., etcd slow queries, control plane anomalies.
• Internal team lacks post-incident review experience.
• Need expert assistance in resource optimization or architecture.

OpsGlobal's SRE team provides 24/7 support to help you recover quickly and prevent recurrence.