Kubernetes Incident Response: Handling OOMKilled Pods and Cluster Reliability

Scenario

Your Kubernetes cluster reports a surge of pod restarts due to Out-of-Memory (OOMKilled) errors. Multiple microservices are timing out, and some nodes enter pressure state. The development team confirms a recent release but is unsure of the cause. As an SRE, you must respond quickly to restore service and prevent data loss.

Symptoms

• Users report 502 errors or slow requests.
• kubectl get pods shows many pods in CrashLoopBackOff state.
• kubectl describe pod <name> shows Exit Code: 137 or Reason: OOMKilled.
• Node metrics (e.g., kubectl top node) show memory usage near 100%.
• Kubernetes events (kubectl get events) indicate FailedKillPod or OOMKilling.

Diagnosis

1. Check pod status: Use kubectl get pods --all-namespaces | grep -E 'CrashLoopBackOff|OOMKilled'.
2. View pod events: kubectl describe pod <pod-name> -n <namespace> and look for Reason: OOMKilled.
3. Check node resources: kubectl top nodes and kubectl describe node <node-name> to see memory under pressure.
4. Review container logs: kubectl logs <pod-name> --previous to get logs before crash.
5. Inspect resource configuration: kubectl get deployment <deployment-name> -o yaml and look at resources.requests and limits.memory.

Commands (with safety notes)

# List all failed pods
kubectl get pods --all-namespaces --field-selector=status.phase=Failed | grep -v Completed

# Get details for a specific pod
export POD_NAME=$(kubectl get pods -n production -l app=myapp -o jsonpath='{.items[0].metadata.name}')
kubectl describe pod $POD_NAME -n production

# View logs from previous container (critical)
kubectl logs $POD_NAME -n production --previous

# Get node resource usage (requires metrics-server)
kubectl top node

# Increase memory limit for a deployment (caution: does not restart, but avoids OOM on next restart)
kubectl set resources deployment myapp -n production --limits=memory=512Mi --requests=memory=256Mi

Safety note: Before increasing resource limits, verify node capacity using kubectl describe node to check allocatable memory.

Risk Controls

• Immediate action: If service is unavailable, consider temporarily scaling up replicas (kubectl scale deployment myapp --replicas=5) to spread load.
• Limit concurrency: Use PodDisruptionBudget to prevent simultaneous disruptions.
• Pause auto-rollout: If the issue is caused by a recent change, halt the CI/CD pipeline to prevent further deployments.
• Monitoring alerting: Set up Prometheus alerts for OOMKilled events and node memory pressure.

Rollback

1. Identify problematic release: Check recent rollout history. kubectl rollout history deployment/myapp -n production.
2. Rollback to previous stable version: kubectl rollout undo deployment/myapp -n production --to-revision=<previous-revision>.
3. Verify rollback: Monitor pod status and error rate.
4. If quick rollback not feasible, adjust resource limits (as above) as a temporary mitigation.

Verification

• Pods should be running without crash loops: kubectl get pods --field-selector=status.phase=Running.
• Node memory usage drops (e.g., kubectl top node shows <80%).
• Application endpoints return 200: curl -I https://myapp.example.com/health.
• Events stop showing OOM: kubectl get events --all-namespaces | grep -i OOM.

When to Submit an OpsGlobal Ticket

Submit an OpsGlobal ticket if: - Root cause is unclear (e.g., a memory leak in code that requires development analysis). - Issue persists after adjusting resources, indicating need for scale-up or architectural changes. - You need cluster-level performance tuning or automated incident response. - Node failures or control-plane issues occur and require external expertise.

OpsGlobal engineers can assist with deep analysis, implementing reliability patterns, and autoscaling strategies to improve cluster resilience.