Kubernetes Incident Response: Handling Memory Pressure and Pod Evictions

Scenario

Imagine you are running a production Kubernetes cluster, and suddenly alarms fire: multiple nodes report MemoryPressure, pods are being evicted, and applications are becoming unstable. This is a common but stressful incident. Without a clear playbook, you risk making things worse. This article walks through a realistic memory-pressure incident, giving you reusable commands and decision points to restore service safely.

Symptoms and Alerts

Typical symptoms include:

• Node conditions showing MemoryPressure or NotReady.
• Pods repeatedly evicted (Evicted or Failed status).
• Application health checks failing, increased latency, or user-facing errors.
• Alerting systems (Prometheus, DataDog, etc.) sending MemoryPressure, NodeNotReady, or PodEvicted notifications.

Diagnostic Workflow

1. Assess Cluster State

Start by getting a global view:

kubectl get nodes
kubectl get pods -A -o wide

If many nodes are NotReady, the problem may be control-plane related. If only a few are affected, focus on node-level resource exhaustion.

2. Inspect Node Conditions

Use describe to examine node status:

kubectl describe node <node-name>

Look at the Conditions section, especially whether MemoryPressure is True. Also check Allocated resources to see requests/limits versus actual load.

3. Check Resource Usage at Node and Pod Level

Use top commands for live metrics:

kubectl top node
kubectl top pods -A --sort-by='memory'

Identify the top memory consumers. If a pod's memory usage far exceeds its limit, a memory leak or oversized heap is likely.

4. Review Events and Kubelet Logs

Events record evictions:

kubectl get events --sort-by='.lastTimestamp' -A | grep -i evict

Check kubelet logs (the path varies by distribution):

journalctl -u kubelet -n 100 --no-pager

These logs often contain Evicting and MemoryPressure entries explaining the decision.

Immediate Risk Controls and Mitigation

1. Limit Blast Radius

• Do not delete all replicas: If the app is stateless and replicated, keep at least one healthy replica.
• Use PodDisruptionBudgets (PDBs): If not already configured, create PDBs for critical workloads before any voluntary disruption (e.g., drain).
• Back up configurations: Before making changes, save current resource definitions so you can rollback.

2. Relieve Node Pressure Immediately

If a specific node is under memory pressure, you can:

• Delete unnecessary pods manually: bash kubectl delete pod <pod-name> -n <namespace>
• Consider removing or pausing low-priority batch jobs.
• Safely drain the node after confirming PDBs are in place: bash kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data Safety note: Draining makes the node unschedulable. Perform this during low-traffic windows and remember to uncordon later.

3. Adjust Workload Resource Requests and Limits

Inspect the problematic application's resource specifications. If requests and limits are missing, the scheduler cannot allocate resources efficiently, and kubelet may evict pods due to OOM. Update the manifest:

resources:
  requests:
    memory: "512Mi"
    cpu: "500m"
  limits:
    memory: "1Gi"
    cpu: "1"

Apply the update and restart the deployment:

kubectl rollout restart deployment/<app-name> -n <namespace>

4. Scale Up Capacity

If the cluster is at full capacity, add more nodes or enable autoscaling. On managed Kubernetes, adjust the node pool. In cloud environments, verify the cluster autoscaler is functioning.

Rollback Strategy

If a change you made worsens the incident, revert immediately:

• Rollback a Deployment: bash kubectl rollout undo deployment/<app-name> -n <namespace>
• Revert configuration: Restore backed-up kubelet or cluster configuration files and restart the affected components.

If you drained a node, re-enable scheduling after confirming the issue is resolved:

kubectl uncordon <node-name>

Verification and Post-Incident Review

1. Verify Cluster Recovery

• Check node status: bash kubectl get nodes Confirm all nodes are Ready and free of pressure conditions.
• Check pod status: bash kubectl get pods -A | grep -v Running Ensure no pods are stuck in Pending or Evicted.
• Monitor application health via dashboards or direct request tests.

2. Conduct a Blameless Postmortem

• Reconstruct the timeline and identify the root cause (deploy change, traffic spike, memory leak).
• Use kubectl describe node and historical metrics to analyze trends.
• Define action items: set appropriate resource limits, add PDBs, tune alert thresholds, and consider cluster autoscaling.

When to Engage OpsGlobal

OpsGlobal’s remote DevOps/SRE support team can help in the following situations:

• The cluster is down and you lack 24/7 coverage to respond immediately.
• The root cause is deep-seated (kernel, kubelet, or cloud-provider issues) and your team needs senior expertise.
• You need to restore production quickly without risking further mistakes from low-level operations.
• You want to build a more robust reliability system: playbooks, monitoring improvements, and failure drills.

If any of these apply, submit an OpsGlobal ticket. Our engineers will respond within minutes and provide hands-on remote support to stabilize your cluster.

Conclusion

Kubernetes incident response demands a calm, methodical approach. This memory-pressure scenario is only one example, but mastering these steps prepares you for many cluster resource issues. Remember: swift mitigation, safe operations, and thorough postmortems are the pillars of cluster reliability.