Kubernetes Incident Response: A Field Guide to Surviving Node Pressure and Pod Evictions

Scenario

It’s 2 a.m., and Prometheus is firing alerts. Your production Kubernetes cluster is serving a critical microservice, but p99 latency has spiked and a handful of HTTP 503s are showing up. Initial checks reveal one worker node is NotReady, while other nodes are experiencing MemoryPressure and DiskPressure. Resource contention is clearly destabilizing the cluster, and pods are being evicted—but the root cause is still unknown.

Symptoms

• Elevated response times and intermittent timeouts.
• Node conditions flip to MemoryPressure or DiskPressure.
• kubectl get events floods with Evicted and OOMKilling messages.
• Pod restarts keep climbing, with CrashLoopBackOff appearing on critical services.
• kubectl top nodes shows one or more nodes exceeding 90% CPU or memory utilization.

Diagnosis

A structured diagnosis distinguishes between a node-level resource meltdown and a rogue pod consuming everything. Start horizontally, then drill down.

1. Cluster health snapshot: Run kubectl get nodes and kubectl get events --all-namespaces --sort-by=.lastTimestamp.
2. Resource usage overview: Use kubectl top nodes and kubectl top pods -A to identify hotspots.
3. Node condition inspection: kubectl describe node <node> | grep -A20 'Conditions' reveals pressure conditions and taints.
4. Pod-level details: For suspicious workloads, check kubectl describe pod <pod> -n <namespace> and kubectl get pod <pod> -o yaml.
5. HPA expansion check: If autoscaling is in play, run kubectl get hpa -A and inspect min/max values.
6. Historical metrics: Use Prometheus queries like node_memory_available and container_cpu_usage_seconds_total to see how long the trend has been building.

Commands

# Fast cluster health
kubectl get nodes -o wide
kubectl get events --all-namespaces --sort-by=.lastTimestamp

# Resource usage top
kubectl top nodes
kubectl top pods -A

# Inspect node pressure
kubectl describe node <node> | grep -A20 'Conditions'

# Inspect problematic pods
kubectl get pod <pod> -n <namespace> -o yaml
kubectl describe pod <pod> -n <namespace>

# HPA behavior
kubectl get hpa -A
kubectl describe hpa <hpa-name> -n <namespace>

# Metrics API health
kubectl get apiservices | grep metrics

Safety note: Before any destructive action (e.g., deleting a node or pod), verify the current replica count and ensure minimum availability. Never run kubectl delete node casually; cordoning and draining is the safer path.

Risk Controls

1. Don’t delete nodes immediately—snowball eviction can destabilize the entire cluster.
2. Avoid global restarts: kubectl rollout restart can increase load; schedule during low traffic if necessary.
3. Isolate unstable nodes: kubectl cordon <node> stops new pods, then gradually drain with kubectl drain --ignore-daemonsets.
4. Throttle ingress: Temporarily shift traffic away from affected nodes using service mesh or ingress rules.
5. Adjust HPA limits: Set a hard cap on replicas to prevent runaway autoscaling.

Rollback

If the incident follows a recent change, rollback should be your first response.

• HPA: Revert to previous min/max values, or delete the HPA to lock replica count.
• Deployment image: Execute kubectl rollout undo deployment/<name> to revert to the last known-good image.
• Resource quotas: If a LimitRange or ResourceQuota was misconfigured, restore the original YAML.
• Manual drains: If you initiated a drain that stuck, cancel or complete it with kubectl uncordon after resolving the pod disruption budget issue.

Verification

Recovery is not just about seeing green again—confirm the root cause is gone.

• All nodes report Ready and no pressure conditions persist.
• kubectl get events remains quiet (no new Evicted or OOMKilled).
• Latency metrics return to baseline; no 503s.
• Observe HPA scaling behavior for a few minutes to ensure stability.

When to Submit an OpsGlobal Ticket

Escalate to OpsGlobal if:

• The incident lasts more than 30 minutes and the root cause still isn’t clear.
• Pods are in CrashLoopBackOff with no obvious fix from your end.
• You suspect underlying infrastructure issues but lack platform-level access.
• Your team is exhausted and needs 24/7 remote SRE backup.

When opening a ticket, add the core kubectl get nodes, kubectl describe node, and event logs. OpsGlobal engineers can hit the ground running with that data.