Scenario

At 3:14 AM, your pager fires. A production cluster running customer-facing e-commerce workloads is degrading. Users see 503 errors, and the SRE dashboard shows a spike in evicted pods. You have three nodes in the ubuntu-west1-a group, two in ubuntu-west1-b, and no one else is awake. This is a classic Kubernetes reliability event: node pressure, unschedulable workloads, and cascading failure.

Symptoms

• Pods restarting constantly or stuck in Pending.
• Nodes show a Pressure condition (MemoryPressure, DiskPressure, or PIDPressure).
• Failed evictions in kubectl get events.
• API server latency increases.
• kubectl top nodes may show near-zero CPU but high memory.

Diagnosis

Quickly scope the issue without making changes. Run status checks first:

• kubectl get nodes -o wide – notice node status and taints.
• kubectl describe node <node> – inspect the Conditions section. Look for MemoryPressure=True or DiskPressure=True.
• kubectl top node – find who is consuming resources.
• kubectl get pods --all-namespaces -o wide – see which pods are running where and which are in Error/OOMKilled.

Then focus on a victim pod:

• kubectl get pod <pod> -n <ns> -o yaml – look at lastState.terminated.reason (usually OOMKilled or Evicted).
• kubectl logs <pod> -n <ns> --previous – see the application log before the crash.

Commands to Stop the Bleeding

Do not panic and delete everything. Use the following to stabilize:

# 1. Cordon the unhealthy node to stop new pods landing on it
kubectl cordon <node>

# 2. Drain the node safely (evict pods gracefully, will respect PDBs)
#    Add --ignore-daemonsets if you have DaemonSets like log agents
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

Draining will evict pods, but if a PodDisruptionBudget is misconfigured, it may hang. Add a timeout: --grace-period=120.

If memory pressure is the cause, reduce replica counts or scale down non-critical workloads:

kubectl scale deploy <non-critical-deploy> --replicas=0 -n <ns>

Risk Controls

• Never drain a node without --ignore-daemonsets if DaemonSets are present.
• Always set --delete-emptydir-data only when you know the pods don't need emptyDir data.
• Use PodDisruptionBudgets (PDB) to ensure minimum availability during drains.
• Before forcing deletions, check if the pod is part of a StatefulSet. Deleting a PVC-backed pod can be safe, but you'll lose the local disk state.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: frontend-pdb
  namespace: prod
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: frontend

Rollback

If a recent deployment triggered the event, roll it back before deep-diving:

kubectl rollout undo deployment/<deployment> -n <namespace>

If the cluster is in a bad state (multiple nodes down), use the cluster's disaster recovery plan. For kubeadm or managed clusters, you can't safely rejoin existing worker nodes without fixing the container runtime.

Verification

After cordon/drain/rollback:

kubectl get nodes -o wide
kubectl get pods --all-namespaces -o wide | grep -v Running
kubectl get evictions -n <ns>

Test the application endpoint externally:

curl -I https://app.example.com/healthz

Check monitoring for error rates and latency. Ensure the cluster is stable for at least 10-15 minutes before thinking the incident is over.

When to Escalate to OpsGlobal

You can handle many incidents solo, but request an OpsGlobal expert when:

• More than one node is unavailable and you're unsure which service is the root cause.
• You see persistent disk full after evictions, or Docker overlay reuse is failing.
• The cluster runs a custom kubelet config that you didn't author.
• You need to restore an etcd backup or rebuild a control plane.
• The incident is still unclear after 60 minutes of active debugging.

Our engineers answer within 15 minutes and can help you prevent recurrence by reviewing your resource requests, monitoring, and load testing.