Scenario

A finance platform's Kubernetes cluster hosts a microservice called ledger-sync. During a routine release, version 2.3.1 introduced a memory leak in a background worker. Within four hours, the cluster begins to show signs of memory pressure: several nodes report MemoryPressure conditions, and kubelet starts evicting low-priority pods to reclaim memory. A pager alert fires: KubeNodeMemoryPressure.

Initial Symptoms

• kubectl get nodes shows multiple nodes with MemoryPressure in the CONDITION column.
• Several pods, including ones from unrelated services, are evicted or restarted.
• kubectl get events -A shows FailedScheduling for pending pods due to Insufficient memory.
• Pod metrics show the ledger-sync worker pod's memory usage climbing steadily to its limit.

Diagnosis Steps

1. Identify the pressure source

Use kubectl top nodes to confirm memory usage. Then drill into workloads:

kubectl top nodes
kubectl top pods -n production --sort-by=memory

The ledger-sync worker pod will likely show near-limit memory.

2. Inspect node and kubelet events

kubectl describe node <nodename> | grep -A10 Conditions
kubectl get events -n production --field-selector involvedObject.name=<pod-name> -o wide

Look for SystemOOM or Evicted reasons.

3. Check resource requests and limits

kubectl describe pod <pod-name> -n production | grep -A2 Requests

If the container has no memory limit, a leak will consume the node. If a limit exists, the container will be OOMKilled.

4. Examine the application log

kubectl logs <pod-name> -n production --previous | tail -50

Look for repeated allocations or error patterns that indicate a leak.

5. Verify HPA and scaling behavior

kubectl get hpa -n production

If the HPA scales based on memory, it may have increased replicas, making the problem worse.

Risk Controls and Safety Checks

• Do not randomly delete pods without understanding the impact. Use kubectl drain only after ensuring the workload is reschedulable.
• Limit blast radius: Use kubectl scale to reduce replica count before rolling back, but be aware that scaling down may not release memory if the existing pods still accumulate.
• Protect critical services: If you must evict pods, use PodDisruptionBudgets (PDBs) to ensure availability.
• Capture evidence: Before making changes, save the current deployment spec: kubectl get deployment ledger-sync -n production -o yaml > ledger-sync-current.yaml.
• Prepare a rollback artefact: Keep the previous image version available in your registry.

Rollback Plan

1. Isolate the faulty workload – Scale the deployment to zero to stop further resource exhaustion:

kubectl rollout undo deployment/ledger-sync -n production --to-revision=<previous-revision>

1. Verify the image tag – Ensure the old image is healthy:

kubectl get deployment ledger-sync -n production -o jsonpath='{.spec.template.spec.containers[].image}'

1. Check for stuck evictions – If nodes remain pressured, cordon the affected nodes and drain them iteratively:

kubectl cordon <nodename>
kubectl drain <nodename> --ignore-daemonsets --delete-emptydir-data --disable-eviction

Warning: drain with --delete-emptydir-data will remove emptyDir data. Use only when you have confirmed the data is not critical.

1. Relieve pressure quickly – If the cluster is critically imbalanced, temporarily delete evicted pods to free metadata:

kubectl delete pods -n production --field-selector=status.phase=Succeeded

Verification

• Node health: kubectl get nodes should show no MemoryPressure.
• Pod stability: kubectl get pods -n production should show all desired pods in Running and Ready.
• Application performance: Monitor the ledger-sync service latency and error rate to ensure the rollback restored normal operation.
• Resource usage: Watch kubectl top nodes over 24 hours to confirm memory usage returns to the baseline.

When to Escalate to OpsGlobal

Submit an OpsGlobal ticket if any of the following occur: - A memory leak is recurring across multiple releases and requires a code-level fix. - Nodes are experiencing SystemOOM kernel events, which can indicate a deeper infrastructure issue. - The cluster is running in a degraded state for more than 30 minutes despite rollback attempts. - You need to design or implement resource quotas, vertical pod autoscaling, or workload rebalancing.

OpsGlobal provides 24/7 incident response, root cause analysis, and cluster hardening so your team can focus on the product while we keep the control plane steady.