Mastering Cloud Capacity Autoscaling and Cost Operations: A Practical SRE Guide

Scenario

Imagine you are running an e-commerce platform on Kubernetes, with pods scaling up and down based on CPU and memory. During a promotional event, the cluster scales to hundreds of nodes, and the cloud bill spikes. After the event, the nodes remain underutilized, but your invoice stays high. This is a common problem when autoscaling is not aligned with cost governance.

Symptoms

• Cloud bill increases without a corresponding traffic spike.
• Pods are evicted due to insufficient node resources.
• Cluster nodes show low average utilization (<20%).
• High number of scale-up and scale-down events causing thrashing.
• Latency spikes even when resources appear available.

Diagnosis

Start with Kubernetes metrics:

kubectl get hpa -n <namespace>
kubectl describe hpa <hpa-name> -n <namespace>
kubectl top nodes
kubectl top pods -n <namespace>

Check cluster autoscaler logs:

kubectl logs -f deployment/cluster-autoscaler -n kube-system

Review cloud cost explorer to identify cost drivers: EC2, EKS, storage, data transfer, etc.

Common issues: - HPA configured with default metrics that lag under bursty traffic. - Cluster autoscaler scale-down thresholds too conservative. - Running multiple replicas of stateful apps with idle replicas. - Using on-demand instances when spot instances could reduce costs significantly.

Commands

Here are commands to inspect and adjust:

# View HPA configuration
kubectl get hpa -A

# Check resource utilization
kubectl top nodes --sort-by=cpu

# Scale down a deployment temporarily
kubectl scale deployment <deployment> --replicas=1 -n <namespace>

# Update HPA min/max
kubectl autoscale deployment <deployment> --min=1 --max=10 --cpu-percent=70 -n <namespace>

# Edit HPA
kubectl edit hpa <hpa-name> -n <namespace>

# Enable cluster autoscaler on a node pool (AWS example)
aws eks update-nodegroup-config --cluster-name <cluster> --nodegroup-name <ng> --scaling-config minSize=1,maxSize=5

# Use spot instances (Azure example)
az aks nodepool add --resource-group <rg> --cluster-name <aks> --name spotpool --node-count 1 --spot-max-price -1 --priority Spot

Risk Controls

• Set hard budgets in the cloud provider (e.g., AWS Budgets, GCP Budget Alerts).
• Use Kubernetes resource requests/limits to prevent pods from consuming more than needed.
• Implement PodDisruptionBudgets to maintain availability during scaling.
• Configure HPA with custom metrics based on request latency or queue depth.
• Use spot instances with a mix of on-demand for critical workloads.

Rollback

• Keep previous HPA configuration files in source control. Use kubectl apply -f previous-config.yaml to roll back.
• For cluster autoscaler changes, revert node pool scaling settings.
• If you scaled down replicas, restore with kubectl scale deployment <deployment> --replicas=<original>.

Verification

• Use load testing tools (e.g., k6, locust) to simulate traffic and observe scaling.
• Monitor cloud cost dashboard for stabilization.
• Check that utilization remains within target range (e.g., 40-70%).
• Verify that there are no pod evictions or throttling events.

When to Submit an OpsGlobal Ticket

If you face: - Complex scaling behaviors that are hard to diagnose. - Recurring cost anomalies despite optimization attempts. - Need for architecture review or implementation of advanced autoscaling. - Time-sensitive incident where you need a second pair of eyes.

Submit a ticket in the OpsGlobal portal and include the diagnostics you've gathered.