Cloud Capacity Autoscaling and Cost Operations: A Playbook for Migrated Workloads

Scenario:

Your team recently migrated a production Kubernetes estate to a managed cloud service. You enabled autoscaling on day one: Horizontal Pod Autoscaler (HPA) for application workloads, Cluster Autoscaler (CA) to add nodes, and maybe Vertical Pod Autoscaler (VPA) for rightsizing. Three months later, the CFO is asking why the cloud bill jumped 40%, and your on-call engineer is paged at 3 AM because orders are timing out. This is the classic autoscaling migration trap: either you over-provisioned to avoid outages, or your scale-down is crippled by conservative thresholds.

Symptoms:

Typical symptoms include: persistent node utilization below 20%, periodic 'Insufficient CPU' pod events, long-running spot node interruptions, HPA metrics that never converge, and cost anomalies that track traffic spikes but never return to baseline.

Diagnosis:

Start by inspecting the cluster autoscaler's view of the world. kubectl get configmap -n kube-system cluster-autoscaler-status -o yaml shows the target and current node counts. For cloud-managed autoscaler, retrieve logs: kubectl logs -n kube-system cluster-autoscaler-xxxxx | grep -E 'scale-down|expander'. Next, review HPA targets: kubectl get hpa -A. If an HPA shows 'Unknown' metrics, the metrics server or custom API is unhealthy. Then, analyze node pressure with kubectl top nodes. A node above 90% CPU is likely causing throttling, but a node below 10% suggests scale-down is not triggering. Finally, map the cost to workload: annotate pods with app and department, then use cloud provider cost explorer or tools like Kubecost to see the offenders.

Commands:

Here's a reproducible diagnosis sequence:

# List all nodes and their resource requests/limits
kubectl describe nodes | grep -E 'Name:|cpu:|memory:' | head -60

# Check HPA status and events
kubectl get hpa -A -o custom-columns='NAME:.metadata.name,NAMESPACE:.metadata.namespace,REFERENCE:.spec.scaleTargetRef.name,TARGET:.status.targetAverage,ACTUAL:.status.currentAverage,MIN/MAX:.spec.minReplicas/.spec.maxReplicas'

# Check cluster autoscaler status
kubectl get configmap cluster-autoscaler-status -n kube-system -o jsonpath='{.data.status}'

# Inspect pending pods
kubectl get pods -o wide | grep Pending

# Verify cluster autoscaler is not throttled
kubectl logs -n kube-system $(kubectl get pods -n kube-system -l app=cluster-autoscaler -o jsonpath='{.items[0].metadata.name}') | tail -50

If you're on EKS, use eksctl get nodegroup; GKE, gcloud container clusters describe; AKS, az aks show. These reveal scalingConfig and autoscale preferences.

Risk Controls:

Autoscaling is powerful but dangerous without controls. Set the following:

• CPU/memory limits on every pod to prevent noisy neighbors.
• Cluster Autoscaler max size: e.g., spec in your managed node pool, usually flagged in provider APIs.
• Pod Disruption Budgets (PDBs) to protect critical services during evictions: kubectl create pdb critical-pdb --selector=app=critical --min-available=2.
• Use VPA in Recommendation mode before applying via kubectl apply -f vpa-offer.yaml.
• For spot instances, use multiple instance types in the node group and a topologySpreadConstraints to survive interruption.
• Set budget alerts: aws budgets or gcloud budgets to alarm at 80% and 100% of forecast.

Rollback:

Every scaling decision should be reversible. Store HPA/VPA/CA manifests in Git. To rollback, kubectl apply -f previous-version.yaml for the HPA. If you need to disable HPA temporarily, kubectl delete hpa your-hpa. For Cluster Autoscaler, to stop scale-down behavior without removing the autoscaler, set the annotation cluster-autoscaler.kubernetes.io/scale-down-disabled=true on specific nodes, or change the autoscaler's command-line flag --scale-down-enabled=false if you manage the deployment. In managed cloud environments, use the cloud console to update the node group's min/max sizes. Never delete nodes directly; let the autoscaler manage it. Always test rollback in staging first.

Verification:

After any change, verify these metrics:

• CPU throttling: kubectl top pods and compare with thresholds.
• Node utilization: should be 30-70% (target 50%).
• Cost: check daily cost anomaly deviation, not just aggregate.
• Autoscaler events: kubectl get events --sort-by=.lastTimestamp | grep autoscaler.
• HPA convergence: kubectl describe hpa shows stable replica counts.

Also, run a load test with hey or k6 to confirm the autoscaler reacts appropriately.

When to Submit an OpsGlobal Ticket:

If your team has spent more than a week tuning these knobs and still sees unstable scaling, or if you're migrating a multi-cluster, multi-cloud estate, it's time to call OpsGlobal. We handle advanced autoscaling policies, custom metrics HPA, cost anomaly detection with ML, and we provide 24/7 observability during major migration windows. Submit a ticket via our portal and we'll inspect your cloud account (with your approval) to build a cost-per-request model and right-size everything.