Mastering Cloud Capacity Autoscaling and Cost Operations: A Practical Guide

Scenario

A company migrates its workloads to the cloud. To ensure a smooth transition, the engineering team initially overprovisions resources, causing cloud bills to skyrocket while utilization remains very low. During peak traffic, some services experience performance degradation, throttling, and SLO violations. The team needs an autoscaling and capacity management strategy that maintains performance while controlling costs.

Symptoms

• Monthly cloud bills are abnormally high, but CPU and memory utilization remain below 20% for long periods.
• During business peaks, Pods experience CPU throttling, request timeouts, and even OOMKills.
• The number of replicas is fixed and does not change with traffic.
• Cluster nodes remain static even when load is low.
• Cost reports show compute resource spend dominating, with no clear explanation.

Diagnosis

First, collect data using native Kubernetes tools and cloud platform monitoring.

# Check resource usage of nodes and Pods
kubectl top nodes
kubectl top pods -A

# Check existing HPA configuration and status
kubectl get hpa -A
kubectl describe hpa -n <namespace>

# Check cluster autoscaler status (example for EKS)
kubectl get clusterautoscaler -n kube-system

# For GKE, use gcloud to inspect cluster autoscaling configuration
gcloud container clusters describe <cluster-name> --region <region>

Analyze the data: - If HPA is not configured or misconfigured, replica count will not scale with load. - If Pod resource requests and limits are not set or are set too high/low, scheduling and scaling will be ineffective. - If cluster autoscaler is not enabled, nodes will not scale based on Pod scheduling needs.

Use cloud cost management tools (e.g., AWS Cost Explorer, GCP Cost Management) to analyze costs by service, namespace, or label, identifying the main sources of waste.

Commands

1. Adjust HPA

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 70

Apply the configuration:

kubectl apply -f hpa.yaml

2. Enable Cluster Autoscaler

• EKS: Enable managed-nodegroup at cluster creation, set --min-size and --max-size.
• GKE: gcloud container clusters update <cluster> --enable-autoscaling --min-nodes 1 --max-nodes 10
• AKS: az aks update --resource-group <rg> --name <cluster> --enable-cluster-autoscaler --min-count 1 --max-count 10

3. Set Resource Requests and Limits

Edit the Deployment and configure accurate requests and limits for each container. Use VPA recommendations as a starting point.

Risk Controls

• Always set PodDisruptionBudgets (PDB) for critical workloads to ensure service continuity during rolling updates or scale-down.
• Set scaling boundaries for HPA (e.g., do not set maxReplicas too high) to avoid over-scaling.
• Test all autoscaling configurations in a non-production environment first to confirm expected behavior.
• Set min/max limits for node groups in cluster autoscaler to prevent accidental expansion to extremely high costs.
• When using Vertical Pod Autoscaler (VPA), start in "recommendation" mode before enabling "auto" mode.
• Monitor scaling frequency to avoid "thrashing" (frequent scaling). Adjust the --horizontal-pod-autoscaler-sync-period or metric thresholds if needed.

Rollback

If autoscaling causes issues, you can quickly roll back:

# Delete HPA and restore manual replica count
kubectl delete hpa my-app-hpa
kubectl scale deployment my-app --replicas=5

# Disable cluster autoscaler
# EKS: remove autoscaling config via AWS console or CLI
# GKE: gcloud container clusters update <cluster> --no-enable-autoscaling
# AKS: az aks update --resource-group <rg> --name <cluster> --disable-cluster-autoscaler

Before rolling back, ensure that the manually set replica count can handle the current load to avoid service disruption.

Verification

• Observe HPA status: kubectl describe hpa shows Current Replicas and Desired Replicas, and check scaling events.
• Use Grafana or cloud monitoring to track CPU, memory, P99 latency, and cost changes.
• Compare SLO achievement rate and cloud bills before and after the changes.
• Check if cluster node count changes with Pod demand via kubectl get nodes.

For example, during peak hours, replicas should automatically scale from 2 to 10, and nodes from 3 to 6; during low hours, they should shrink to reduce costs.

When to Submit an OpsGlobal Ticket

We recommend contacting us immediately if:

• Autoscaling thrashes frequently, causing application instability that conventional tuning cannot fix.
• Cloud platform quota limits (e.g., vCPU limits) prevent cluster expansion, requiring quota adjustments or architecture redesign.
• Cost anomalies spike with no explainable source using available tools.
• You need to design complex scaling strategies, such as HPA based on custom metrics (Kafka lag, request queue depth), or mix Spot and On-Demand instances.
• Cluster scaling generates internal errors that require deep investigation.

OpsGlobal's SRE experts can help you design and implement robust capacity management and cost optimization solutions, allowing you to run faster and save more in the cloud.