Mastering Cloud Capacity Autoscaling: Cost-Efficient Kubernetes Operations

Introduction

Migrating to the cloud often brings a promise of elasticity—pay only for what you use. Yet, many organizations find their cloud bills skyrocketing after migration to Kubernetes. Why? Because autoscaling is misconfigured, or not aligned with cost governance. This post walks through a real-world scenario of cloud capacity autoscaling, from symptoms to resolution, using Kubernetes-native tools and cloud provider integrations.

Scenario

You're running a large e-commerce platform on AWS using Amazon EKS. During a recent migration from virtual machines, you enabled the Kubernetes Horizontal Pod Autoscaler (HPA) and the Cluster Autoscaler. Traffic spikes during flash sales, but also has quiet periods. After two months, your cloud bill is 40% higher than the previous static infrastructure, even though average CPU utilization across the cluster is below 15%. You suspect over-provisioning and inefficient scaling decisions.

Symptoms

1. High cloud spend: Monthly AWS bill shows high EC2 costs, especially for instances that are often idle.
2. Low resource utilization: kubectl top nodes shows most nodes under 20% CPU and memory usage most of the time.
3. Scaling thrashing: Cluster Autoscaler frequently adds and removes nodes, causing EC2 billing by the hour to be wasteful.
4. Slow response to load: During small traffic bursts, the HPA waits too long to scale pods, causing latency spikes.
5. Spot instance interruptions: If you're using spot instances, frequent replacements interfere with batch jobs.

Diagnosis

Start by inspecting the current autoscaling setup:

# Check node resource usage
kubectl top nodes

# List all HPA configurations
kubectl get hpa --all-namespaces

# Get details of a specific HPA
kubectl describe hpa -n <namespace> <hpa-name>

# Check Cluster Autoscaler status (if deployed as a pod)
kubectl get pods -n kube-system -l app=cluster-autoscaler
kubectl logs -n kube-system deployment/cluster-autoscaler --tail=50

In our scenario, we discover the following:

• HPA is set to scale based on CPU at a target of 70%, but the application's CPU usage is bursty and not correlated with actual load (e.g., batch processing).
• The HPA has a minReplicas of 10 and maxReplicas of 50, but even at the minimum, the total cluster capacity is massive, leading to low average utilization.
• Cluster Autoscaler is configured to expand from 3 to 20 nodes, but it uses a scale-down-utilization-threshold of 0.5 (50%), meaning it will not scale down nodes until utilization is below 50% for a sustained period. This prevents timely scale-down.

Also, review CloudWatch metrics to correlate scaling events with cost:

# Get a list of EC2 instances in the EKS cluster (using AWS CLI)
aws ec2 describe-instances --filters "Name=tag:eks:cluster-name,Values=<cluster-name>" --query "Reservations[*].Instances[*].{ID:InstanceId,Type:InstanceType,LaunchTime:LaunchTime}" --output table

Check scaling history in AWS Auto Scaling groups:

aws autoscaling describe-scaling-activities --auto-scaling-group-name <asg-name> --max-items 10

This will show if nodes are being added and removed too frequently.

Commands and Solutions

Based on the diagnosis, we implement a multi-layered fix.

1. Right-Size Your HPA

Move from CPU-only to custom or external metrics that align with business load. For an e-commerce app, consider scaling on request per second (RPS) or queue length. Use the Kubernetes Metrics Server and Prometheus Adapter.

Example: Install Prometheus Adapter and define an HPA based on RPS:

# Suppose you have an external metric for RPS from your ingress controller
kubectl create hpa my-app --from=custom-metric:nginx_ingress_controller_requests_per_second --target=1000 --min=5 --max=20

Or edit existing HPA:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app
spec:
  minReplicas: 5
  maxReplicas: 20
  metrics:
  - type: External
    external:
      metric:
        name: nginx_ingress_controller_requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"

2. Tune Cluster Autoscaler

Update the Cluster Autoscaler deployment to allow faster scale-down and to respect pod disruption budgets.

In AWS, if you're using the CA on EKS, edit the deployment:

kubectl edit deployment/cluster-autoscaler -n kube-system

Key flags:

• --scale-down-utilization-threshold=0.35 – Scale down nodes only if utilization is below 35%.
• --scale-down-delay-after-add=10m – Wait 10 minutes after a node is added before considering it for removal.
• --max-nodes-total=50 – Safeguard against runaway expansion.

3. Implement Vertical Pod Autoscaler (VPA) for Early-Stage Tuning

Before you have stable autoscaling based on load, use VPA in recommendation mode to right-size pod requests and limits. This reduces waste and improves bin-packing.

kubectl apply -f - <<EOF
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Auto"
  recommenders:
  - name: VPAReccomender
EOF

4. Use Spot Instances for Stateless Workloads

For cost savings, configure a separate node group with Spot Instances. Use nodeSelectors and taints to schedule stateless workloads there. Ensure you have a cluster autoscaler that can diversify instance types.

5. Set Budget Alerts

Use AWS Budgets to monitor spend. Create a budget with an alert threshold at 80% of the expected amount.

Risk Controls

• Pod Disruption Budgets (PDBs): Ensure that scaling down does not disrupt critical services. Define PDBs for your workloads.
• Readiness Gates: Delay pod deletion until the pod is ready to receive traffic.
• Graceful Shutdown: Implement preStop hooks to handle draining.
• Gradual Rollout: Test autoscaling changes in a staging environment first.
• Limit CPU and Memory: Set resource limits to prevent a container from consuming all node resources.

Rollback

If the new configuration causes issues, you can quickly revert to the previous state.

• For HPA changes, use kubectl apply -f with the previous YAML file, or edit back in place.
• For Cluster Autoscaler, revert the deployment flags.
• For VPA, remove the VPA object and update deployments to the previous requests/limits.

Make sure to keep a backup of your manifests in Git.

Verification

After implementing the changes, monitor over a week:

• Check if cloud costs have decreased. Use AWS Cost Explorer to compare before/after.
• Verify cluster utilization: kubectl top nodes should show higher average utilization (40-60%).
• Look at HPA events and Cluster Autoscaler logs to ensure no thrashing.
• Confirm that application latency and error rates remain stable.

Commands:

kubectl get hpa --all-namespaces
kubectl get vpa --all-namespaces
kubectl logs -n kube-system deployment/cluster-autoscaler --tail=50
aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name CPUUtilization --dimensions Name=InstanceId,Value=<instance-id> --start-time <date> --end-time <date> --period 86400 --statistics Average

When to Submit an OpsGlobal Ticket

Getting autoscaling wrong can cost you thousands monthly. If you're facing any of the following, it's time to bring in OpsGlobal's SRE experts:

• Complex, multi-workload autoscaling with interdependent services.
• Custom metrics integration (Prometheus, CloudWatch, Datadog) and autoscaling policies.
• Cost governance and FinOps alignment across multiple cloud accounts.
• 24/7 monitoring and proactive incident response for scaling-related failures.
• You need a second pair of eyes on your production cluster.

OpsGlobal can audit your current setup, implement best practices, and monitor your infrastructure so you can focus on building product.

Conclusion

Autoscaling is not a "set and forget" operation. It requires continuous tuning and alignment with cost operations. By right-sizing HPA, tuning cluster autoscaler, using VPA, and implementing budget alerts, you can achieve both elasticity and cost efficiency. Remember to always have a rollback plan and involve experts when the complexity exceeds your team's bandwidth.