Mastering Cloud Capacity Autoscaling and Cost Operations: A Practical Guide for SREs

Scenario

Your Kubernetes cluster frequently experiences Pod scheduling failures during business peaks, while idle resources during low traffic lead to excessive cloud bills. You need autoscaling to match load and control costs.

Symptoms

• Pods stuck in Pending due to insufficient node resources
• Node utilization below 30% for hours
• Monthly cloud bill increases >20% without corresponding business growth

Diagnosis

1. Check cluster resources: kubectl top nodes and kubectl top pods
2. Inspect Pod events: kubectl describe pod <pod-name> for Insufficient memory/cpu
3. Analyze costs: In AWS Cost Explorer, filter by service (EC2, EKS) and tags to identify idle resources
4. Check HPA status: kubectl get hpa to see target vs actual metrics

Commands

Configure HPA (Horizontal Pod Autoscaler)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Apply: kubectl apply -f hpa.yaml

Configure Cluster Autoscaler (node autoscaling)

Assuming Cluster Autoscaler is deployed, set min/max nodes:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-status
  namespace: kube-system
data:
  config: |
    {
      "minNodes": 3,
      "maxNodes": 20
    }

Or via AWS CLI to update Auto Scaling Group:

aws autoscaling update-auto-scaling-group --auto-scaling-group-name my-asg --min-size 3 --max-size 20

Cost tagging and reporting

# Label nodes for cost tracking
kubectl label nodes --all cost-center=production
# Use AWS Cost Explorer to group by tag

Risk Controls

• Set HPA cooldown: --horizontal-pod-autoscaler-downscale-stabilization-window=5m
• Set Cluster Autoscaler scale-up delay: --scale-down-delay-after-add=10m
• Use PodDisruptionBudgets for critical services
• Test in staging before production

Rollback

1. Delete HPA: kubectl delete hpa my-app-hpa
2. Restore Deployment replicas: kubectl scale deployment my-app --replicas=3
3. Reset Cluster Autoscaler config: revert ASG min size
4. To fully disable: kubectl scale deployment cluster-autoscaler --replicas=0 -n kube-system

Verification

• Check Pod scheduling: kubectl get pods -o wide all running
• Monitor resource usage: kubectl top nodes showing 40-80% utilization
• Compare cost: review AWS bills for two weeks before and after, expect at least 15% savings
• Load test: kubectl run -it --rm load-generator --image=busybox -- /bin/sh -c "while true; do wget -q -O- http://my-app-service; done" and observe scaling

When to Submit an OpsGlobal Ticket

• Autoscaling fails to trigger despite metrics, logs show metric server issues
• Cluster autoscaler oscillates causing instability
• Need cost optimization advice across multi-region, multi-cluster deployments
• Scaling policy errors cause service disruption