Cloud Capacity Autoscaling and Cost Operations: A Practical Guide

Scenario

A company is migrating its e-commerce platform to the cloud. Traffic varies greatly with peaks during promotions. Initially, the team manually scaled nodes and pods, resulting in low resource utilization (average 30%) and high costs. The goal is to optimize costs through autoscaling while maintaining performance.

Symptoms

• Cloud bill increases 20%+ month-over-month, but median CPU/memory utilization is below 40%.
• During peak hours, pod startup delays and request timeouts (5xx error rate >2%).
• Node count stays constant during low load periods.

Diagnosis

1. Check pod resource requests/limits: kubectl describe pods -n production – some containers have excessive requests.
2. Inspect HPA status: kubectl get hpa -n production – HPA not configured or thresholds inappropriate.
3. Analyze node utilization: kubectl top nodes plus cloud monitoring (e.g., AWS CloudWatch).
4. Examine Cluster Autoscaler logs: kubectl logs -n kube-system cluster-autoscaler-xxx for scaling events.

Commands

Configure HPA (CPU-based)

kubectl autoscale deployment my-app --cpu-percent=60 --min=3 --max=20 -n production

Enable VPA (set min/max limits)

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: "*"
      minAllowed:
        cpu: 100m
        memory: 100Mi
      maxAllowed:
        cpu: 1
        memory: 1Gi

Configure Cluster Autoscaler (AWS example)

kubectl edit deployment/cluster-autoscaler -n kube-system
# Add --nodes=1:10:eks-worker-group-1 argument

Risk Controls

• Set HPA min/max replicas to avoid aggressive scaling.
• Use PodDisruptionBudget for critical services:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-app

• Enable cloud budget alerts (e.g., GCP Budget Alerts).
• Roll out in stages: test in staging first, then canary in production.

Rollback

If issues arise (e.g., cost spike due to over-scaling or instability): 1. Stop Cluster Autoscaler: kubectl scale deployment/cluster-autoscaler --replicas=0 -n kube-system 2. Revert HPA: kubectl delete hpa my-app -n production or adjust thresholds. 3. If VPA causes restarts, delete VPA: kubectl delete vpa my-app-vpa -n production 4. Set deployment replicas manually: kubectl scale deployment my-app --replicas=5 -n production

Verification

• Check scaling events: kubectl describe hpa my-app -n production – observe Last Scale Time.
• Cost monitoring: compare weekly cloud bill; utilization should rise above 60%.
• Performance: peak 5xx error rate <1%, P95 latency <200ms.

When to Submit an OpsGlobal Ticket

• Autoscaling still not working or causing severe oscillation after following steps.
• Need optimization for complex scenarios (e.g., custom metrics HPA, multi-scaling strategies).
• Unusual cost growth (e.g., frequent Spot instance interruptions causing migration).
• Node count hitting cloud quota limits; need account adjustments.

OpsGlobal's SRE experts can intervene within 30 minutes to configure production-grade autoscaling and set up cost governance dashboards.