Deep Dive into Cloud Capacity Autoscaling and Cost Operations

Scenario

An e-commerce platform runs Kubernetes on AWS EKS with Cluster Autoscaler (CA) and Horizontal Pod Autoscaler (HPA). They experience unexpected cost spikes and performance degradation: high latency during peak hours, and node utilization below 30% during off-peak.

Symptoms

• Monthly cloud bill increases by >25%
• Node CPU average utilization below 20% during off-peak
• HPA triggers frequently but new Pods are slow to serve requests
• Cluster Autoscaler logs show "scale-up blocked" errors

Diagnosis

1. Check HPA configuration: kubectl get hpa -n <namespace> to review target metrics. Verify resource requests and limits are set correctly.
2. Analyze CA logs: kubectl logs -n kube-system deployment/cluster-autoscaler to find reasons for scaling failures (e.g., node group max limits, AWS resource quotas).
3. Inspect node utilization: kubectl top nodes and kubectl describe nodes for resource allocation and idle capacity.
4. Use AWS Cost Explorer with tag filters to identify wasted spend on idle nodes.

Commands

# Check HPA status
kubectl get hpa --all-namespaces

# View Cluster Autoscaler logs
export CA_POD=$(kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-cluster-autoscaler -o jsonpath='{.items[0].metadata.name}')
kubectl logs -n kube-system $CA_POD --tail=50

# Adjust node group min/max (via AWS CLI)
aws autoscaling update-auto-scaling-group --auto-scaling-group-name <ASG_NAME> --min-size 2 --max-size 10

# Set pod resource requests & limits (example deployment)
kubectl set resources deployment <app> -n <ns> --requests=cpu=500m,memory=512Mi --limits=cpu=1,memory=1Gi

Risk Controls

• Before changing node group sizes, create a PodDisruptionBudget for critical services: kubectl create pdb <name> --selector=app=<app> --min-available=2
• Configure CA scale-down thresholds (e.g., utilization <50% for 10 minutes) via CA startup arguments: add --scale-down-utilization-threshold=0.5 --scale-down-delay-after-add=10m
• Use node affinity and anti-affinity to prevent disruptive workloads from consuming burst capacity

Rollback

• Restore original node group sizes: aws autoscaling update-auto-scaling-group --auto-scaling-group-name <ASG_NAME> --min-size <original> --max-size <original>
• Rollback HPA resource changes: kubectl rollout undo deployment/<deployment-name> -n <namespace>
• Revert CA parameters by editing the deployment: kubectl edit deployment cluster-autoscaler -n kube-system and restore previous arguments

Verification

• Monitor cost trends: Use AWS Cost Explorer or Kubecost to observe daily cost changes over the next week.
• Validate scaling response: Run a load test (e.g., with k6) to stress the system and confirm HPA and CA scale without delays.
• Check node utilization: kubectl top nodes and kubectl describe nodes to ensure off-peak utilization rises above 40%.

When to Submit an OpsGlobal Ticket

• Scaling delays persist even after adjustments (no new nodes within 5 minutes)
• CA logs show "Failed to find node group" or AWS API throttling errors
• Need assistance configuring custom metrics HPA (e.g., based on queue length)
• Frequent Spot instance interruptions affecting workload stability