Cloud Capacity Autoscaling and Cost Operations: A Practical Kubernetes Guide

Scenario

An e-commerce platform experiences traffic spikes during promotions and low load otherwise. Manual scaling caused latency during peaks and resource waste during troughs, with monthly cloud costs exceeding budget by 40%.

Symptoms

• Node CPU utilization consistently above 85% or below 20%
• Pods frequently OOMKilled or pending
• Cloud bill showing abnormal growth in EC2/EKS instance costs

Diagnosis

1. Analyze cluster metrics: Use kubectl top nodes and kubectl top pods to view resource usage.
2. Review Horizontal Pod Autoscaler (HPA) config: Ensure metrics and thresholds are appropriate.
3. Check Cluster Autoscaler (CA) logs: kubectl logs -n kube-system deployment/cluster-autoscaler
4. Cost analysis: Use cloud provider cost tags (e.g., AWS Cost Explorer) grouped by namespace or labels.

Key Commands

Enable HPA

kubectl autoscale deployment frontend --cpu-percent=70 --min=3 --max=20

Configure Cluster Autoscaler

Edit ConfigMap:

kubectl edit configmap -n kube-system cluster-autoscaler-config

Add scale-down-utilization-threshold: 0.5 to lower scale-down threshold.

Cost Optimization Example

Use Spot instances:

kubectl label nodes --all spot=yes
# Then specify node selector in deployments

Risk Controls

• Set Pod Disruption Budget (PDB) for critical services: yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: web-pdb spec: maxUnavailable: 1 selector: matchLabels: app: web
• Implement graceful scale-down: set stabilizationWindowSeconds in HPA
• Enable cost alerts: e.g., AWS Budgets notify when spending exceeds 80%

Rollback Plan

1. Revert HPA: kubectl delete hpa frontend then recreate previous version.
2. Disable Cluster Autoscaler: kubectl scale deployment/cluster-autoscaler --replicas=0 -n kube-system
3. If backed up, reapply previous Deployment and Service manifests.

Verification

• Use kubectl describe hpa frontend to check current metrics and desired replicas.
• Monitor node count: kubectl get nodes.
• Compare cost reports: actual spending should drop without SLO violations.

When to Submit an OpsGlobal Ticket

• Autoscaling causing stability issues (e.g., frequent evictions, PDB blocking scale-down)
• Costs remain anomalous or not reducing as expected
• Large clusters (100+ nodes) needing strategy optimization
• Multi-cloud or hybrid cloud environments

OpsGlobal's SRE team can respond within 15 minutes, offering custom autoscaling strategies and cost audits.