Building Robust CI/CD Guardrails for Kubernetes Deployments

Scenario

Your team is automating the deployment of a critical microservice to a Kubernetes cluster. The CI/CD pipeline passes all tests, but after deployment the service is unresponsive. You suspect resource limits or probe configuration issues.

Symptoms

• Pods show CrashLoopBackOff or Running but fail readiness probes.
• Logs indicate OOMKilled or insufficient resources.
• New version does not receive traffic; old version still serves.

Diagnosis

Check with these commands:

# Inspect pod status and events
kubectl describe pod <pod-name> -n <namespace>

# View container logs
kubectl logs <pod-name> -n <namespace> --previous

# Check resource usage
kubectl top pod <pod-name> -n <namespace>

# Verify readiness probe configuration
kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A10 readinessProbe

Commands Example

# Suppose pod restarts due to OOMKilled
kubectl get events --field-selector involvedObject.name=<pod-name> -n <namespace>
# Output shows: OOMKilled
# Adjust Deployment resource limits
kubectl set resources deployment/<deployment-name> -n <namespace> --limits=cpu=500m,memory=512Mi --requests=cpu=250m,memory=256Mi

Risk Controls

• Add resource limit checks in CI pipeline: use kubectl dry-run and custom validation.
• Enforce readiness and liveness probes.
• Implement progressive delivery (e.g., rolling updates, canary).
• Use Pod Security Policies or OPA Gatekeeper.

Rollback

If deployment fails, roll back immediately:

kubectl rollout undo deployment/<deployment-name> -n <namespace>
# Or to a specific revision
kubectl rollout undo deployment/<deployment-name> -n <namespace> --to-revision=2

Verification

After rollback, verify:

kubectl rollout status deployment/<deployment-name> -n <namespace>
kubectl get pods -n <namespace> | grep <deployment-name>
# Check all pods are healthy

When to Submit an OpsGlobal Ticket

• When you need expert design of complete CI/CD guardrails strategy.
• When the failure involves platform-level issues (Ingress, certificates, cluster exhaustion).
• When you require 24/7 monitoring and alerting configuration support.