DevOps/SRE Platform Security Hardening: A Practical Guide to Kubernetes Cluster Protection

Scenario

An e-commerce platform suffered a 30-minute production outage when a developer accidentally deleted Deployments in a critical namespace due to overly permissive RBAC. Post-incident review revealed the developer had cluster-admin privileges, and no NetworkPolicies or Pod Security Policies were in place.

Symptoms

• Unauthorized deletion of production resources by non-admin users
• Container escape or resource abuse
• Unexpected network traffic blocks or unauthorized access

Diagnosis

# Check RBAC bindings
kubectl get rolebindings,clusterrolebindings --all-namespaces -o wide

# Find pods violating Pod Security Standards
kubectl get pod --all-namespaces -o json | jq '.items[] | select(.spec.containers[].securityContext.privileged == true or .spec.containers[].securityContext.runAsUser == 0) | {name: .metadata.name, namespace: .metadata.namespace}'

# Verify NetworkPolicy coverage
kubectl get networkpolicies --all-namespaces | grep -v NAMESPACE | awk '{print $1}' | sort | uniq -c

Hardening Steps

1. RBAC Least Privilege

# Create read-only role
kubectl create role read-only --verb=get,list,watch --resource=pods,services --dry-run=client -o yaml > read-only-role.yaml
kubectl apply -f read-only-role.yaml

# Bind to user
kubectl create rolebinding dev-readonly --role=read-only --user=dev-user --namespace=default

2. Implement NetworkPolicy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

3. Enable Pod Security Admission

# Label namespace to enforce baseline
kubectl label ns production pod-security.kubernetes.io/enforce=baseline

Risk Controls

• Before changes, snapshot RBAC and NetworkPolicy: kubectl get clusterrolebindings -o yaml > backup-rbac.yaml
• Test NetworkPolicy with canary deployments
• Enable audit logging: kubectl logs kube-apiserver -n kube-system | grep audit

Rollback

# Restore RBAC bindings
kubectl delete rolebinding dev-readonly -n default
kubectl apply -f backup-rbac.yaml

# Remove NetworkPolicy
kubectl delete networkpolicy default-deny -n production

Verification

# Test permissions
kubectl auth can-i delete pods --as=dev-user -n production

# Test network isolation
kubectl run test-pod --image=busybox --rm -it -- wget -O- http://other-service

When to Submit an OpsGlobal Ticket

• Active attack or data breach
• Security changes causing widespread service disruption
• Need for penetration testing or compliance audit