DevOps Security Hardening for Kubernetes SRE Platforms

Scenario

An SRE team manages multiple Kubernetes clusters. A developer accidentally deletes all pods in a production namespace due to having cluster-admin privileges, causing a service outage. Investigation reveals excessive permissions and lack of least-privilege principle.

Symptoms

• Production pods unexpectedly deleted
• Audit logs show kubectl delete pods --all executed by the user
• User has cluster-admin ClusterRoleBinding

Diagnosis

1. Check user bindings: kubectl get clusterrolebinding <user-binding> -o yaml
2. List user permissions: kubectl auth can-i --list --as=<user>
3. Review API server logs: kubectl logs -n kube-system <apiserver-pod> --tail=100 | grep <user>

Commands (Hardening)

1. RBAC Least Privilege

# Create a read-only cluster role
kubectl create clusterrole readonly --verb=get,list,watch --resource=pods,services,deployments
# Bind to user at namespace level
kubectl create rolebinding readonly-binding --clusterrole=readonly --user=<user> --namespace=<namespace>
# Remove old cluster-admin binding
kubectl delete clusterrolebinding <old-binding>

2. Pod Security Standards

# Enable PodSecurity admission controller (in kube-apiserver args)
--enable-admission-plugins=PodSecurity
# Apply baseline policy to namespace
kubectl label ns <namespace> pod-security.kubernetes.io/enforce=baseline

3. Network Policies

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

4. Secrets Management with External Secrets Operator (ESO)

# Install ESO
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace
# Create a SecretStore (e.g., Vault)
kubectl apply -f secretstore.yaml
# Create an ExternalSecret
kubectl apply -f externalsecret.yaml

Risk Controls

• Backup existing RBAC: kubectl get clusterrolebinding -o yaml > backup-rbac.yaml
• Test changes with a canary user
• Enable audit logging (--audit-log-path=/var/log/kube-apiserver-audit.log)

Rollback

# Restore RBAC bindings
kubectl apply -f backup-rbac.yaml
# Remove Pod Security label
kubectl label ns <namespace> pod-security.kubernetes.io/enforce-
# Delete network policy
kubectl delete networkpolicy deny-all

Verification

# Verify user cannot delete pods
kubectl auth can-i delete pod --as=<user> -n <namespace>
# Check secret sync status
kubectl get externalsecret -n <namespace>
# Test network isolation
kubectl exec -it <pod> -- curl <other-pod-ip>  # Should timeout

When to Submit an OpsGlobal Ticket

• Need to audit and refactor cluster-wide RBAC permissions
• Integration with external secret stores (Vault, AWS Secrets Manager, etc.)
• Incident response requiring forensic analysis and remediation
• Meeting compliance requirements (PCI-DSS, SOC2) for network and pod security