Hardening Kubernetes Clusters for SRE Operations: A Practical Guide

Scenario

A fast-growing startup runs its production workloads on a Kubernetes cluster with default RBAC and permissive network policies. The security team detects unusual API calls from an external IP in audit logs, suggesting unauthorized access.

Symptoms

• Audit logs show kubectl get secrets from unexpected users.
• Pods can reach resources in other namespaces.
• Service accounts have excessive privileges, e.g., default default service account bound to cluster-admin.

Diagnosis

1. Review RBAC bindings: kubectl get clusterrolebindings -o wide and kubectl get rolebindings --all-namespaces -o wide.
2. Test permissions: kubectl auth can-i --list --as=system:serviceaccount:default:default.
3. Check network policies: kubectl get networkpolicies --all-namespaces.
4. Examine Pod Security Policies: kubectl get psp.

Hardening Commands

# Remove dangerous cluster-admin binding
kubectl delete clusterrolebinding default-admin

# Create minimal service account
kubectl create serviceaccount my-app -n production
kubectl create role my-app-role --verb=get,list,watch --resource=pods,services -n production
kubectl create rolebinding my-app-binding --role=my-app-role --serviceaccount=production:my-app -n production

# Apply default deny network policy
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
EOF

# Enable audit logging (if not already)
audit-policy.yaml:
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse

Risk Controls

• Backup existing RBAC before changes: kubectl get clusterrolebindings -o yaml > clusterrolebindings-backup.yaml
• Test network policies in non-production clusters first.
• Tighten permissions incrementally to avoid breaking services.

Rollback Plan

If hardening causes legitimate requests to fail:

kubectl apply -f clusterrolebindings-backup.yaml
kubectl delete networkpolicy default-deny

Re-apply permissive policies and note affected services.

Verification

• Re-run diagnosis commands; sensitive actions should be denied.
• kubectl auth can-i list secrets --as=system:serviceaccount:default:default should return no.
• Check audit logs for denied unauthorized calls.

When to Submit a Ticket

• Internal team lacks RBAC best practices expertise for designing a least-privilege model.
• Cluster has been compromised, requiring forensic analysis and incident response.
• Need advanced security controls (e.g., OPA/Gatekeeper, kube-bench scanning) but lack experience.
• Contact OpsGlobal at support@opsglobal.com for 24/7 remote SRE assistance.