Hardening Kubernetes Clusters for SRE: A Practical Guide to Security Controls

Scenario

An SRE team manages multiple Kubernetes clusters for production workloads. A security audit reveals critical vulnerabilities: over-permissive RBAC (e.g., service accounts bound to cluster-admin), missing Pod Security Standards, absent Network Policies, weak secrets management (etcd unencrypted), and no container image scanning. These flaws could lead to unauthorized access, lateral movement, and data breaches.

Symptoms

• Security scan reports highlight high-risk RBAC bindings.
• Developers report that certain pods can access resources they shouldn't.
• Compliance team flags PCI-DSS or SOC2 gaps.
• Monitoring catches unusual API calls from namespaces with low trust.

Diagnosis

Run the following commands to assess the current state: 1. Check RBAC bindings: bash kubectl get clusterrolebindings,rolebindings --all-namespaces -o wide kubectl auth can-i --list --as=system:serviceaccount:<namespace>:<sa-name> 2. Check Pod Security Admission (PSA) or Pod Security Policies (PSP, deprecated): bash kubectl get psp kubectl get ns <namespace> -o yaml | grep pod-security 3. Review Network Policies: bash kubectl get networkpolicies --all-namespaces 4. Verify secrets encryption: bash kubectl get secrets --all-namespaces -o json | jq '.items[] | select(.metadata.annotations."kubernetes.io/encrypted" != "true")' 5. Run kube-bench benchmark: bash kube-bench run --targets master,node --version 1.24

Commands & Risk Controls

1. Least Privilege RBAC

• Create specific roles and bindings: bash kubectl create role pod-reader --verb=get,list,watch --resource=pods kubectl create rolebinding pod-reader-binding --role=pod-reader --serviceaccount=default:svc-account --namespace=default
• Use tools like rakkess to audit excessive permissions.

2. Implement Pod Security Admission

• Label namespaces with enforcement level: bash kubectl label ns default pod-security.kubernetes.io/enforce=restricted
• Use audit mode first: audit or warn to avoid breaking existing workloads.

3. Network Policies

• Create a default deny-all policy: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all spec: podSelector: {} policyTypes:
  • Ingress
  • Egress ```
• Gradually add allow rules.

4. Secrets Encryption

• Enable encryption at rest for etcd: configure encryption-provider-config.yaml and restart kube-apiserver.
• Use external KMS like AWS KMS or HashiCorp Vault.

5. Container Image Scanning

• Integrate Trivy or Clair in CI/CD: bash trivy image --severity CRITICAL myregistry/myapp:latest
• Block deployments with critical vulnerabilities.

Rollback

• For RBAC: keep backup yaml files of old RBAC objects; restore via kubectl apply -f backup.yaml.
• For PSA: if a namespace policy breaks pods, remove label: kubectl label ns default pod-security.kubernetes.io/enforce-.
• For Network Policies: delete added policies: kubectl delete networkpolicies default-deny-all.
• For Encryption: reverting encryption config requires ensuring all secrets are decrypted first; data corruption risk. Proceed with caution.

Verification

• Re-run diagnosis commands to confirm RBAC tightened, PSA enforced, Network Policies active, secrets encrypted.
• Attempt to simulate unauthorized access: bash kubectl auth can-i create pods --as=system:serviceaccount:default:svc-account Should return "no".
• Deploy a test pod and verify network isolation: kubectl exec test-pod -- curl http://other-pod-svc should time out.
• Check encryption: bash kubectl get secrets mysecret -o yaml | grep "encryption:"

When to Submit an OpsGlobal Ticket

• When you need to enforce unified security policies across multiple clusters; OpsGlobal's team can design and automate deployment.
• If existing workloads break due to security changes and your team lacks expertise for urgent fixes.
• When you require compliance audit reports (e.g., SOC2, HIPAA) – OpsGlobal provides professional auditing and remediation support.
• If your cluster size is large (>50 nodes) and operations risk is high; external security expert assistance is advisable.