Hardening Kubernetes Clusters for SRE Teams: A Practical Security Guide

Scenario

A company’s Kubernetes cluster was compromised due to overly permissive RBAC and missing Pod Security Policies. Attackers used a high-privileged ServiceAccount to deploy a malicious pod that gained node-level root access.

Symptoms

• Unknown pods appearing in the cluster with no clear origin.
• kubectl get pods shows unusual containers (e.g., crypto-miners).
• Audit logs show excessive create pod events from an unexpected user.
• Node CPU/memory usage spikes unexpectedly.

Diagnosis

1. List all RBAC bindings: bash kubectl get rolebindings,clusterrolebindings -A -o wide Identify bindings granting cluster-admin to suspicious subjects.
2. Inspect audit logs (if enabled): bash kubectl logs -n kube-system apiserver | grep -i “user=<suspicious>”
3. Check existing Pod Security Policies (or Pod Security Admission): bash kubectl get psp # if using deprecated PSP
4. Review NetworkPolicies: bash kubectl get networkpolicies -A Typically none are defined, allowing unrestricted traffic.

Commands & Remediation

Step 1: Least Privilege RBAC

• Remove dangerous bindings: bash kubectl delete clusterrolebinding <binding-name>
• Create fine-grained roles: bash kubectl create role pod-reader --verb=get,list,watch --resource=pods kubectl create rolebinding dev-pod-reader --role=pod-reader --user=dev-user

Step 2: Enforce Pod Security Standards

• Label namespaces to enforce restricted profile (K8s v1.23+): bash kubectl label --overwrite ns <namespace> pod-security.kubernetes.io/enforce=restricted

Step 3: Network Policies

• Create a default deny ingress policy: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-ingress spec: podSelector: {} policyTypes:
  • Ingress ```

Step 4: Secrets Management

• Stop storing secrets in ConfigMaps. Use external secret stores (e.g., HashiCorp Vault) or Sealed Secrets.

Risk Controls

• Test all changes on a non-production cluster first or use a clone.
• Use --dry-run=client when applying RBAC changes.
• Keep backup YAML files for every modified resource.

Rollback

• If new policies break applications: bash kubectl apply -f backup-rbac.yaml kubectl label ns <namespace> pod-security.kubernetes.io/enforce- # remove label

Verification

• Test user permissions: bash kubectl auth can-i create pods --as=dev-user # Should return 'no'
• Attempt to run a privileged pod (should be rejected): bash kubectl run test-pod --image=nginx --privileged
• Review audit logs to confirm no unauthorized activity.

When to Submit an OpsGlobal Ticket

• The cluster is actively compromised and needs immediate forensics and recovery.
• Your team lacks expertise in security hardening and requires expert guidance.
• Advanced security controls (e.g., runtime security, CIS benchmarks) are needed.
• You suspect insider threats or long-term malicious persistence.

OpsGlobal’s SRE experts can help restore quickly, deploy security tooling, and provide ongoing monitoring.