DevOps Security Hardening: RBAC and Pod Security for Ops/SRE Platforms

Scenario

An SRE team manages multiple Kubernetes clusters with default RBAC that allows developers to create privileged Pods. Security audit reveals a malicious Pod mounted host directories to gain node-level access.

Symptoms

• Unknown Pods appear with host volume mounts.
• Audit logs show non-admin users using privileged: true or hostPID: true.
• Security scanners report missing Pod security policies.

Diagnosis

1. Check RBAC bindings: kubectl get clusterrolebindings -o wide for overly broad cluster-admin bindings.
2. Inspect PodSecurityPolicy (PSP) or Pod Security Admission (PSA) status: kubectl get psp or kubectl get admissionregistration.k8s.io/v1/mutatingwebhookconfigurations.
3. Audit Pod specs: kubectl get pods --all-namespaces -o jsonpath="{range .items[*]}{.metadata.name}{'\t'}{.spec.containers[*].securityContext.privileged}{'\n'}{end}".

Hardening Commands

1. Apply Pod Security Standards (PSA): kubectl label ns default pod-security.kubernetes.io/enforce=restricted.
2. Create minimal RBAC: Bind developers to a Role with only get, list permissions, avoid cluster-admin.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: developer-restricted
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

1. Remove risky ClusterRoleBinding: kubectl delete clusterrolebinding <name>.

Risk Controls

• Test PSA labels in non-production first to avoid impacting running workloads.
• Apply in audit mode initially, then switch to enforce after reviewing violations.
• Maintain break-glass admin accounts for emergency access.

Rollback

• Remove PSA label: kubectl label ns default pod-security.kubernetes.io/enforce-.
• Restore RBAC: Reapply previous RoleBinding backups.

Verification

• Check PSA rejects privileged Pods: kubectl run test --image=nginx --privileged should error.
• Confirm non-admin cannot list cluster secrets: kubectl get secrets --as=developer.

When to Submit an OpsGlobal Ticket

• Cluster scales beyond 20 nodes requiring unified policy management.
• Need custom admission controllers or dynamic security policies.
• Compliance mandates (e.g., PCI-DSS) require professional audit reports.