Hardening Kubernetes Clusters for SRE: A Practical Security Guide

Scenario

A company runs a multi-tenant Kubernetes cluster where multiple teams deploy workloads. A security audit reveals that some pods can access the host node's filesystem due to privileged containers or hostPath volumes.

Symptoms

• Pods can read sensitive host files like /etc/shadow.
• Security scanners report violations of Pod Security Standards.
• Unexpected network connections or resource access.

Diagnosis

1. Check for privileged containers: bash kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{.metadata.namespace} {.metadata.name} {.spec.containers[*].securityContext.privileged}{"\n"}{end}' An output containing true indicates privileged containers.
2. Check for hostPath volumes: bash kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{.metadata.namespace} {.metadata.name} {.spec.volumes[*].hostPath}{"\n"}{end}'
3. Review RBAC permissions: bash kubectl get rolebindings,clusterrolebindings --all-namespaces -o wide Look for inappropriate bindings, e.g., granting cluster-admin to regular users.

Risk Controls

• Enforce Pod Security Admission with a global restricted policy.
• Use OPA/Gatekeeper or Kyverno to deny privileged containers and hostPath.
• Implement NetworkPolicies to restrict inter-pod traffic.
• Apply least-privilege RBAC, avoid wildcards (*).
• Enable audit logging and review regularly.

Rollback

If policies break applications, roll back via GitOps:

git revert <commit-hash>
kubectl apply -f <previous-config>

Alternatively, temporarily delete the admission controller webhook configuration.

Verification

• Re-run diagnosis commands to confirm no privileged containers.
• Run kube-bench for security benchmarks: bash kubectl run kube-bench --image=aquasec/kube-bench:latest --restart=Never --rm --command -- /kube-bench run --targets=master,node
• Check audit logs for policy violation denials.

When to Submit an OpsGlobal Ticket

• When mass remediation of legacy workloads is needed.
• If a security breach has occurred, requiring rapid forensics and isolation.
• When cluster-wide policy enforcement is beyond the team's expertise.