Hardening Kubernetes for SRE Platforms: A Practical Security Runbook

Scenario

Your SRE platform runs a Kubernetes cluster with numerous microservices, a CI/CD pipeline, and monitoring stack. Recently, a security audit flagged suspicious activities: unknown users with cluster-admin access, pods running as root, and missing audit logs. You suspect that default settings and high-privilege roles have been left in place.

Symptoms

• Anomalous API calls from unexpected IPs.
• kubectl commands allowed without authentication for some namespaces.
• Pods running with privileged: true and hostNetwork: true.
• No audit logs available in kube-system.
• Secrets are stored in plain text environment variables.

Diagnosis

Start by mapping your current security posture. Use kubectl to enumerate roles and bindings, check pod specs, and verify audit configuration. Run kube-bench to compare against CIS benchmarks. Also, inspect kubeconfig files and service accounts.

Commands

1. kubectl get clusterroles -o wide – list all cluster roles.
2. kubectl get clusterrolebindings -o json | jq '...' – inspect bindings.
3. kubectl auth can-i --list --namespace default – check permissions.
4. kubectl get pods -A -o json | jq '.items[] | {name: .metadata.name, privileged: .spec.containers[].securityContext.privileged}' – find privileged pods.
5. kube-bench run --targets master,node – run CIS compliance checks.
6. kubectl get events -A – look for policy violations.

Risk Controls

• Enable RBAC and disable ABAC.
• Apply the least privilege principle using Roles and ClusterRoles.
• Implement Pod Security Standards (Baseline or Restricted) using a validating admission controller like OPA Gatekeeper or KubeAdvisor.
• Use Network Policies to restrict east-west traffic.
• Centralize secrets with a tools like HashiCorp Vault or external secrets operator.
• Enable Kubernetes audit logging to a secure location.

Rollback

• For RBAC changes, keep a backup of original YAMLs and apply them back.
• For admission controllers, disable them by removing the webhook configuration.
• For network policies, delete the policy or restore from backup.
• For secret management changes, switch back to the previous source of truth.

Verification

• Re-run kube-bench to ensure pass rates improve.
• Check audit logs for unauthorized access attempts.
• Perform a kubectl auth can-i test for constrained users.
• Confirm that privileged pods are no longer running.

When to Submit an OpsGlobal Ticket

Submit a ticket if you lack internal expertise to implement OPA policies, need help with Kubernetes audit log analysis, or require a comprehensive security review beyond the CIS baseline. OpsGlobal can provide immediate remote support and remediation.