Hardening Kubernetes for SRE: A Practical Security Guide

Scenario

An SRE team observes unusual pod creation and a spike in outbound network traffic from a Kubernetes cluster. Initial investigation suggests a potential security breach.

Symptoms

• Unauthorized pods running in unexpected namespaces
• Abnormal egress traffic patterns
• Suspicious ClusterRoleBindings in kube-system

Diagnosis

1. Review RBAC:

kubectl get clusterrolebindings --all-namespaces
kubectl describe clusterrolebinding <suspicious-name>

1. Inspect Network Policies:

kubectl get networkpolicies --all-namespaces

Absence of policies means all traffic is allowed by default. 3. Check Pod Security Standards:

kubectl get podsecuritypolicies

If empty, no restrictions apply.

Hardening Commands

Least-Privilege RBAC

• Remove unnecessary ClusterRoleBindings:

kubectl delete clusterrolebinding <suspicious-name>

• Apply a restricted Role and RoleBinding:

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: User
  name: jane
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
EOF

Network Policies

• Default deny all ingress and egress:

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
EOF

• Allow essential egress (e.g., DNS):

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: myapp
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
EOF

Pod Security Standards (via Pod Security Admission)

• Enforce restricted mode on a namespace:

kubectl label ns default pod-security.kubernetes.io/enforce=restricted

⚠️ May disrupt existing pods; test in non-production first.

Risk Controls

• Backup all current RBAC and network policies before changes:

kubectl get clusterrolebindings -o yaml > backup-rbac.yaml

• Roll out changes gradually, starting with a staging namespace.
• Enable audit logging to detect anomalous access.

Rollback

• Restore RBAC from backup:

kubectl apply -f backup-rbac.yaml

• Delete network policies:

kubectl delete networkpolicy default-deny-all

• Remove Pod Security Admission label:

kubectl label ns default pod-security.kubernetes.io/enforce-

Verification

• Test RBAC:

kubectl auth can-i get pods --as=jane

• Confirm network policies are active:

kubectl describe networkpolicy default-deny-all

• Verify Pod Security enforcement:

kubectl run test --image=busybox -- sleep 1

Should be denied.

When to Submit an OpsGlobal Ticket

• If you suspect an active breach and need rapid incident response.
• When you require expert assistance with complex policy design or compliance audits.
• If your team lacks in-depth Kubernetes security expertise.