DevOps Security Hardening: Production-Grade Practices for Ops/SRE Platforms

Scenario

An SRE team at a fintech firm discovered that a low-privilege pod in a production Kubernetes cluster accessed sensitive Secrets in another namespace via a mounted ServiceAccount token. The pod was intended only for log collection, but the default RBAC configuration allowed cross-namespace access.

Symptoms

• Security audit logs show unexpected kubectl get secrets calls.
• Monitoring alerts: spike in API requests to unintended namespaces.
• Plaintext database password from another service appears in error logs of the log collector.

Diagnosis

1. Check the pod's ServiceAccount: bash kubectl get pod <pod-name> -n <namespace> -o json | jq '.spec.serviceAccountName'
2. View RoleBindings for that ServiceAccount: bash kubectl get rolebinding,clusterrolebinding -n <namespace> | grep <serviceaccount>
3. Default ClusterRole often includes get secrets permissions.

Commands: Security Hardening

1. Create a Least-Privilege ServiceAccount

kubectl create serviceaccount log-collector -n logging

2. Define a Read-Only Log Access Role

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: logging
  name: log-reader
rules:
- apiGroups: [""]
  resources: ["pods/log", "pods"]
  verbs: ["get", "list", "watch"]

3. Bind the Role

kubectl create rolebinding log-reader-binding --role=log-reader --serviceaccount=logging:log-collector -n logging

4. Update Pod to Use New ServiceAccount

Edit the pod spec to set serviceAccountName: log-collector and remove old Secret mounts.

5. Apply NetworkPolicy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-other-namespaces
  namespace: logging
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: logging

6. Enforce Pod Security Standards (Restricted)

kubectl label ns logging pod-security.kubernetes.io/enforce=restricted

Risk Controls

• Test RBAC changes in non-production first.
• Network policies may block legitimate traffic; plan allowlists ahead of time.
• Pod Security Standards might prevent existing pods from starting; adapt workloads individually.

Rollback Plan

kubectl delete rolebinding log-reader-binding -n logging
kubectl delete role log-reader -n logging
kubectl delete networkpolicy deny-other-namespaces -n logging
kubectl label ns logging pod-security.kubernetes.io/enforce- # remove label

Verification

1. Confirm pod cannot access other namespaces: bash kubectl exec -it <pod> -n logging -- curl -k -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://kubernetes.default.svc/api/v1/namespaces/default/secrets Should return 403.
2. Verify log collection still works.

When to Submit an OpsGlobal Ticket

Submit a ticket immediately if: - You have discovered unauthorized access but lack Kubernetes security expertise. - You need a large-scale cluster security audit and policy migration. - You require assistance with Pod Security Standards compatibility assessment or custom policy authoring.

OpsGlobal's SRE experts provide 24/7 remote support to quickly diagnose and execute hardening.