DevOps Security Hardening: A Practical Guide for Kubernetes Platforms

Scenario

An e-commerce platform's Kubernetes cluster (v1.28) experiences unexpected failures: CI pipeline interruptions and Pod deletions. Security audit reveals a non-admin ServiceAccount with cluster-admin privileges and unauthorized API calls. The team must harden security urgently.

Symptoms

• Unexpected Pod restarts or deletions
• Unauthorized API calls in audit logs
• High-risk RBAC bindings
• Exposed sensitive Secrets

Diagnosis

1. Check RBAC permissions: bash kubectl get clusterrolebindings -o wide | grep -v system: kubectl get rolebindings --all-namespaces -o wide
2. Audit logs: Enable auditing and filter suspicious activity: bash kubectl logs -n kube-system kube-apiserver --tail=1000 | grep -E "User|verb=delete|verb=create"
3. Review Pod Security Policies (PSP) or Pod Security Admission (PSA): bash kubectl get psp -A kubectl get podsecurityconfiguration -A
4. Check Secrets usage: bash kubectl get secrets --all-namespaces | grep -E "token|cert|pass"

Commands (Hardening)

1. Remove dangerous ClusterRoleBinding: bash kubectl delete clusterrolebinding malicious-binding
2. Apply least-privilege RBAC: yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: production name: readonly rules: - apiGroups: [""] resources: ["pods", "services"] verbs: ["get", "list", "watch"]
3. Enforce Pod Security Standards (baseline): bash kubectl label ns production pod-security.kubernetes.io/enforce=baseline
4. Harden network policies: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-all namespace: production spec: podSelector: {} policyTypes:
   • Ingress
   • Egress ```
5. Rotate secrets: bash kubectl delete secret my-secret kubectl create secret generic my-secret --from-literal=key=NEW_VALUE

Risk Controls

• Test all changes in non-production first.
• Backup existing RBAC and network policies: bash kubectl get clusterrolebindings -o yaml > backup-clusterrolebindings.yaml kubectl get networkpolicies --all-namespaces -o yaml > backup-networkpolicies.yaml
• Use kubectl auth can-i to verify permissions: bash kubectl auth can-i create deployments --as=system:serviceaccount:production:my-sa

Rollback

• Revert RBAC: bash kubectl apply -f backup-clusterrolebindings.yaml
• Revert network policies: bash kubectl apply -f backup-networkpolicies.yaml
• Remove Pod Security label: bash kubectl label ns production pod-security.kubernetes.io/enforce-

Verification

• Confirm RBAC tightening: bash kubectl auth can-i delete pods --as=system:serviceaccount:production:app-sa
• Test network isolation: bash kubectl exec -n production test-pod -- curl -m 3 http://other-service should timeout.
• Ensure secrets are not exposed: bash kubectl get secrets --all-namespaces -o json | jq '.items[] | select(.type=="Opaque") | .data | keys'

When to Submit an OpsGlobal Ticket

• Persistent attacks requiring immediate containment.
• Emergency P0 incidents (e.g., compromised credentials).
• Compliance audits (SOC2, PCI) needing expert assessment.
• Lack of in-house Kubernetes security expertise.

OpsGlobal provides 24/7 remote SRE support for rapid diagnosis and remediation, ensuring platform security.