DevOps Security Hardening for Ops/SRE Platforms: A Practical Kubernetes Deep Dive

Scenario

An Ops team manages a multi-tenant Kubernetes cluster. Recent audit logs show denied RBAC requests and unexpected privileged Pods appearing in production namespaces. Developers complain about inconsistent access controls.

Symptoms

• Non-admin users attempting to list cluster-scoped resources (e.g., Nodes, PVs) and being denied.
• A privileged container appearing in a namespace where it shouldn't exist.
• Network policies not effectively isolating traffic across namespaces.

Diagnosis

1. RBAC Review

# List all ClusterRoleBindings and RoleBindings
kubectl get clusterrolebinding -o wide
kubectl get rolebinding -A -o wide
# Check effective permissions for a specific user or ServiceAccount
export USER="john.doe@example.com"  # replace with actual user
kubectl auth can-i --list --as=$USER --all-namespaces | grep -v "no"

2. Network Policy Audit

# List all network policies
kubectl get networkpolicy --all-namespaces
# Check if a default-deny policy exists (if missing, all traffic is allowed)
NAMESPACE="production"
kubectl describe networkpolicy -n $NAMESPACE

3. Pod Security Context Check

# Find pods with privileged containers or allowPrivilegeEscalation
kubectl get pods --all-namespaces -o jsonpath="{range .items[*]}{.metadata.namespace}{' '}{.metadata.name}{' '}{.spec.containers[*].securityContext.privileged}{'\\n'}{end}" | grep true
# Check PodSecurityAdmission settings
kubectl get pods -n $NAMESPACE -o yaml | grep -A5 "pod-security.kubernetes.io/enforce"

Risk Controls & Hardening Commands

1. Tighten RBAC

# Create least-privilege Role and RoleBinding (example: read-only access to pods in a namespace)
kubectl create role pod-reader --verb=get,list,watch --resource=pods -n $NAMESPACE
kubectl create rolebinding $USER-pod-reader --role=pod-reader --user=$USER -n $NAMESPACE
# Remove overly permissive ClusterRoleBindings
kubectl delete clusterrolebinding overpermissive-binding

2. Enforce Network Policies

# Default deny all ingress and egress traffic, then allow necessary communication
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: $NAMESPACE
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

kubectl apply -f default-deny-all.yaml
# Add allow rules, e.g., frontend pods can talk to backend pods

3. Apply Pod Security Standards

# Enable Pod Security Admission at the restricted level on the namespace
kubectl label ns $NAMESPACE pod-security.kubernetes.io/enforce=restricted
kubectl label ns $NAMESPACE pod-security.kubernetes.io/warn=restricted

Safe Backup & Rollback

Before applying changes, back up current resources:

kubectl get clusterrolebinding -o yaml > clusterrolebinding-backup.yaml
kubectl get rolebinding -A -o yaml > rolebinding-backup.yaml
kubectl get networkpolicy -A -o yaml > networkpolicy-backup.yaml

To rollback:

kubectl apply -f clusterrolebinding-backup.yaml
kubectl apply -f rolebinding-backup.yaml
kubectl apply -f networkpolicy-backup.yaml
kubectl label ns $NAMESPACE pod-security.kubernetes.io/enforce-  # remove label

Verification

Repeat the diagnosis commands to confirm restrictions:

kubectl auth can-i list pods --as=$USER -n $NAMESPACE  # should return yes
kubectl auth can-i list nodes --as=$USER  # should return no
kubectl get pods -n $NAMESPACE -o yaml | grep privileged  # no output

When to Submit an OpsGlobal Ticket

• Your team cannot interpret complex audit log denials.
• You need consistent security policies across multiple clusters.
• You face regulatory compliance requirements (e.g., PCI-DSS) without in-house expertise.
• Hardening breaks an application unexpectedly and you need expert debugging.

OpsGlobal's SRE engineers can remotely diagnose and implement bank-grade security configurations, ensuring your cluster meets industry best practices.