Ops/SRE Platform Security Hardening: From Pod Security to Runtime Protection

Scenario

An SRE team at a fintech company detects a suspicious container attempting to read /etc/shadow from the host. The container runs in the default namespace without security context.

Symptoms

• Audit logs show repeated failed file read attempts
• Container runs as root
• No NetworkPolicy restricting egress traffic

Diagnosis

# Check Pod security context
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].securityContext}'

# List PodSecurityPolicies or Pod Security Admission configuration
kubectl get psp
kubectl get podsecurityadmissionconfiguration

# Check NetworkPolicies
kubectl get networkpolicy -A

Commands (Hardening Steps)

1. Apply Pod Security Standards (baseline)

kubectl label ns default pod-security.kubernetes.io/enforce=baseline

1. Create least-privilege RBAC

kubectl create clusterrole app-reader --verb=get,list --resource=pods
kubectl create rolebinding app-reader --clusterrole=app-reader --serviceaccount=default:default

1. Implement egress NetworkPolicy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-egress
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Egress

1. Install Falco for runtime security

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco -n falco-system --create-namespace

Risk Controls

• Test on non-critical namespaces first
• Validate NetworkPolicy before applying to production
• Monitor Falco alerts and set up notifications

Rollback

kubectl label ns default pod-security.kubernetes.io/enforce-
kubectl delete clusterrole app-reader
kubectl delete rolebinding app-reader
kubectl delete networkpolicy deny-all-egress
helm delete falco -n falco-system

Verification

# Verify non-root execution
kubectl run test --image=busybox -- sh -c 'id'
# Check Pod security labels
kubectl get ns default -o yaml | grep pod-security
# Test NetworkPolicy
kubectl run test2 --image=busybox -- sh -c 'wget http://evil.com'
# Review Falco alerts
kubectl logs -n falco-system -l app=falco

When to Submit an OpsGlobal Ticket

• Hardening causes service outage and rollback fails
• Need cross-cluster unified security policy design
• Suspected container escape or zero-day vulnerability
• Compliance audit (e.g., PCI-DSS) requires expert sign-off

Provide cluster kubectl access and existing security logs; OpsGlobal responds within 15 minutes.