Hardening DevOps/SRE Platforms: A Practical Security Playbook for Kubernetes Operations

Scenario

A company's SRE team manages multiple Kubernetes clusters for production workloads. During an internal security audit, they discovered that RBAC permissions were excessively broad—several service accounts had cluster-admin privileges—and some secrets were stored as plaintext in Git. Additionally, no NetworkPolicies were configured, allowing any pod to communicate with any other. The team was too busy with daily operations to address these issues until a near-miss container breakout incident occurred.

Symptoms

The security audit reported the following issues:

• Multiple service accounts bound to the cluster-admin role, far beyond actual needs.
• Developers' kubeconfig files contained long-lived tokens and were stored unencrypted.
• Some Deployment environment variables contained database passwords in plaintext.
• No NetworkPolicies existed in the cluster, so all pods could communicate freely.
• Containers ran as root without resource limits.

These symptoms can lead to severe risks such as data leakage, privilege escalation, and lateral movement.

Diagnosis

To accurately identify security configuration issues, systematically inspect various aspects of the Kubernetes cluster. Auditing RBAC, secret storage, pod security contexts, and network policies helps locate potential vulnerabilities.

Commands

The following commands help diagnose the cluster's security posture (adjust to your environment):

# Check current user privileges
kubectl auth whoami

# List all ClusterRoleBindings and RoleBindings
kubectl get clusterrolebindings -o yaml
kubectl get rolebindings -A -o yaml

# Enumerate all service accounts
kubectl get serviceaccounts -A -o json | jq -r '.items[] | .metadata.namespace + "/" + .metadata.name'

# List all Secrets (be alert for plaintext storage)
kubectl get secrets -A

# Inspect environment variables of Deployments for confidential data
kubectl get deployments -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: {.spec.template.spec.containers[*].env[*].name}{"\n"}{end}'

# List NetworkPolicies
kubectl get networkpolicies -A

# Check whether pods run as root
kubectl get pods -A -o jsonpath='{.items[*].spec.containers[*].securityContext.runAsUser}'

Risk Controls

Based on the diagnosis, implement the following hardening measures:

1. Enforce Least-Privilege RBAC

Create fine-grained Roles and RoleBindings that grant only necessary permissions. For example, define a role that can only manage Deployments in a specific namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: prod
  name: app-deployer
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]

Never grant cluster-admin to non-admin users. Regularly audit and remove redundant bindings.

2. Securely Store and Manage Secrets

Avoid plaintext secrets in Git. Use Kubernetes External Secrets or Sealed Secrets to encrypt secrets at rest. For example, Sealed Secrets allows committing encrypted manifests. Additionally, enable etcd encryption:

kubectl edit apiserver --allow-missing-template-driver

Add an EncryptionConfiguration to the API server:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aesgcm:
      keys:
      - name: key1
        secret: <base64-encoded-32-byte-key>
  - identity: {}

3. Configure Network Policies

Default-deny all ingress/egress traffic and allow only required communication. The following example allows pods from namespace frontend to access port 80 of pods in namespace backend:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: backend
spec:
  podSelector: {}
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend
    ports:
    - port: 80

4. Harden Pod Security Contexts

Apply Pod Security Standards (PSS) restricted policy or use Pod Security Admission. Ensure containers run as non-root, with a read-only root filesystem, and drop all capabilities:

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  readOnlyRootFilesystem: true
  capabilities:
    drop:
    - ALL

5. Use Container Image Scanning and Signing

Integrate image scanning tools (e.g., Trivy) into your CI/CD pipeline to prevent deploying vulnerable images. Sign images with cosign to enhance supply chain trust.

Rollback

Security hardening may impact business operations, so always prepare a rollback plan.

• For RBAC changes, save the original ClusterRoleBinding YAML and use kubectl apply to restore if permissions become insufficient.
• Enabling NetworkPolicies can cause service interruptions. Apply them one by one and verify. If issues arise, delete a policy with kubectl delete networkpolicy <name>.
• Before changing secret storage, back up existing secrets and test decryption.
• Changing pod security context requires recreating pods; perform a canary deployment before full rollout.

Verification

After hardening, verify the effectiveness of the controls:

1. RBAC verification: Try to perform unauthorized actions with a non-admin account; they should be denied.
2. Secret encryption verification: Check that secrets are stored as ciphertext in etcd.
3. Network policy verification: Attempt to access a service from a non-allowed pod; it should time out.
4. Pod security verification: Try to run a pod as root; it should be rejected.
5. Vulnerability scan: Re-run Trivy to confirm the fix rate.

When to Submit an OpsGlobal Ticket

If your team lacks Kubernetes security expertise, or the hardening effort spans many legacy clusters with strict uptime requirements, consider outsourcing to OpsGlobal. Our on-demand DevOps support team can conduct a comprehensive audit and hardening swiftly, with 24/7 monitoring. When submitting a ticket, provide details such as cluster size, security audit reports, access credentials (e.g., kubeconfig), and business criticality. OpsGlobal will help you design and execute a hardening plan while ensuring business continuity.