Hardening Kubernetes for SRE: A Practical Guide to Securing Your Cluster

Scenario

As an SRE, you discover that a Kubernetes worker node has an unsecured kubelet port (10250) exposed to the internal network, allowing unauthenticated access to pods and containers. An attacker could enumerate running workloads or even execute commands.

Symptoms

• Security scans report open kubelet ports.
• Unexpected pod creation in system namespaces.
• API server logs show numerous unauthenticated requests to kubelet endpoints.

Diagnosis

1. Check kubelet startup flags: ps aux | grep kubelet. Look for missing --authentication-token-webhook and --authorization-mode=Webhook.
2. Confirm port exposure: nmap -p 10250 <node-ip>.
3. Attempt direct access: curl -k https://<node-ip>:10250/pods. If it returns pod data, the kubelet is insecure.

Commands & Actions

1. Enable Kubelet Authentication & Authorization

Edit the kubelet config file (e.g., /var/lib/kubelet/config.yaml) and add:

authentication:
  webhook:
    enabled: true
  anonymous:
    enabled: false
authorization:
  mode: Webhook

Restart: systemctl restart kubelet.

2. Network Restriction

Use a Kubernetes NetworkPolicy or cloud firewall to limit kubelet access to only the API server and monitoring tools. Example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: restrict-kubelet
  namespace: kube-system
spec:
  podSelector:
    matchLabels:
      component: kubelet
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          component: kube-apiserver
    ports:
    - port: 10250

3. Enable NodeRestriction Admission Controller

Add --enable-admission-plugins=NodeRestriction to the kube-apiserver arguments to ensure nodes can only modify their own related resources.

4. RBAC for Kubelet API

Create a ClusterRole and ClusterRoleBinding that grants the system:nodes group only necessary permissions (e.g., nodes/proxy).

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:kubelet-api-admin
rules:
- apiGroups: [""]
  resources: ["nodes/proxy"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kubelet-api-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kubelet-api-admin
subjects:
- kind: Group
  name: system:nodes
  apiGroup: rbac.authorization.k8s.io

5. Enforce Pod Security Standards

Enable Pod Security Admission (PSA) by labeling namespaces:

kubectl label ns default pod-security.kubernetes.io/enforce=restricted

For custom policies, consider OPA/Gatekeeper.

Risk Controls

• Test all changes in a staging environment first.
• Ensure the API server has enough memory to handle extra admission webhooks.
• Backup kubelet config files before modification.
• Roll out kubelet changes gradually (e.g., one node at a time).

Rollback

1. Restore the original kubelet config and restart the service.
2. Delete added RBAC resources: kubectl delete clusterrole system:kubelet-api-admin.
3. Remove NetworkPolicy.
4. Remove PSA labels if applied.

Verification

1. Attempt unauthenticated access: curl -k https://<node-ip>:10250/pods should return 401 or 403.
2. Test with a valid ServiceAccount token: kubectl get --raw /api/v1/nodes/<node>/proxy/pods should succeed (requires appropriate RBAC).
3. Review API server logs for denied kubelet requests.

When to Submit an OpsGlobal Ticket

• If after hardening you observe widespread connectivity issues or node unavailability.
• When you need help implementing Pod Security Standards at scale or integrating OPA/Gatekeeper.
• For audit log configuration and alerting rules.
• OpsGlobal’s senior SRE engineers provide 24/7 remote support to keep your production cluster secure and stable.