Mastering Docker Container Runtime Troubleshooting: A Practical Guide for SREs

Scenario

A microservice in a Kubernetes cluster enters CrashLoopBackOff. Docker logs show: container_linux.go:367: starting container process caused: process_linux.go:578: executing setns process caused: exit status 1: unexpected EOF. The container fails to start, impacting service availability.

Symptoms

• Container status repeatedly CrashLoopBackOff.
• docker logs output is empty or only shows errors.
• Docker daemon logs on the host show runtime errors.
• System resource usage normal, no OOM.

Diagnosis

1. Check Container Logs

Run docker logs <container-id> to inspect stdout/stderr. If empty, the container fails during initialization.

2. Inspect Container State and Configuration

docker inspect <container-id> --format '{{.State.Status}} {{.State.ExitCode}}'

Non-zero exit codes: 139 (segfault), 137 (OOM), 127 (command not found).

3. Analyze OCI Runtime Error

The error originates from runc (default OCI runtime). Check /var/log/syslog or journalctl -u docker for detailed stack traces.

journalctl -u docker -n 100 --no-pager | grep -i error

Common causes: - cgroups version mismatch (some apps require v1, system uses v2). - Mount point issues (e.g., /proc, /sys not mounted correctly). - Kernel security policies (AppArmor/SELinux blocking operations).

4. Test runc Independently

runc exec <container-id> /bin/sh

If this fails, the runtime itself is problematic.

5. Check cgroups Configuration

cat /proc/self/cgroup
mount | grep cgroup

If using cgroup v2, verify controllers exist under /sys/fs/cgroup.

6. Use strace for System Call Tracing

strace -f -e trace=process docker run --rm alpine echo hello

Identify which syscall fails.

Risk Controls

• Never run diagnostic commands like runc exec or strace directly on production containers; they may alter container state.
• Set resource limits (--memory, --cpu) to prevent resource exhaustion triggering other issues.
• Reproduce the issue in a staging environment first.
• Enable Docker debug mode (dockerd --debug) for verbose logs, but beware of log volume.

Rollback

1. Revert to a previous container image: bash kubectl set image deployment/<name> <container>=<image>:<previous-tag>
2. If the Docker daemon on the Kubernetes node is misbehaving, restart it: bash systemctl restart docker Note: This stops all containers on the node; assess impact.
3. If runc version is suspect, downgrade or reinstall: bash apt-get install --reinstall runc=1.0.0-1

Verification

• After rollback, wait for Pod startup and check status: bash kubectl rollout status deployment/<name> kubectl logs -f <pod-name>
• Hit the health check endpoint (e.g., /healthz).
• Monitor Docker daemon logs for new errors.

When to Submit an OpsGlobal Ticket

• The issue involves kernel modules or cgroups version conflicts requiring kernel upgrade or system config changes.
• OCI runtime needs customization or upgrade affecting multiple services.
• Problem persists across multiple nodes, requiring a full runtime audit.
• Your team lacks permissions to modify host-level settings (e.g., managed Kubernetes nodes).