Docker Container Runtime Troubleshooting: A Practical Guide

Scenario

In production, Docker containers may fail to start, exit unexpectedly, suffer performance degradation, or experience network issues. This guide uses a typical scenario: a microservice container in a Kubernetes cluster repeatedly enters CrashLoopBackOff with no clear error in logs, requiring systematic troubleshooting.

Symptoms

• Container exits immediately after start with status Exited (1) or Exited (137)
• docker logs shows no output or only generic messages
• kubectl describe pod indicates CrashLoopBackOff
• Host dmesg shows OOM or kernel errors
• Processes inside container are unresponsive or timeout

Diagnosis

1. Check Container Logs

docker logs --tail 100 <container_id>
docker logs --since 5m <container_id>

If logs are not helpful, try attaching to stdin/stdout:

docker attach --sig-proxy=false <container_id>

2. Check Container State and Exit Code

docker inspect <container_id> --format '{{.State.ExitCode}}'
docker inspect <container_id> | jq '.[0].State'

• Exit code 137: SIGKILL, typically due to OOM or docker stop
• Exit code 1: application error, check application logs
• Exit code 0: normal exit, but may be taken over by init process

3. Check Resource Limits and OOM

docker stats --no-stream <container_id>
cat /sys/fs/cgroup/memory/docker/<container_id>/memory.oom_control

Check if oom_kill counter increased.

4. Check Filesystem and Storage Driver

docker info | grep -i "Storage Driver"
df -h /var/lib/docker

Storage driver (e.g., overlay2) full or inode exhaustion prevents container writes.

5. Check Network and DNS

docker exec <container_id> ping -c 3 google.com
docker exec <container_id> nslookup <service_name>

If DNS resolution fails inside container, check /etc/resolv.conf or Docker network mode.

6. Check Kernel and cgroup Version

uname -r
cat /sys/fs/cgroup/unified/

Docker 20.10+ defaults to cgroup v2; some older applications may be incompatible.

7. Use docker events for Real-Time Monitoring

docker events --filter 'container=<container_id>'

Observe container lifecycle events.

Commands

# View process tree
docker top <container_id>

# Enter namespace for debugging
docker run --rm -it --pid=container:<container_id> --net=container:<container_id> --cap-add SYS_PTRACE nicolaka/netshoot

# Export container filesystem
docker export <container_id> -o container.tar

# Inspect image layers
docker history <image_name>

Risk Controls

• Never execute docker restart or docker rm -f directly in production without first running docker pause or health checks
• During diagnosis, use read-only commands like docker logs and docker inspect to avoid altering container state
• Before restarting, create a snapshot or run docker commit to save current state
• When adjusting kernel parameters, first verify in a test environment

Rollback

1. Record current container configuration: docker inspect <container_id> > container_config.json
2. Stop the problematic container: docker stop <container_id>
3. Restore previous image version: docker pull <image_name>:<previous_tag>
4. Restart with original parameters: docker run --name <container_name> ... <image_name>:<previous_tag>
5. If managed by orchestrator, update configuration and redeploy: kubectl rollout undo deployment/<deployment_name>

Verification

• Confirm container running status: docker ps -a | grep <container_name>, Status should be "Up"
• Check application health endpoint: curl -f http://localhost:<port>/health
• Monitor logs for errors: docker logs --tail 50 <container_id> shows no exception stacks
• Verify resource usage: docker stats --no-stream <container_id> shows CPU/memory within expected range
• Test network connectivity: access container from other services without errors

When to Submit an OpsGlobal Ticket

Contact OpsGlobal experts immediately when:

• Container crashes involve kernel panic, irrecoverable filesystem corruption, or hardware failure
• Diagnostic commands report "permission denied" or "system call restricted" and cannot be safely escalated with --privileged
• Container runtime behavior deviates significantly from official documentation, suggesting a bug in Docker or containerd
• Failure affects multiple nodes or the entire cluster, requiring multi-resource coordination
• After exhausting the above steps, the container still cannot recover and business downtime exceeds RTO

When submitting a ticket, attach: docker info, docker version, relevant syslog excerpts, the problematic container ID, and output of diagnostic commands.