Docker Container Runtime Troubleshooting: A Practical Guide for Site Reliability Engineers

Scenario

In production environments, Docker containers may suddenly stop, fail to start, or exhibit erratic behavior. Common scenarios include: containers repeatedly restarting (CrashLoopBackOff), Out-of-Memory (OOM) kills due to memory leaks, disk space exhaustion, and volume permission errors.

Symptoms

• Container status shows Exited or CrashLoopBackOff (in Kubernetes)
• Host logs contain "out of memory" or "cannot allocate memory"
• docker ps output missing expected containers
• Application unresponsive but container appears running (zombie process)

Diagnosis

1. Check Container State

docker ps -a | grep <container-name>
docker inspect <container-id> --format '{{.State.Status}}'

2. View Logs

docker logs --tail 100 <container-id>
docker logs --since 5m <container-id>

3. Monitor Resource Usage

docker stats --no-stream <container-id>

4. Examine Host Resources

free -h
df -h
top -bn1 | head -20

5. Inspect Docker Daemon

journalctl -u docker --since "5 minutes ago"

Commands

• Restart container: docker restart <container-id> (temporary relief)
• Remove and recreate: docker rm -f <container-id> then re-run
• Debug running container: docker exec -it <container-id> /bin/sh (if still running)
• Export filesystem: docker export <container-id> > container.tar

Risk Controls

• Before executing docker restart in production, ensure health checks are configured with proper stop_grace_period.
• Using docker rm -f kills processes immediately and may lose unsaved data. Prefer docker stop to allow signal handling.
• In Kubernetes, avoid direct docker commands on Pod containers; use kubectl to prevent orchestration state mismatch.

Rollback

• If issue stems from an image update, revert to previous stable tag: bash docker pull <image>:<previous-tag> docker-compose up -d (if using Compose)
• In Kubernetes, use kubectl rollout undo deployment/<name> to revert to the previous revision.

Verification

• Confirm container state Up: docker ps or kubectl get pods
• Application health endpoint returns 200: curl http://localhost:<port>/health
• Resource usage stable: docker stats shows memory/CPU within expected limits
• No errors in recent logs: docker logs --tail 50 <container-id> shows no anomalies

When to Submit an OpsGlobal Ticket

Submit a ticket for our SRE team when: - Container still unrecoverable after following the above steps - Docker storage driver (e.g., overlay2) corruption requiring filesystem repair - Multiple hosts experience Docker daemon failures simultaneously - Need analysis of Docker daemon coredump or debug mode (-D) logs - Must evaluate and adjust Docker runtime parameters (e.g., --default-ulimit or --oom-score-adjust)