Practical Docker Container Runtime Troubleshooting Guide

Scenario

A production Docker container exits unexpectedly with status "Exited (1)". The application had been running stable for days. The container is based on a custom image and uses default runtime settings.

Symptoms

• docker ps shows the container state as "Exited"
• docker logs <container> prints runtime errors like "exec: permission denied" or "OCI runtime create failed"
• System logs (e.g., journalctl -u docker) show containerd or runc errors
• Container fails to start even after manual restart

Diagnosis Steps

1. Check container logs: docker logs <container> for the most recent output
2. Inspect container state: docker inspect <container> --format='{{json .State}}' to view exit code and error message
3. Check Docker daemon health: systemctl status docker or journalctl -u docker -n 100
4. Test basic runtime: docker run --rm busybox echo test to verify the container engine works
5. Examine resource limits: docker inspect <container> | jq '.[].HostConfig.Memory' to confirm memory and CPU settings
6. Verify image integrity: docker run --rm <image> /bin/sh -c 'ls' to ensure the image is functional

Key Commands

docker logs <container>
docker inspect <container>
docker system df   # Check disk usage
journalctl -u docker -n 50 --no-pager
cat /var/log/syslog | grep docker

Risk Controls

• Use --restart=always or --restart=on-failure to auto-restart containers
• Set resource limits (--memory, --cpus) to prevent OOM kills
• Avoid using --privileged unnecessarily
• Implement HEALTHCHECK instructions in Dockerfiles

Rollback Procedure

• If a new image version caused the issue, revert to the stable tag: docker pull <image>:stable then recreate the container
• If configuration files were modified, restore from backup and restart
• Use docker commit to snapshot the current container state (temporary measure only)

Verification Steps

1. After starting the container, tail logs: docker logs -f <container>
2. Check container status: docker ps should show "Up"
3. Perform application health checks (e.g., curl endpoints)
4. Load test to ensure stability under expected traffic

When to Submit an OpsGlobal Ticket

• Basic troubleshooting fails and deep system-level analysis is needed
• Suspected kernel, storage driver, or network plugin (e.g., overlay) issues
• Need to modify Docker daemon configuration or upgrade Docker version
• Require professional assistance to recover data or optimize performance