Docker Container Runtime Troubleshooting: A Practical Guide for DevOps/SRE

Scenario

You are maintaining a microservices application that suddenly stops working — containers fail to start or behave erratically. This could be due to container runtime issues such as Docker Engine, containerd, or runc failures.

Symptoms

• Container exits immediately after starting
• Container state shows "Exited" or "CrashLoopBackOff"
• Container is running but unresponsive or erroring
• Abnormal resource usage (high CPU/memory)
• Docker daemon logs showing errors

Diagnosis

1. Check Container Logs

docker logs --tail 100 <container_id>

Look for application-level errors. If the container never started, use docker logs <container_id> to capture the last output.

2. Inspect Container State

docker inspect <container_id> | jq '.[0].State'

Examine exit code, restart count, and error details.

3. Monitor Live Events

docker events --filter 'container=<container_id>' --since '1m'

Watch container lifecycle events (start, die, OOM, etc.).

4. Check Resource Usage

docker stats --no-stream <container_id>

Review CPU, memory, and I/O metrics.

5. Inspect Underlying Runtime State

Use crictl to communicate directly with containerd:

crictl ps -a
crictl inspect <container_id>

Get detailed info about the container managed by kubelet.

6. Check Docker Daemon

journalctl -u docker.service --since '1 hour ago' | grep ERROR

Or view Docker log file: /var/log/docker.log.

Diagnostic Commands Summary

Risk Controls

• Avoid destructive commands like docker rm -f or docker system prune directly in production.
• Reproduce the issue in an isolated test environment.
• Backup container volumes and important data before making changes.
• Have a rollback plan ready before starting diagnostics.

Rollback Strategies

1. Restart the container: Try docker restart <container_id>.
2. Rollback image version: Pull the previous known-good image tag: bash docker pull myapp:v1.0.0 docker stop <container_id> docker rm <container_id> docker run -d --name myapp_container myapp:v1.0.0
3. Use orchestration rollback: e.g., kubectl rollout undo deployment/myapp on Kubernetes.
4. Restore container volumes: If using bind mounts, restore data from backup.

Verification

• Confirm container is running: docker ps shows status "Up".
• Check health endpoint: curl http://localhost:8080/health returns 200.
• Review application logs for no errors.
• Run functional tests to ensure service is healthy.

When to Submit an OpsGlobal Ticket

• Issue persists after initial troubleshooting.
• Kernel-level errors (OOM kill, kernel panic) appear.
• Container runtime components (containerd, runc) fail affecting multiple containers.
• Deep performance analysis or root cause analysis is needed.
• Production environment experiencing sustained outage — requires 24/7 expert support.

OpsGlobal provides remote DevOps/SRE support to quickly triage and resolve complex runtime issues, minimizing downtime.