Docker Container Runtime Troubleshooting: A Step-by-Step Engineering Guide

Scenario

Your production web service just returned a 502. The monitoring dashboard shows the container running your API has vanished. You SSH into the host and see that a container is stuck in a restart loop—or worse, has exited with an OCI runtime error. This is not a crash of your application code; it's the container runtime failing to start or maintain execution. This post walks through a real-world troubleshooting path for Docker container runtime issues on Linux hosts.

Symptoms

Common indicators that the runtime, not your app, is the problem:

• Container exits immediately with code 137 (SIGKILL) or 139 (SIGSEGV).
• docker inspect shows a State.Error message like: OCI runtime exec failed: exec failed: unable to start container process: exec: '/bin/sh': stat /bin/sh: no such file or directory.
• Health checks are failing and the container is in a restart backoff loop.
• Docker daemon returns: Error response from daemon: containerd: container did not start before required deadline.
• Containers are being killed due to OOM or hitting cgroup limits.

Diagnosis

Start with docker ps -a to get the container ID and exit status. Then use docker inspect to see restart counts, exit codes, and the exact error. Always collect logs before touching the container.

Useful commands:

docker ps -a --filter 'name=your-service'
docker inspect <container_id> --format '{{.State.ExitCode}} {{.State.Error}}'
docker logs <container_id> --tail 200 --timestamps
docker stats --no-stream

The Docker runtime relies on containerd and runc. Check their health:

systemctl status docker containerd
docker info | grep -i runtime
runc --version

If the error is low-level, inspect kernel messages:

dmesg | tail -50
journalctl -u containerd --since '10 minutes ago'

Risk Controls

• Always snapshot the current state with docker inspect and docker logs before any action.
• Prefer graceful shutdown: docker stop -t 30 <container_id> instead of killing immediately.
• Never run docker rm -f on a production container without first pulling the logs and confirming the replacement image is available.
• When starting a new container, apply memory and CPU limits, e.g. --memory=512m --cpus=0.5, to avoid exhausting the host.
• If using a custom runtime like gVisor, verify the runtime is correctly installed and configured before a restart.

Rollback

If the failure happened right after a deployment, rollback to the previous image tag. For example:

docker pull your-app:stable
docker run -d --name your-app-rollback --env-file .env your-app:stable

Update your deployment manager (Kubernetes, Docker Swarm, or systemd unit) to use the old tag. If the error is caused by a missing binary or misconfigured entrypoint in the image, a rollback to a previously working image will typically restore service within minutes.

Verification

After recovery, verify the container is stable:

• docker ps shows the container as running.
• docker logs --tail 20 shows normal startup logging.
• The service's health endpoint returns 200 OK.
• Run docker inspect to confirm no restart policy spikes.

Monitor for at least 10 minutes to ensure the fix is durable.

When to Submit an OpsGlobal Ticket

Escalate to OpsGlobal if:

• The runtime error persists after rolling back to a known-good image.
• You see messages like runc: symbol lookup error or overlayfs: mount error in system logs.
• Multiple hosts in the cluster exhibit the same failure pattern.
• You suspect kernel or hardware-level issues (page faults, I/O errors, NUMA problems).

OpsGlobal can perform in-depth diagnostics on the Docker daemon, containerd, runc, and kernel parameters, then apply hotfixes or long-term stability configurations.