Deep Dive: Docker Container Runtime Troubleshooting for SREs

Scenario

It's 2 AM. Your phone rings. A critical containerized API on a production host has been restarting for the last 20 minutes. docker ps shows Restarting, docker logs is empty. The Service Level Objective is already at risk. You need to act quickly but methodically.

The host runs several containers. The application team reports that one service keeps restarting, but there are no errors in the logs. The host load is slightly elevated, but not at capacity. You suspect the Docker runtime itself, not the application code.

Symptoms

These common indicators will help you narrow down the problem early:

• docker ps shows the container in Restarting state with an increasing restart count.
• docker inspect reports ExitCode: 137 or OOMKilled: true — the classic signature of an out-of-memory kill.
• journalctl or dmesg contains lines like Out of memory: Kill process or oom-killer.
• df -h shows the root partition or Docker’s data directory /var/lib/docker is full.
• Container start fails with errors such as failed to create shim task or runc create failed.
• The container starts but exits silently after a few minutes, with no output in docker logs.

Diagnosis

Work from the outside in. Follow this sequence:

1. Check Docker Daemon Status

systemctl status docker
journalctl -u docker --since "10 minutes ago"

See if the daemon is crashing due to resource exhaustion or a deadlock. If the service is inactive (dead) or in a restart loop, the problem is at the Docker level.

2. Get Detailed Container Information

docker ps -a --filter "name=your-service"
docker inspect <container-id> --format '{{.State.Status}} | ExitCode={{.State.ExitCode}} | OOMKilled={{.State.OOMKilled}}'

If OOMKilled is true, the container was killed by the kernel OOM killer.

3. Check Kernel Logs

dmesg -T | grep -i -E "oom|killed process" | tail -20
journalctl -k --since "10 minutes ago" | grep -i oom

These commands show which process/container triggered the OOM and what the memory pressure looked like at the time.

4. Inspect Container Logs and Stdout

docker logs --tail 50 <container-id>

Note: if the container restarts rapidly, use docker logs --tail 50 --timestamps to view the timeline. Sometimes the application writes to stderr but the logging driver doesn’t forward it correctly.

5. Check Host Disk and Docker Disk Usage

df -h
docker system df
du -sh /var/lib/docker/*

Uncleaned images, volumes, or log files can saturate the disk and prevent containers from creating files or writing.

6. Inspect cgroup Limits

cat /sys/fs/cgroup/memory/docker/<container-id>/memory.oom_control
cat /sys/fs/cgroup/memory/docker/<container-id>/memory.limit_in_bytes
cat /sys/fs/cgroup/pids/docker/<container-id>/pids.current

Verify whether the container’s memory and PID limits are too low. If the host itself is out of memory, the container can still be killed even with a generous limit.

7. Verify Runtime Components (containerd/runc)

docker info | grep -i runtime
systemctl status containerd

A faulty runc or containerd will prevent containers from starting.

Risk Controls

Before starting remediation, reduce the blast radius:

• Avoid restarting the daemon: systemctl restart docker will interrupt all containers unless you are ready to isolate the risk.
• Prefer read-only commands: inspect, logs, and dmesg do not change state.
• Preserve evidence: Save the output of docker inspect, dmesg, and journalctl for later analysis.
• Drain the node: If the host runs replicas, mark the node as maintenance in your load balancer (e.g., Nginx/LVS) and then troubleshoot in peace.

Rollback Strategy

If the problem started after a recent change, a quick rollback is often the safest move:

• Switch image tag: If latest was just updated, roll back to the previous stable version, e.g., v1.0.1.
• Adjust container parameters: If memory limits are too low, recreate the container with a larger --memory and --memory-swap.
• Recreate the container: Use docker run instead of docker start to preserve mounts and network settings.

Before rolling back, verify that the old image tag is still available locally (docker images).

Verification

After fixing, don’t just check that the process is alive:

docker ps --filter "status=running" --filter "name=your-service"
docker inspect --format '{{.State.Status}}, RestartCount={{.RestartCount}}' <container-id>

Then test the actual functionality:

curl -f http://localhost:8080/healthz

Observe for at least 15 minutes to ensure no OOM or restart recurrence. Use docker stats to monitor resources:

docker stats --no-stream <container-id>

Confirm that memory utilization stays below 60% of the limit and CPU stays within expectations.

When to Submit an OpsGlobal Ticket

If you encounter any of the following, don’t continue debugging in the dark — open a ticket immediately:

• You’ve followed all the steps above but the issue hasn’t been resolved within 1 hour.
• The issue requires tuning kernel parameters (e.g., vm.overcommit_memory) or there’s a suspected runc/containerd bug.
• Multiple containers on the same host are affected, and the blast radius is large.
• You suspect hardware problems or low-level driver issues, like a faulty device mapper.

OpsGlobal’s SRE experts can take over within 15 minutes, providing 24/7 remote support to protect your SLO and data integrity.