Docker overlay2 disk full: diagnose and reclaim space with targeted cleanup

Short answer: when Docker reports no space left on device and overlay2 or /var/lib/docker appears to be full, do not start by deleting directories or running a blanket prune. Check whether the filesystem is out of blocks or inodes, identify Docker's actual data root and storage backend, then attribute usage to images, build cache, logs, writable layers, or volumes. That order lets you reclaim space while knowing what each action can remove.

Safety first: take a restorable VPS snapshot and an application-consistent backup before deleting Docker objects. A disk snapshot alone may not be a consistent backup of a live database. Never delete or move files by hand under /var/lib/docker; Docker owns the references and mounts in that directory. Pruned objects have no local undo: images can be pulled or rebuilt, but a deleted volume requires a backup.

A safe decision tree

1. df -hT is at 100%: the filesystem is out of blocks. Find which filesystem backs Docker and which category is consuming it.
2. df -ih is at 100%: the filesystem is out of inodes. Look for very large counts of small files; removing one large log will not solve inode exhaustion.
3. Both have headroom: investigate quotas, deleted-but-open files, a different mount than the one you checked, or a limit applied to the container's writable layer.
4. docker system df -v reports reclaimable space: clean that specific object class, starting with the lowest-impact option.
5. Prune reports 0 B: stop repeating it. Check logs, volumes, bind mounts, active writable layers, open files, and the containerd image store.

What the symptoms actually tell you

The same error can surface while unpacking an image, creating a layer, appending to a container log, starting a database, or writing a temporary file. An overlay2 path in the error identifies where the write landed; it does not prove that an arbitrary directory is orphaned. With the classic driver, OverlayFS combines read-only image layers with one writable upper layer per container. Docker's overlay2 documentation explains the lowerdir, upperdir, and merged relationship.

• Block exhaustion: old images, build cache, unbounded logs, persistent data, or application files in writable layers are common causes.
• Inode exhaustion: millions of small files in layers, application caches, or volumes can fill the inode table while gigabytes remain free.
• The root filesystem fills even though Docker has another disk: system logs, deleted-open files, or containerd's separate data path may still be on root.
• Only one container fails: inspect that container's mounts, quotas, and writable filesystem rather than assuming the host is full.

Scope, prerequisites, and recovery protection

This runbook targets Docker Engine and Docker Compose on Ubuntu or Debian. You need Docker access, sudo, an inventory of stateful services, and a maintenance window if a container must be recreated. The commands are documented procedures; they are not presented as tests run on Teramont infrastructure.

Before cleanup, take a provider snapshot where available and confirm the restore procedure. For PostgreSQL, MySQL, or another live database, also create an application-consistent backup with the database's own tooling. Establish whether uploads, databases, and configuration live in a named volume, a bind mount, remote storage, or only the container layer. Data that exists only in the writable layer must be copied or backed up before recreation.

Capture an inventory as well. It helps rebuild object references, but it is not a data backup:

umask 077
inventory="$HOME/docker-recovery-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$inventory"
docker ps -a --no-trunc > "$inventory/containers.txt"
docker image ls --digests --no-trunc > "$inventory/images.txt"
docker volume ls > "$inventory/volumes.txt"
ids=$(docker ps -aq)
[ -z "$ids" ] || docker inspect $ids > "$inventory/container-inspect.json"

Step-by-step diagnosis

1. Separate blocks, inodes, and open files

Start with the host rather than Docker:

df -hT
df -ih
findmnt -T /var/lib/docker
sudo lsof +L1

Read Use% in the first report and IUse% in the second. findmnt identifies the backing filesystem for that path, although the next step must confirm that it is the real Docker data root. When df shows substantially more usage than du, lsof +L1 can expose deleted files still held open by a process. Restart only the owning process, in a controlled window, to release those blocks.

2. Identify the version, Docker Root Dir, and backend

docker version --format 'Client={{.Client.Version}} Server={{.Server.Version}}'
docker info --format 'DockerRootDir={{.DockerRootDir}}'
docker info --format 'StorageDriver={{.Driver}}'
docker info --format 'DriverStatus={{json .DriverStatus}}'

An overlay2-only guide may be wrong for a recently installed host. Fresh Docker Engine 29.0 and later installations use the containerd image store by default. Hosts upgraded from an earlier release normally remain on classic graph drivers until the image store is explicitly enabled. containerd uses snapshotters—overlayfs by default—instead of the classic overlay2 driver. Docker's storage-driver overview describes the current selection model.

io.containerd.snapshotter.v1 in DriverStatus identifies the containerd image store. A classic setup commonly reports StorageDriver=overlay2. Do not switch backends as an emergency cleanup technique: objects from the previous backend can become hidden while their bytes remain on disk. containerd also stores compressed and unpacked image data and can use a path separate from Docker's data root. Moving Docker's data-root does not automatically relocate containerd.

3. Measure the filesystem that matters

docker_root=$(docker info --format '{{.DockerRootDir}}')
printf 'Docker Root Dir: %s\n' "$docker_root"
df -hT "$docker_root"
df -ih "$docker_root"
sudo du -xhd1 "$docker_root" 2>/dev/null | sort -h
[ ! -d /var/lib/containerd ] || sudo du -xhd1 /var/lib/containerd 2>/dev/null | sort -h

This use of du is read-only inventory, not permission to remove a large-looking subdirectory. -x keeps the scan on one filesystem. If containerd is configured with a custom root, inspect that path instead of assuming /var/lib/containerd. A directory's size alone does not tell you whether Docker still references it.

4. Attribute the disk usage Docker can report

docker system df
docker system df -v
docker ps -a --size --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Size}}'

docker system df breaks usage into images, containers, local volumes, and build cache. Verbose mode shows image shared and unique sizes. Treat RECLAIMABLE as an estimate under Docker's object rules, not proof that removing everything listed is operationally safe.

The report is not a complete host accounting. Container logs, bind mounts, remote data, deleted-open files, and some containerd storage paths need separate checks.

5. Tell writable layers from persistent storage

docker ps -a --size
docker inspect --format '{{range .Mounts}}{{println .Type .Name .Source .Destination}}{{end}}' CONTAINER_NAME
docker inspect --size --format 'SizeRw={{.SizeRw}} SizeRootFs={{.SizeRootFs}}' CONTAINER_NAME

SizeRw estimates what a container has written on top of its image. A large value often means uploads, caches, temporary files, or even a database are landing in the ephemeral layer. Move durable state to a volume or bind mount through a backed-up migration; do not copy files directly out of an overlay2 directory. Docker volumes have a lifecycle independent of a container and are the preferred Docker-managed persistence mechanism.

6. Find large container logs without modifying them

docker ps -aq | while read -r id; do
  name=$(docker inspect --format '{{.Name}}' "$id")
  path=$(docker inspect --format '{{.LogPath}}' "$id")
  [ -n "$path" ] || continue
  bytes=$(sudo stat -c '%s' "$path" 2>/dev/null || printf '0')
  printf '%s\t%s\t%s\n' "$bytes" "$name" "$path"
done | sort -nr

The loop asks Docker for each managed log path and reads file metadata with stat. Do not truncate, delete, or rotate those files with external tools. Docker warns that json-file logs are for exclusive daemon access; external writes can interfere with logging. The supported fix is bounded logging followed by controlled container recreation.

Cause-to-action matrix

Reclaim space selectively

Docker's pruning guide notes that unused objects are retained until you ask Docker to remove them, and provides a command for each object class. Use that granularity. Record df and docker system df before and after every action so the result—not the command's reputation—drives the next step.

1. Remove only reviewed stopped containers

docker ps -a --filter status=exited --format 'table {{.ID}}\t{{.Names}}\t{{.Image}}\t{{.Status}}'
docker inspect CONTAINER_NAME
docker rm CONTAINER_NAME

Check mounts, labels, and purpose before removal. docker container prune --filter "until=168h" can batch-remove older stopped containers, but it affects every match. A stopped container may still hold valuable data in its writable layer or keep a required image referenced.

2. Prune dangling images before all unused images

docker image ls --filter dangling=true
docker image prune
docker image prune -a --filter "until=168h"

The first prune removes dangling images only. -a also removes tagged images not used by an existing container, so run it only after confirming they can be pulled or rebuilt. Keep the interactive confirmation during incident response instead of adding -f.

3. Trim build cache

docker system df -v
docker builder prune --filter "until=168h"

Build cache is usually lower risk than application data, but the next build will be slower and may fail if old upstream artifacts have disappeared. If you use separate Buildx builders, review docker buildx ls and docker buildx du before applying their corresponding prune.

4. Handle volumes one name at a time

docker volume ls --filter dangling=true
docker volume inspect VOLUME_NAME
docker volume rm VOLUME_NAME

An unreferenced volume is not automatically disposable. Review its labels, Compose project, logical contents, and backup, then remove only the approved name. Avoid beginning an incident with docker volume prune.

docker system prune does not remove volumes by default. Adding --volumes changes that risk, and combining it with -a broadens the deletion set further. Do not use docker system prune -a --volumes as a first-line shortcut; deleted volume data has no local undo.

Why did prune reclaim 0 B?

Prune only removes objects that meet its criteria. A zero-byte result points to a different investigation:

• Active container logs are large, but they are not reclaimable images.
• The data lives in an in-use volume or bind mount and is not garbage.
• An active container is growing its writable layer.
• The filesystem is out of inodes rather than bytes.
• A process still holds a deleted file open.
• A separate containerd path is full, or a backend change left the old store hidden on disk.
• Image layers are shared, so apparent virtual size greatly exceeds the unique bytes that can be released.

If there is no safe deletion candidate, expand the filesystem, attach storage and plan a data-root move, or fix the application. Trading a production volume for a few minutes of headroom is not recovery.

Prevent log-driven disk exhaustion

Docker keeps json-file as the default for compatibility, and it does not rotate logs unless limits are configured. Docker's logging configuration guide recommends the local driver for many non-Kubernetes cases because it rotates automatically and uses a more efficient format. If integrations require json-file, set both max-size and max-file.

Back up the existing daemon configuration first. If it already contains registry mirrors, a data root, metrics, or snapshotter settings, merge the logging keys rather than replacing the whole object.

sudo install -d -m 0755 /etc/docker
if sudo test -f /etc/docker/daemon.json; then
  sudo cp -a /etc/docker/daemon.json "/etc/docker/daemon.json.bak.$(date +%Y%m%d-%H%M%S)"
fi
sudoedit /etc/docker/daemon.json

A bounded json-file policy might look like this:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Values under log-opts in daemon.json must be strings. These limits are an operational example, not a universal retention target; tune them to log rate, retention needs, and external observability. Docker's local driver is the alternative with built-in rotation and compression.

Validate JSON syntax and daemon semantics before a restart:

sudo python3 -m json.tool /etc/docker/daemon.json >/dev/null
sudo dockerd --validate --config-file=/etc/docker/daemon.json
sudo systemctl restart docker
docker info --format 'LoggingDriver={{.LoggingDriver}}'

Syntactically valid JSON can still conflict with a daemon flag or contain an unsupported option, which is why dockerd --validate matters. If validation fails, do not restart; fix the file or restore the backup.

Compose requires container recreation

Restarting Docker does not retrofit logging settings onto existing containers. Daemon defaults apply when a container is created. You can also make the policy explicit per Compose service:

services:
  app:
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

Validate the Compose model, then recreate one service after confirming its persistent data is mounted and backed up:

docker compose config
docker compose up -d --no-deps --force-recreate app
docker compose ps

Recreation removes the previous container's Docker-managed log and writable layer. That is the supported way to retire an oversized log, but it also destroys any application data stored outside volumes or bind mounts. Do not use docker compose down -v; the -v flag asks Compose to remove project volumes.

Verify the recovery

docker_root=$(docker info --format '{{.DockerRootDir}}')
df -hT "$docker_root"
df -ih "$docker_root"
docker system df
docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'
docker info --format 'LoggingDriver={{.LoggingDriver}}'

Confirm that both blocks and inodes have headroom, expected containers are healthy, the application can read and write, persistent mounts are still present, and usage is no longer rising at the previous rate. Check containerd's filesystem separately when applicable. Recovery is not complete merely because df shows a smaller percentage.

Rollback boundaries

If Docker will not start after editing daemon.json, restore the most recent backup, validate it, and start the daemon. Reverting the default logging driver does not change containers already recreated with another driver; each keeps its creation-time setting until another planned recreation.

Docker cannot undo removed containers, images, or build cache. Recreate services from Compose, pull or rebuild images, and recover state from verified backups. If a volume was removed, stop application writes before restoration to avoid combining incompatible states.

Monitoring and maintenance

• Alert on both block and inode utilization for Docker's filesystem and any separate containerd filesystem.
• Track docker system df, log sizes, and container SizeRw as trends rather than one-time thresholds.
• Set log retention at daemon or service level and forward important logs to external storage.
• Automate only scoped, age-filtered prunes with tested label exclusions; account for every builder and deployment image first.
• Practice volume and database restores. An untested backup is still an assumption.

Common recovery snags

Docker cannot start because the filesystem is completely full

Create a small amount of headroom outside Docker-managed data: use the system's own tools to clear known package caches or rotate system logs, or temporarily extend the filesystem. Then return to targeted Docker cleanup. Do not delete files under /var/lib/docker just to make the daemon start.

du and df disagree

Check lsof +L1 for deleted-open files, hidden mount points, filesystem snapshots, and whether the du scan excluded another filesystem. A Docker operation may be the first failure even when Docker is not the main consumer.

overlay2 remains large after cleanup

In-use and shared layers are expected to remain. Compare unique image size, container references, and writable-layer size. If the storage backend changed, the previous store may be hidden but still present; document and plan its retirement with backups instead of deleting directories by visual inspection.

FAQ

Is docker system prune safe?

Only when its deletion set matches what you can rebuild. By default it removes stopped containers, unused networks, dangling images, and build cache—not volumes. A stopped container can still contain state in its writable layer, so review the prompt and prefer per-object prune commands.

Can I delete /var/lib/docker/overlay2 manually?

No. Directory IDs are not a reliable orphan list, and Docker owns the references and mounts. Use the Docker CLI or a backed-up, documented migration or rebuild procedure.

Will removing an image delete application data?

Persistent application data should not live in an image, but a local image can still be an unreproducible deployment artifact. Preserve the Dockerfile, immutable tag or digest, and registry access before cleanup.

What should I do when inodes are full?

Find the object producing very large numbers of small files. Remove the approved container, cache, or volume through its managing tool, or migrate the workload to a filesystem sized for that pattern. Deleting a multi-gigabyte log may recover blocks without releasing enough inodes.

Does the containerd image store mean OverlayFS is gone?

Not necessarily. containerd commonly uses the overlayfs snapshotter; the storage architecture and paths change even when the kernel mechanism is related. Query docker info instead of inferring the backend from a path in an error.