The per-container panels in my search engine's Grafana had shown almost nothing. Memory per container, CPU per container: mostly empty, apart from a single series for the whole machine. The comments in the monitoring config blamed a known limitation, that filesystem metrics didn't work with Docker's newer storage backend. That was wrong in an important way. It wasn't the filesystem metrics that were missing. cAdvisor, the component that reports per-container metrics, wasn't seeing any containers at all.
What was running
Each of the four servers ran Grafana Alloy as its monitoring agent, and Alloy has cAdvisor built in, as prometheus.exporter.cadvisor. It was configured with docker_only = true. What it reported was container_memory_usage_bytes for the root cgroup, id="/", and nothing per container. That one series duplicates what the machine-level exporter already reports, so on these servers the built-in component added nothing.
The storage backend
All four servers ran Docker with the containerd image store. docker info -f '{{ .DriverStatus }}' shows it as driver-type io.containerd.snapshotter.v1. Docker's documentation says the containerd image store is the default storage backend for Docker Engine 29.0 and later on fresh installations, while upgraded installs keep the older overlay2 driver until you switch. So a new server gets it without anyone choosing it. My laptop's Docker, 29.9.1, uses it too, which let me reproduce the whole thing locally.
The failure, reproduced
The cAdvisor bundled in that Alloy version was old enough to look for a file that only the legacy storage driver creates. I ran standalone cAdvisor with the same mounts as the servers, in two versions, and counted the containers it reported by name:
- v0.49.2: zero, with an error for every container
- v0.55.1: all ten that were running
The v0.49.2 error is specific:
failed to identify the read-write layer ID for container "e004...": open .../image/overlayfs/layerdb/mounts/e004.../mount-id: no such file or directory
The legacy driver keeps a mount-id file per container under the image's layerdb directory. With the containerd snapshotter, that file doesn't exist. The old cAdvisor treated the missing file as a reason to give up on the container entirely, not just on its filesystem statistics, so every container was dropped. That's the gap between what the config comments said and what was happening: the comments assumed a partial loss, and it was a total one.
The same test at the time, on the servers, gave the same result: v0.49.2 failed the same way, and v0.55.1 listed every container with its Docker Compose labels. So the problem was fixed upstream in cAdvisor, not something specific to Alloy.
The fix: run cAdvisor on its own
Alloy's built-in component doesn't let you choose which cAdvisor version it bundles. The fix removed it and ran cAdvisor v0.55.1 as a separate container on each server, scraped by Alloy as an ordinary target: the same pattern the project already used for OpenSearch, which has no built-in Alloy exporter either.
On three servers, cAdvisor joined the monitoring network and was addressed by its service name. On the database server, Alloy ran with host networking, so cAdvisor did too, with --listen_ip=127.0.0.1. With host networking there's no Docker port mapping keeping an unpublished port private, so the listen address is the only thing keeping its metrics endpoint off the server's public interfaces.
The result, checked on each server: 40 containers on the pipeline server, 25 on the secondary, 14 on the frontend and 5 on the database server, each with the right host label and full Compose metadata.
What generalises
If container metrics are empty or suspiciously thin, check what storage backend Docker is using before trusting any explanation in a config comment: docker info -f '{{ .DriverStatus }}'. If it says io.containerd.snapshotter.v1, an older cAdvisor, including one bundled inside another tool, may be quietly ignoring every container. And when a component is bundled inside an agent you can't upgrade separately, running the standalone version and scraping it is often the shortest fix.
Leave a comment
Comments are reviewed before they appear.