feat(orchestrator): ship per-container metrics via prometheus#1071
Merged
Conversation
added 3 commits
June 5, 2026 12:52
Adds optional cadvisor + prometheus-agent services to the generated docker-compose.yml, gated by METRICS_REMOTE_WRITE_URL. cadvisor exposes per-container cpu/memory/pids/network/fs metrics; the prometheus agent scrapes it locally and remote_writes to a central endpoint, mirroring the push-only model the asb hosts already use for logs. The bearer token and host label are reused from the Promtail config so metrics and logs authenticate identically and share one Grafana selector.
v0.49.1's docker client speaks API 1.41, which Docker Engine 29 rejects (min 1.44), so the docker factory fails to register and per-container series lose their name=/image= labels. Bump to v0.52.1 and add cgroup: host so the cadvisor container can see the host's docker-*.scope cgroups under cgroup v2 (without it, scopes enumerate empty).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an opt-in metrics pipeline to the orchestrator-generated docker-compose, parallel to the existing Promtail log shipping. Setting METRICS_REMOTE_WRITE_URL adds two services: cadvisor, which exposes per-container cpu, memory, pids, network and filesystem metrics, and a prometheus agent that scrapes cadvisor locally and remote_writes the samples to a central endpoint. This mirrors the push-only model the asb hosts already use for logs, since the hosts only make outbound connections and cannot be scraped.
Metrics deliberately reuse the Promtail bearer token and the host instance label rather than introducing their own, so the central collector authorizes logs and metrics with the same token and a deployment selects its metrics and logs with the same host query in Grafana. Because of that reuse, enabling metrics without Promtail configured is a hard error rather than a silent unauthenticated push.