Repository navigation
Description
Goal
Adopt the shared Milo instrumentation library (milo-os/instrumentation) in every Galactic binary, so Galactic's metrics and logs follow the same conventions as other Milo services.
The library standardizes three things: a production logger (structured JSON at info level, carrying the binary's version and the library's version on every line), a single metrics registry whose constructors refuse label names the collection pipeline reserves, and a build-info metric to join on when breaking a query down by version. A conformance test fails the build when any of these stops being true. The adoption guide is ADOPTING.md.
Galactic is already close on metrics: nothing registers on Prometheus's default registry, and most metrics are served from the controller-runtime registry. The main gaps are:
- Several long-running binaries hand controller-runtime a development-mode logger in production, so its log lines are unstructured console text at debug level.
- Logging is built differently in each binary (slog JSON, slog text, zap, or a slog bridge).
- Metric constructors receive a registry as a parameter, and a few components serve metrics from a private registry. The library expects one shared registry.
- A few metrics use label names the collection pipeline reserves for itself (
node,namespace,provider,component). These collide with the pipeline's own values, and some alert rules group by them. - Many datapath metrics are custom collectors that compute values at scrape time. The library registers these but can't check their label names, so they need a manual review.
The library requires Kubernetes 0.37 libraries and controller-runtime 0.25 or later. Galactic is on earlier versions, so that upgrade comes first.
Done when
- Every long-running Galactic binary logs through the library's logger in production format. The CNI plugins write the same format to their log file. Existing log call sites keep working through a slog bridge.
- Every binary reports its version through the library, and the build-info metric is served beside its other metrics.
- Every metric is created through the library or registered through it as a custom collector, on the one registry each binary serves. No component serves a private registry.
- No metric sets a label the collection pipeline reserves. Each alert rule, its rule tests, and every downstream copy of the rules match the renamed labels before the rename ships.
- The library's adoption test runs in CI with no skipped directories, alongside its metric-name lint and a scrape test for each binary's metrics endpoint.
- The repository's agent instructions state the rule, so coding agents follow it before the test fails.
Out of scope
- New metrics, or changes to what existing metrics measure.
- Tracing.
- Changes to the collection pipeline itself.
Milestones
- Use a production-format logger for controller-runtime in every binary #1145
- Upgrade to Kubernetes 0.37 libraries and controller-runtime 0.25 #1146
- Adopt the standard logger and build version in every binary #1147
- Add the instrumentation adoption test with a shrinking skip list #1148
- Create every metric through the instrumentation library #1149
- Rename metric labels reserved by the collection pipeline #1150
- Require the instrumentation library in Galactic's agent instructions #1153
Activity
privateip commentedon Oct 28, 2026
Every milestone has landed on
main:On
main, the adoption test runs with no directories skipped. The lint, the per-command scrape tests and the reserved-label check for custom collectors run in the unit suite.AGENTS.mdstates the rule.One item from "Done when" is still in flight. The downstream copy of the fabric routing and Edge alerts, and the traffic policy dashboard, are updated to the renamed labels in a deployment repository change that is in review with auto-merge on. Until it merges, those staging alerts don't match the renamed series.