Skip to content

Adopt the shared Milo instrumentation library #1144

Closed
9 of 9 issues completed
Closed

Description

@privateip

Goal

Adopt the shared Milo instrumentation library (milo-os/instrumentation) in every Galactic binary, so Galactic's metrics and logs follow the same conventions as other Milo services.

The library standardizes three things: a production logger (structured JSON at info level, carrying the binary's version and the library's version on every line), a single metrics registry whose constructors refuse label names the collection pipeline reserves, and a build-info metric to join on when breaking a query down by version. A conformance test fails the build when any of these stops being true. The adoption guide is ADOPTING.md.

Galactic is already close on metrics: nothing registers on Prometheus's default registry, and most metrics are served from the controller-runtime registry. The main gaps are:

  • Several long-running binaries hand controller-runtime a development-mode logger in production, so its log lines are unstructured console text at debug level.
  • Logging is built differently in each binary (slog JSON, slog text, zap, or a slog bridge).
  • Metric constructors receive a registry as a parameter, and a few components serve metrics from a private registry. The library expects one shared registry.
  • A few metrics use label names the collection pipeline reserves for itself (node, namespace, provider, component). These collide with the pipeline's own values, and some alert rules group by them.
  • Many datapath metrics are custom collectors that compute values at scrape time. The library registers these but can't check their label names, so they need a manual review.

The library requires Kubernetes 0.37 libraries and controller-runtime 0.25 or later. Galactic is on earlier versions, so that upgrade comes first.

Done when

  • Every long-running Galactic binary logs through the library's logger in production format. The CNI plugins write the same format to their log file. Existing log call sites keep working through a slog bridge.
  • Every binary reports its version through the library, and the build-info metric is served beside its other metrics.
  • Every metric is created through the library or registered through it as a custom collector, on the one registry each binary serves. No component serves a private registry.
  • No metric sets a label the collection pipeline reserves. Each alert rule, its rule tests, and every downstream copy of the rules match the renamed labels before the rename ships.
  • The library's adoption test runs in CI with no skipped directories, alongside its metric-name lint and a scrape test for each binary's metrics endpoint.
  • The repository's agent instructions state the rule, so coding agents follow it before the test fails.

Out of scope

  • New metrics, or changes to what existing metrics measure.
  • Tracing.
  • Changes to the collection pipeline itself.

Milestones

Activity

  1. added
    epicIssue represents an epic that tracks a theme with subissues
    on Oct 27, 2026
  2. self-assigned this
    on Oct 28, 2026
  3. privateip commented on Oct 28, 2026

    @privateip
    CollaboratorAuthor

    Every milestone has landed on main:

    On main, the adoption test runs with no directories skipped. The lint, the per-command scrape tests and the reserved-label check for custom collectors run in the unit suite. AGENTS.md states the rule.

    One item from "Done when" is still in flight. The downstream copy of the fabric routing and Edge alerts, and the traffic policy dashboard, are updated to the renamed labels in a deployment repository change that is in review with auto-merge on. Until it merges, those staging alerts don't match the renamed series.

  4. added theissue type on Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

epicIssue represents an epic that tracks a theme with subissues

Type

Fields

No fields configured for Epic.

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions