Repository navigation
Description
Parent: #970
Deliverable
Metrics and structured diagnostics that let an engineer determine why a connector cannot forward and whether it owns an installed /96 route.
Dependencies
#980, #981, foundation #1001; live evidence may share #1000/#1002 fixtures.
Galactic Edge integration (#992)
Implement this milestone as a consumer of the Galactic Edge provider contract. #992 owns provider authorization, host attachment lifecycle, shared-state fencing and maintenance, independently enforced forwarding expiry, prefix conflict arbitration/publication, and gVPC traffic policy. MASQUE owns client authentication, connector /96 assignment and retention, session packet checks, transport behavior and its provider adapter. Use the foundation's versioned contract and existing provider helpers; do not introduce a second host-networking controller, lease store or BGP publisher. Any required foundation change must be tracked and qualified under #992 before this milestone claims integrated acceptance.
Work
- Expose active/admitted/failed connector and tunnel counts by transport, connection failure reasons, policy drops, resource usage and route programming status.
- Expose ownership expiry and revocation outcomes and distinguish admitted, programmed and forwarding-ready states.
- Provide structured correlation using stable connector, gVPC, attachment and ownership identifiers; bound metric label cardinality and exclude credentials and packet contents.
- Document a short diagnostic workflow using the emitted telemetry. Do not add a portal or public management API.
Acceptance criteria
- Automated tests or fixtures demonstrate telemetry for successful connection, authentication denial, policy denial, route-programming failure and ownership expiry.
- An engineer can correlate a connector to its active attachment and route state without inspecting secrets.
- Metric cardinality remains bounded as connector identities churn, and diagnostics distinguish HTTP/3 and HTTP/2 failures.
- The capacity harness can consume the exposed resource and connector measurements.
Additional integration acceptance
- Reuse Deliver foundation diagnostics #1001 foundation metrics/events and
docs/edge/troubleshooting.md; add protocol/client-session telemetry rather than duplicate foundation state ownership. - Correlate connector/session/transport with provider, service UID, instance/lease epoch, attachment and accepted route epoch in bounded diagnostics. Keep unique identities out of unbounded metric labels.
- Exercise a live troubleshooting flow distinguishing client denial, provider denial, programmed-but-not-ready, unaccepted claim, expired gate, policy denial and remote convergence. Confirm actual exported metrics/alerts for the pinned build, not merely a metric definition.
Scope follows #970: CONNECT-IP with HTTP/2 fallback; no public API development, plugin/portal integration, product-client development or production rollout.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity
[-]MASQUE: expose connector lifecycle, policy and route diagnostics[/-][+]Expose connector lifecycle, policy and route diagnostics[/+]privateip commentedon Oct 28, 2026
Verdict: valid. The telemetry this issue asks for does not exist yet.
I checked
main(bb6d1f2) and the head of the #980/#981 stack (78a1b53, #1207), which this work builds on.internal/edge/connectipandinternal/edge/masqueexport one metric,galactic_connectip_packets_denied_total{direction, reason}. Everything else is in free-text logs only:Server.rejectlogs a status and a sentence, with no transport and no counter. A 401, a 403, a hit limit and a backend failure can't be told apart from/metrics.session.endlogsgrant expired,grant revoked,replacedorrevoked. Nothing counts these, and the server'sCONNECT-IP request endedline doesn't give a cause.request endedlog line.request admittedline is the session ID.The foundation already exports attachment and route state (
galactic_edge_foundation_*, #1001). The new telemetry adds per-request and per-transport state on top of that and doesn't duplicate it.No live reproduction was needed, because this is a missing capability. The capacity harness criterion depends on #984, which doesn't have a harness yet.
privateip commentedon Oct 28, 2026
Plan. This work ships as three PRs added to stack #1205, above #1207.
internal/edge/connectip). Adds request outcomes by transport (galactic_connectip_requests_total{transport, outcome}), gauges of open connections, admissions and tunnels by transport, and connection rejections by reason. It also counts tunnel ends by cause (lease ended, client closed, backend error, server closed), packets and bytes by transport and direction, and admission latency. Session and reject logs gain transport, connector, attachment, claim and epoch.internal/edge/masque). Adds admission outcomes, request ends by reason (grant expired, revoked, changed or invalid, replaced, connector revoked, lease lost), route claim outcomes (accepted, refused, failed, no longer accepted), device-full drops and lease losses. A scrape-time gauge reports requests as admitted, programmed or forwarding, and another reports whether the lease is held. End logs carry the service UID, attachment UID, claim epoch and lease epoch.docs/edge/troubleshooting.mdand the new metrics to the observability reference.Metric labels are only transport, direction and fixed enums. Connector, principal, VPC, prefix and attachment stay in logs, so the series count doesn't grow as connectors churn. No credential or packet content is logged.
Tests cover a successful connection, authentication denial, policy denial, route claim failure, grant expiry and revocation over both transports, plus a guard that rejects identity labels.
Not in these PRs: alerts, the live troubleshooting flow (#985 lab rows) and the capacity harness (#984). Each PR says "Related to #983", and this issue stays open for those items.
docs(edge): Add a MASQUE connector diagnostic workflow
privateip commentedon Oct 28, 2026
The fix is three PRs added to stack #1205, above #1210:
All three say "Related to #983". This issue stays open for the live troubleshooting flow (#985 lab rows) and the capacity harness (#984).
docs(edge): Add a MASQUE connector diagnostic workflow
docs(edge): Document the cancelled admission outcome
docs(edge): Add a MASQUE connector diagnostic workflow
docs(edge): Document the cancelled admission outcome
docs(edge): Add a MASQUE connector diagnostic workflow
docs(edge): Document the cancelled admission outcome
feat(connectip): Export CONNECT-IP request and tunnel telemetry
docs(connectip): Name the http1 transport label
feat(masque): Export provider admission, route and request telemetry
fix(masque): Keep client cancellations out of failure counts
docs(edge): Add a MASQUE connector diagnostic workflow
docs(edge): Document the cancelled admission outcome
privateip commentedon Oct 29, 2026
Closing this. The capacity harness criterion moves to #1242, and the comment below states two partial items. The telemetry merged in #1215 (CONNECT-IP server) and #1216 (MASQUE adapter). The diagnostic workflow merged in #1217, alerts in #1227, and the lease-loss attribution fix in #1241. The containerlab lab's diagnostics-walk row passed on main at d3430bb, with images built from that commit.
transportlabel, and the walk counted HTTP/3 and HTTP/2 denials separatelylease_ended/lease_lost, and a new request got 503no_lease. Unaccepted claim: 503route_failed. Remote convergence: the compute node dropped the route 700 ms after the request ended. Programmed-but-not-ready lasts at most one renewal attempt, and no scrape caught it live. A unit test covers that state