Skip to content

Expose connector lifecycle, policy and route diagnostics #983

Closed

Description

@privateip

Parent: #970

Deliverable

Metrics and structured diagnostics that let an engineer determine why a connector cannot forward and whether it owns an installed /96 route.

Dependencies

#980, #981, foundation #1001; live evidence may share #1000/#1002 fixtures.

Galactic Edge integration (#992)

Implement this milestone as a consumer of the Galactic Edge provider contract. #992 owns provider authorization, host attachment lifecycle, shared-state fencing and maintenance, independently enforced forwarding expiry, prefix conflict arbitration/publication, and gVPC traffic policy. MASQUE owns client authentication, connector /96 assignment and retention, session packet checks, transport behavior and its provider adapter. Use the foundation's versioned contract and existing provider helpers; do not introduce a second host-networking controller, lease store or BGP publisher. Any required foundation change must be tracked and qualified under #992 before this milestone claims integrated acceptance.

Work

  • Expose active/admitted/failed connector and tunnel counts by transport, connection failure reasons, policy drops, resource usage and route programming status.
  • Expose ownership expiry and revocation outcomes and distinguish admitted, programmed and forwarding-ready states.
  • Provide structured correlation using stable connector, gVPC, attachment and ownership identifiers; bound metric label cardinality and exclude credentials and packet contents.
  • Document a short diagnostic workflow using the emitted telemetry. Do not add a portal or public management API.

Acceptance criteria

  • Automated tests or fixtures demonstrate telemetry for successful connection, authentication denial, policy denial, route-programming failure and ownership expiry.
  • An engineer can correlate a connector to its active attachment and route state without inspecting secrets.
  • Metric cardinality remains bounded as connector identities churn, and diagnostics distinguish HTTP/3 and HTTP/2 failures.
  • The capacity harness can consume the exposed resource and connector measurements.

Additional integration acceptance

  • Reuse Deliver foundation diagnostics #1001 foundation metrics/events and docs/edge/troubleshooting.md; add protocol/client-session telemetry rather than duplicate foundation state ownership.
  • Correlate connector/session/transport with provider, service UID, instance/lease epoch, attachment and accepted route epoch in bounded diagnostics. Keep unique identities out of unbounded metric labels.
  • Exercise a live troubleshooting flow distinguishing client denial, provider denial, programmed-but-not-ready, unaccepted claim, expired gate, policy denial and remote convergence. Confirm actual exported metrics/alerts for the pinned build, not merely a metric definition.

Scope follows #970: CONNECT-IP with HTTP/2 fallback; no public API development, plugin/portal integration, product-client development or production rollout.

Activity

  1. changed the title [-]MASQUE: expose connector lifecycle, policy and route diagnostics[/-] [+]Expose connector lifecycle, policy and route diagnostics[/+] on Oct 22, 2026
  2. added theissue type on Oct 22, 2026
  3. self-assigned this
    on Oct 28, 2026
  4. privateip commented on Oct 28, 2026

    @privateip
    CollaboratorAuthor

    Verdict: valid. The telemetry this issue asks for does not exist yet.

    I checked main (bb6d1f2) and the head of the #980/#981 stack (78a1b53, #1207), which this work builds on. internal/edge/connectip and internal/edge/masque export one metric, galactic_connectip_packets_denied_total{direction, reason}. Everything else is in free-text logs only:

    • Request rejections: Server.reject logs a status and a sentence, with no transport and no counter. A 401, a 403, a hit limit and a backend failure can't be told apart from /metrics.
    • Open connections, admissions and tunnels: the server counts them for its limits, but exports none of the counts, for either transport.
    • Why a session ended: the adapter's session.end logs grant expired, grant revoked, replaced or revoked. Nothing counts these, and the server's CONNECT-IP request ended line doesn't give a cause.
    • Route claims: an accepted, refused or timed-out claim, and a route that core stops accepting, show up only as log lines.
    • Device-full drops: the per-request count appears only in the request ended log line.
    • Correlation: the server's session log carries session, principal, VPC and prefix, but no transport, connector, attachment, claim or epoch. The only way to join it to the adapter's request admitted line is the session ID.

    The foundation already exports attachment and route state (galactic_edge_foundation_*, #1001). The new telemetry adds per-request and per-transport state on top of that and doesn't duplicate it.

    No live reproduction was needed, because this is a missing capability. The capacity harness criterion depends on #984, which doesn't have a harness yet.

  5. privateip commented on Oct 28, 2026

    @privateip
    CollaboratorAuthor

    Plan. This work ships as three PRs added to stack #1205, above #1207.

    1. connectip (internal/edge/connectip). Adds request outcomes by transport (galactic_connectip_requests_total{transport, outcome}), gauges of open connections, admissions and tunnels by transport, and connection rejections by reason. It also counts tunnel ends by cause (lease ended, client closed, backend error, server closed), packets and bytes by transport and direction, and admission latency. Session and reject logs gain transport, connector, attachment, claim and epoch.
    2. masque adapter (internal/edge/masque). Adds admission outcomes, request ends by reason (grant expired, revoked, changed or invalid, replaced, connector revoked, lease lost), route claim outcomes (accepted, refused, failed, no longer accepted), device-full drops and lease losses. A scrape-time gauge reports requests as admitted, programmed or forwarding, and another reports whether the lease is held. End logs carry the service UID, attachment UID, claim epoch and lease epoch.
    3. docs. Adds a connector diagnostic workflow to docs/edge/troubleshooting.md and the new metrics to the observability reference.

    Metric labels are only transport, direction and fixed enums. Connector, principal, VPC, prefix and attachment stay in logs, so the series count doesn't grow as connectors churn. No credential or packet content is logged.

    Tests cover a successful connection, authentication denial, policy denial, route claim failure, grant expiry and revocation over both transports, plus a guard that rejects identity labels.

    Not in these PRs: alerts, the live troubleshooting flow (#985 lab rows) and the capacity harness (#984). Each PR says "Related to #983", and this issue stays open for those items.

  6. privateip commented on Oct 28, 2026

    @privateip
    CollaboratorAuthor

    The fix is three PRs added to stack #1205, above #1210:

    1. feat(connectip): Export CONNECT-IP request and tunnel telemetry #1215: CONNECT-IP server telemetry, by transport.
    2. feat(masque): Export provider admission, route and request telemetry #1216: MASQUE adapter telemetry, with request states.
    3. docs(edge): Add a MASQUE connector diagnostic workflow #1217: the metrics reference and a connector diagnostic workflow.

    All three say "Related to #983". This issue stays open for the live troubleshooting flow (#985 lab rows) and the capacity harness (#984).

  7. privateip commented on Oct 29, 2026

    @privateip
    CollaboratorAuthor

    Closing this. The capacity harness criterion moves to #1242, and the comment below states two partial items. The telemetry merged in #1215 (CONNECT-IP server) and #1216 (MASQUE adapter). The diagnostic workflow merged in #1217, alerts in #1227, and the lease-loss attribution fix in #1241. The containerlab lab's diagnostics-walk row passed on main at d3430bb, with images built from that commit.

    Criterion Evidence Status
    Telemetry for a successful connection, authentication denial, policy denial, route-programming failure and ownership expiry Unit tests in both packages cover lifecycle, rejections, admission outcomes, request states and request ends. diagnostics-walk produced each case on the lab Met
    Correlate a connector to its attachment and route state without secrets The workflow in the troubleshooting guide, walked live by diagnostics-walk. Logs carry connector, attachment and route identifiers. No credentials or packet contents Met
    Bounded cardinality, with HTTP/3 and HTTP/2 told apart Label-bound tests in both packages. Every CONNECT-IP metric carries a transport label, and the walk counted HTTP/3 and HTTP/2 denials separately Met
    The capacity harness consumes the exposed measurements The benchmark that reads them moved to staging Moved to #1242
    A live flow tells apart client denial, provider denial, programmed-but-not-ready, unaccepted claim, expired gate, policy denial and remote convergence Client denial: 401 counted per transport, 403 counted. Policy denial: 150 source and 150 destination drops. Expired gate: the request ended 25.9 s after the authority stopped. Lease loss: both requests ended lease_ended/lease_lost, and a new request got 503 no_lease. Unaccepted claim: 503 route_failed. Remote convergence: the compute node dropped the route 700 ms after the request ended. Programmed-but-not-ready lasts at most one renewal attempt, and no scrape caught it live. A unit test covers that state Met, except programmed-but-not-ready, which is unit-tested only
    Reuse the #1001 foundation telemetry and confirm the exported metrics and alerts for the build MASQUE telemetry adds request and session state beside the foundation metrics and does not duplicate them. The metrics were read from the running provider. The alert rules (#1227) are verified with promtool tests against those metric names. The lab has no Prometheus, so the alerts did not fire live Met for metrics; alerts by promtool only
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions