From 3140dbdf4b194df5e534321fea176b9d960dc40d Mon Sep 17 00:00:00 2001 From: Peter Sprygada Date: Mon, 26 Oct 2026 16:43:27 -0400 Subject: [PATCH] docs(masque): Define connector contracts Define client authority and retained prefix allocation on the existing Edge provider contract so transport implementations share ownership and expiry rules. Separate per-connector service deadlines from the shared provider instance lease. Map feature requirements and historical findings to acceptance checks, including replacement, revocation and resource budgets. Mark the older proposal as superseded for this implementation. Fixes #972 --- docs/designs/README.md | 3 +- docs/designs/galactic-edge.md | 2 +- docs/designs/masque-gateway.md | 2 + docs/designs/masque/acceptance.md | 103 +++++++++++++++++++ docs/designs/masque/contract.md | 161 ++++++++++++++++++++++++++++++ 5 files changed, 269 insertions(+), 2 deletions(-) create mode 100644 docs/designs/masque/acceptance.md create mode 100644 docs/designs/masque/contract.md diff --git a/docs/designs/README.md b/docs/designs/README.md index b54ae44a..32247a4d 100644 --- a/docs/designs/README.md +++ b/docs/designs/README.md @@ -10,6 +10,7 @@ These documents explain decisions and planned work. Their status describes imple | [Looking Glass backend service](looking-glass-service.md) | Proposed implementation contract; authentication scaffolding does not enable service execution | [Current fabric-api](../fabric-api/README.md) | | [Shared DNS64](dns64/README.md) | Proposed; end-to-end validation outstanding | [NAT configuration](../nat/configuration.md) for implemented translation | | [Latency-aware routing](latency-aware-routing.md) | Exploratory proposal | [Router configuration](../router/configuration.md) for available settings | -| [MASQUE gateway](masque-gateway.md) | Exploratory proposal; external references may describe historical designs | [Gateway configuration](../gateway/configuration.md) for the implemented XDP/DSR gateway | +| [MASQUE gateway](masque-gateway.md) | Superseded for #970 by the connector contract | [Gateway configuration](../gateway/configuration.md) for the implemented XDP/DSR gateway | +| [MASQUE connector](masque/contract.md) | Proposed contract; [acceptance checks](masque/acceptance.md) not yet run | [Provider contract](../edge/provider-contract.md) for the implemented foundation | When implementing a proposal, move the available interface or operating procedure into its component guide and update this status. Retain decisions and alternatives here. Keep measurements tied to their revision and environment in the [evidence archive](../archive/README.md), rather than presenting old results as current readiness. diff --git a/docs/designs/galactic-edge.md b/docs/designs/galactic-edge.md index a522421b..c5dbdad1 100644 --- a/docs/designs/galactic-edge.md +++ b/docs/designs/galactic-edge.md @@ -86,7 +86,7 @@ Consider a service that connects an external site to a tenant's VPC. It establis The provider requests an attachment for the tenant and reachability for the prefix. Galactic Edge verifies the grant, prepares the isolated connection, and enables forwarding and routing only when their prerequisites hold. Return traffic follows the published prefix to the attachment; the provider delivers it to the external site. -A second service using a different protocol follows the same lifecycle. This is the reuse requirement in [#992](https://github.com/datum-cloud/galactic/issues/992), illustrated through an abstract provider. The [MASQUE proposal (#970)](https://github.com/datum-cloud/galactic/issues/970) is one potential consumer; its transport requirements are outside this design. +A second service using a different protocol follows the same lifecycle. This is the reuse requirement in [#992](https://github.com/datum-cloud/galactic/issues/992), illustrated through an abstract provider. The [MASQUE connector contract](masque/contract.md) defines the #970 consumer: client authority, retained /96 allocations and transport sessions. Its [acceptance matrix](masque/acceptance.md) separates foundation evidence from MASQUE integration checks; those transport requirements remain outside this foundation design. ### Risks and Mitigations diff --git a/docs/designs/masque-gateway.md b/docs/designs/masque-gateway.md index a5d4d294..588df9b2 100644 --- a/docs/designs/masque-gateway.md +++ b/docs/designs/masque-gateway.md @@ -1,5 +1,7 @@ # MASQUE Gateway Architecture +> For the #970 implementation, this proposal is superseded by the [MASQUE connector contract](masque/contract.md) and [acceptance matrix](masque/acceptance.md). The resource model, client work and transport scope below are historical context. + Status: proposed. This exploratory design is separate from the implemented XDP/DSR galactic-gateway and is not deployment guidance. References to external client and connector architectures are historical context, not verified current integrations. ## Executive Summary diff --git a/docs/designs/masque/acceptance.md b/docs/designs/masque/acceptance.md new file mode 100644 index 00000000..034eed08 --- /dev/null +++ b/docs/designs/masque/acceptance.md @@ -0,0 +1,103 @@ +# MASQUE acceptance and evidence matrix + +This matrix accompanies the [connector contract](contract.md) for [#972](https://github.com/datum-cloud/galactic/issues/972). Every check below is a required deliverable, **not an executed result**. [#985](https://github.com/datum-cloud/galactic/issues/985) owns the final evidence index. A closed foundation issue or a passing controlled-backend protocol test does not qualify the integrated MASQUE service. + +## Evidence rules + +Each result records its row ID, exact command, test name, UTC date, commit and image digests, schema/manifests, client/library versions, hardware, Talos/Cilium/Kubernetes/kernel versions, actual provider privileges, and pass/fail. Keep core, independent-provider and MASQUE artifact identities separate. Include the versions containing the fixes from [#1064](https://github.com/datum-cloud/galactic/issues/1064), [#1066](https://github.com/datum-cloud/galactic/issues/1066) and [#1067](https://github.com/datum-cloud/galactic/issues/1067) and rerun affected cases. Pin the tenant addressing reference revision used for provisioning. A failure or required skip blocks the row; readiness and mock-only tests are insufficient packet evidence. + +Packet cases use sequence-numbered traffic in both directions, capture at sender and all relevant receivers, and first prove a positive control. A denied packet must reach neither its workload nor another connector. Record attempted, delivered and denied counts, capture loss and clock uncertainty. Failure tests sample at most 20 ms apart and report the last delivered packet relative to the named trigger. Include uncertainty in the upper bound; do not add an undocumented tolerance to the pass threshold. Keep local cutoff, local BGP withdrawal, reflector withdrawal and remote convergence as separate measurements. + +[#985](https://github.com/datum-cloud/galactic/issues/985) packages setup, fault injection, collection and cleanup commands in a clean-checkout fixture. It asserts removal of test-created links, routes, maps, namespaces and API resources and preservation of unrelated resources. The ledger's retained/tombstoned allocations are expected state, identified explicitly rather than called a leak. Cleanup errors fail the run. Tests never deploy to production as part of this milestone. + +## Epic and implementation checks + +Run packet, authentication, isolation and failure cases on both HTTP/3 and HTTP/2 unless the row explicitly tests one transport. For HTTP/2, record negotiated h2/TLS/TCP and block UDP/443; a QUIC handshake does not count. + +| ID | [#970](https://github.com/datum-cloud/galactic/issues/970) requirement / deliverable | Executable scenario and pass condition | Owner | +| --- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| M01 | Protocol selection and independent clients | Inventory current connectip code; pin candidate libraries and independent clients; Extended CONNECT, capsules, framing, cancellation and bidirectional 1280-byte IPv6 work against a controlled backend. Publish exact commands and actual negotiated versions; failed candidates require retest before implementation. | [#973](https://github.com/datum-cloud/galactic/issues/973) | +| M02 | Done 1: transport lifecycle | Against real local and remote workloads, connect/send/receive/disconnect/reconnect on each transport; UDP-blocked HTTP/2 works with no QUIC dependency. Certificate/hostname failure refuses connection. | [#980](https://github.com/datum-cloud/galactic/issues/980), [#981](https://github.com/datum-cloud/galactic/issues/981), [#985](https://github.com/datum-cloud/galactic/issues/985) | +| M03 | Done 1: MTU | Deliver 1280-byte inner packets both ways; reduce live path MTU and send oversized packets. Observe correct ICMPv6 PTB or explicit tunnel failure, addressed to the right tenant. Measure device, tunnel, SRv6 and underlay limits separately, including NAT egress; HTTP/2 capsules span TCP reads without a QUIC datagram limit. | [#982](https://github.com/datum-cloud/galactic/issues/982) | +| M04 | Done 2: authentication and grants | Missing, malformed, expired, rotated, disabled and wrong-VPC credentials fail before probes/I/O. Test authority TLS identity/trust rotation, forged provider identity and deleted/recreated VPC/service/grant UIDs. Replay a stale successful authority response after its deadline: no admission. No raw credentials in logs. | [#977](https://github.com/datum-cloud/galactic/issues/977) | +| M05 | Done 2: single-connector expiry | Two connectors share one renewing EdgeInstance. Expire A's credential, then separately revoke A's grant, while B carries traffic. A meets expiry + 1 s / revocation + 31 s bounds; B continues. Repeat with authority unavailable, provider control worker stalled, and delayed grant reads/writes. Test issuer/node clocks at both 2 s skew limits. | [#976](https://github.com/datum-cloud/galactic/issues/976), [#977](https://github.com/datum-cloud/galactic/issues/977) | +| M06 | Done 2: bidirectional policy | Policy allow/deny on each tenant in each direction, missing policy, invalid policy and live policy change. Positive controls precede denials; capture non-arrival at all wrong receivers and assert drop reasons/counters, including ICMP. | [#979](https://github.com/datum-cloud/galactic/issues/979) | +| M07 | Done 3: tenant isolation | Identical /96s in different VPCs, same node and same provider process; send to an address existing only in the other VPC. No cross-VPC delivery in either direction, with cold neighbors and after restart. Demux never falls back to another tenant. | [#974](https://github.com/datum-cloud/galactic/issues/974), [#979](https://github.com/datum-cloud/galactic/issues/979) | +| M08 | Done 4: independent expiry | Provider crash/pause, control-worker-only pause with data still flowing, agent pause, API-only partition, node reboot. Verify foundation 15.52 s real-time cutoff after last applied renewal start; local withdrawal within 1 s when router runs. Resume cannot reopen from cached authority. | [#976](https://github.com/datum-cloud/galactic/issues/976), [#985](https://github.com/datum-cloud/galactic/issues/985) | +| M09 | Done 4: same-instance replacement | Race HTTP/3 and HTTP/2 requests for one connector; pause an old writer at each I/O boundary, then run late Close/release and queued return delivery. One generation forwards; predecessor queues are discarded. If quiescence exceeds 1 s, replacement fails closed. Other connectors survive. | [#976](https://github.com/datum-cloud/galactic/issues/976), [#981](https://github.com/datum-cloud/galactic/issues/981) | +| M10 | Done 4: cross-instance replacement | Keep old client/gVPC connectivity available; retire its service while its instance renews for other connectors. Replace on same/other node, including API partition. Captures show no dual forwarders; service/pool reuse waits for release. Record potentially unbounded availability wait separately from bounded packet cutoff. Stale cleanup cannot affect successor. | [#975](https://github.com/datum-cloud/galactic/issues/975), [#976](https://github.com/datum-cloud/galactic/issues/976), [#985](https://github.com/datum-cloud/galactic/issues/985) | +| M11 | Done 5: durable /96 | Race allocations through ledger CAS; lose write replies, restart authority, delete routes and reconnect. Same connector keeps one /96; another cannot obtain it. Missing/recreated ledger refuses allocations. Retirement tombstones, never reassigns. New VPC UID cannot reuse old authority. | [#978](https://github.com/datum-cloud/galactic/issues/978) | +| M12 | Done 5: aggregate routing | Reach multiple addresses in at least two client-managed subdivisions from local and remote workloads, including an attachment node with NAT egress. Exactly one aggregate /96 is published for the holder. No dependence on attachment-address publication; conflicting route, pool, own-address and later subnet claims are rejected/revoked. | [#978](https://github.com/datum-cloud/galactic/issues/978) | +| M13 | Done 5: packet authority | Spoof outside /96, impersonate another connector, exceed destination grants, inject IPv4, malformed IPv6, extension-header abuse and fragments. Reject with bounded parsing in both directions; independently verify library checks and generated ICMP behavior. | [#979](https://github.com/datum-cloud/galactic/issues/979) | +| M14 | Done 6: capacity | Execute every baseline/load point and overload fault in the contract's benchmark. Meet per-transport goodput, CPU/RSS/queue/API/map budgets; report actual counts and all runs. Enforce limits before admission and recover within 30 s. Failure bounds still pass under flood, reconnect churn and slow devices. | [#984](https://github.com/datum-cloud/galactic/issues/984) | +| M15 | Done 6: diagnostics | Exercise success, client/provider denial, Programmed but not Ready, unaccepted claim, closed gate, policy denial, route failure and remote convergence. Correlate connector/generation with service/instance/attachment/route UIDs and epochs in logs, not unbounded metric labels. Verify live exports/alerts, transport distinction and cardinality after churn. | [#983](https://github.com/datum-cloud/galactic/issues/983) | +| M16 | Unprivileged provider integration | On target Talos/Cilium, actual agent creates the device; provider opens it with no host namespaces, hostPath, NET_ADMIN or BPF. Forged pod/service, unsupported versions and missing device fail before forwarding. Check actual token extras, RoleBindings and denied EdgeService/BGP/index/status writes. | [#974](https://github.com/datum-cloud/galactic/issues/974), [#975](https://github.com/datum-cloud/galactic/issues/975) | +| M17 | Lifecycle and maintenance | Partial create/lost reply, partial programming, pod recreation, process restart, GC while preparing/active, stale delete, drain and cleanup. Preserve unrelated/replacement resources. Same attachment can serve successive HTTP/3/HTTP/2 generations; cancelling predecessor cannot close successor or shared instance. | [#975](https://github.com/datum-cloud/galactic/issues/975), [#985](https://github.com/datum-cloud/galactic/issues/985) | +| M18 | Authority refresh and service turnover | Repeated 5 s validUntil refresh causes no unintended drain/policy outage at capacity. Freeze one service's refresh, renew other connectors, rotate credentials, and race authority leaders and revocation. Old incarnation is never recreated/refreshed. Retire old service, await Released, recreate successor with retained pool and take the stable route; no overlap or duplicate allocation. | [#976](https://github.com/datum-cloud/galactic/issues/976), [#977](https://github.com/datum-cloud/galactic/issues/977), [#978](https://github.com/datum-cloud/galactic/issues/978), [#984](https://github.com/datum-cloud/galactic/issues/984) | +| M19 | Runnable artifacts and operator setup | Build separate provider and authority artifacts; document internal configuration, scoped test grants, readiness and shutdown. Operator can issue/revoke a test credential without portal/public API/client changes; provider cannot mint an EdgeService. | [#975](https://github.com/datum-cloud/galactic/issues/975), [#977](https://github.com/datum-cloud/galactic/issues/977), [#980](https://github.com/datum-cloud/galactic/issues/980) | +| M20 | Integrated completion | Reproduce every row on pinned qualified core; include foundation evidence below and all finding dispositions. Independent-provider results remain separate. Failed protocol/platform proofs require design revision and a passing retest. Publish results and cleanup, not only a checklist. | [#985](https://github.com/datum-cloud/galactic/issues/985) | + +The one-service-per-connector design avoids a new foundation deadline API. M05 and M18 are mandatory feasibility gates before [#980](https://github.com/datum-cloud/galactic/issues/980)/#981 expose this path. If either fails, [#976](https://github.com/datum-cloud/galactic/issues/976) records the required extension under [#992](https://github.com/datum-cloud/galactic/issues/992) with its own contract and qualification evidence; dependent integration stays blocked. A design review may complete [#972](https://github.com/datum-cloud/galactic/issues/972) while those implementation checks remain assigned, but cannot mark [#970](https://github.com/datum-cloud/galactic/issues/970) integrated acceptance complete. + +## Foundation evidence reuse + +Use the complete [#993 failure matrix](../../edge/provider-contract.md#failure-test-matrix). The row groups below enumerate every numbered and lettered case; [#985](https://github.com/datum-cloud/galactic/issues/985) attaches a result locator for each individual case, not one blanket link to [#992](https://github.com/datum-cloud/galactic/issues/992). Existing `tests/edgeconformance` runs only the cases its lab can host and may skip without its environment; that is not target qualification. Reuse [#1002](https://github.com/datum-cloud/galactic/issues/1002)'s executed target evidence only when artifact versions and assumptions match the MASQUE deployment. + +| Foundation rows | Foundation owner and required evidence | MASQUE additions | +| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| 1–5 | [#994](https://github.com/datum-cloud/galactic/issues/994): admission, compatibility, trusted pod identity and capacity refusals | M04, M16, M19; verify provider cannot write grants and limits precede transport work | +| 6, 8, 9 | [#995](https://github.com/datum-cloud/galactic/issues/995): isolation, partial provisioning and delayed cleanup | M07, M17 with real MASQUE sessions and generation-bound devices | +| 7, 7a, 7b, 7c, 7d | [#998](https://github.com/datum-cloud/galactic/issues/998) plus [#994](https://github.com/datum-cloud/galactic/issues/994)/#1074: same/overlapping prefix, foreign attachment, later subnet and own-address refusals | M11, M12; include retained allocations and connector pool separation | +| 10, 11, 12, 13, 13a–13e | [#997](https://github.com/datum-cloud/galactic/issues/997): crash/stall/API partition, replacement, early takeover, stale leader, release while renewing, spec-only swap, removal with agent stuck | M05, M08–M10, M18; other connectors keep instance alive; require old/new packet intervals | +| 14, 15, 15a, 23 | [#995](https://github.com/datum-cloud/galactic/issues/995)/#997: agent pause/restart, provider restart in place and node reboot | M08, M17; process boot ID and stale session callbacks cannot reopen I/O | +| 16 | [#998](https://github.com/datum-cloud/galactic/issues/998): real BGP local/reflector withdrawal; configured negotiated hold time and remote convergence | M08, M12; collect all timing stages independently | +| 17, 18 | [#997](https://github.com/datum-cloud/galactic/issues/997)/#999: revocation and finite grant ceiling | M05, M18; short service refresh and credential ceilings while neighboring services remain live | +| 19, 20 | [#999](https://github.com/datum-cloud/galactic/issues/999): missing/invalid/live-changing policy | M06, M13; per-session source/destination checks supplement foundation policy | +| 21, 22 | [#996](https://github.com/datum-cloud/galactic/issues/996): edge-only GC and stale maintenance fences | M17; active and preparing connector attachments survive unrelated pod churn | +| 24 | [#995](https://github.com/datum-cloud/galactic/issues/995): drain | M17, M19; security expiry takes precedence over graceful shutdown | +| 25 | [#1000](https://github.com/datum-cloud/galactic/issues/1000)/#1002: independent provider and actual target platform, without core changes | M16, M20; MASQUE cannot replace this independent conformance evidence | + +[#993](https://github.com/datum-cloud/galactic/issues/993) supplies the authoritative contract; [#994](https://github.com/datum-cloud/galactic/issues/994)–[#999](https://github.com/datum-cloud/galactic/issues/999) supply the mechanisms above. [#1000](https://github.com/datum-cloud/galactic/issues/1000) supplies independent-provider conformance. [#1001](https://github.com/datum-cloud/galactic/issues/1001) additionally requires live diagnostics evidence mapped to M15 and [the foundation troubleshooting guide](../../edge/troubleshooting.md). [#1002](https://github.com/datum-cloud/galactic/issues/1002) supplies environment, capacity, actual privileges and executed failure evidence for the pinned foundation. All ten milestones under [#992](https://github.com/datum-cloud/galactic/issues/992) therefore have an explicit consumer gate. + +## Review findings ledger + +These are current-path dispositions and required regression checks, not claims that every historical bug persists or has been fixed. [#985](https://github.com/datum-cloud/galactic/issues/985) must attach a fixing revision/test result or a demonstrated non-applicability result to **each** row. Read the full issue comments when collecting evidence: [#989](https://github.com/datum-cloud/galactic/issues/989), [#991](https://github.com/datum-cloud/galactic/issues/991), [#986](https://github.com/datum-cloud/galactic/issues/986). + +| ID | Historical finding | Selected path and required disposition | Owner / check | +| --- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| A01 | [#989](https://github.com/datum-cloud/galactic/issues/989): caller-supplied grants/VPC; empty provider/VPC | Authority reads trusted grant/VPC; core admission validates pod/service. Empty or forged values cannot allocate or forward. | [#975](https://github.com/datum-cloud/galactic/issues/975), [#977](https://github.com/datum-cloud/galactic/issues/977) / M04, M16 | +| A02 | [#989](https://github.com/datum-cloud/galactic/issues/989): no-prefix requests skip prefix validation | MASQUE always requests its one allocated /96; empty assignment or grant fails admission. | [#977](https://github.com/datum-cloud/galactic/issues/977), [#978](https://github.com/datum-cloud/galactic/issues/978) / M04, M12 | +| A03 | [#989](https://github.com/datum-cloud/galactic/issues/989): Core may set arbitrary non-Ready condition | Provider writes Ready only; verify deployed admission/status permissions reject arbitrary conditions. Core writer restrictions belong to foundation. | [#975](https://github.com/datum-cloud/galactic/issues/975) / M16 | +| A04 | [#989](https://github.com/datum-cloud/galactic/issues/989): attachment MTU has no underlay ceiling; old MTU readback drift | Measure actual encapsulation headroom and drift repair with pinned versions; fix host behavior under [#992](https://github.com/datum-cloud/galactic/issues/992) if necessary. | [#982](https://github.com/datum-cloud/galactic/issues/982) / M03 | +| A05 | [#989](https://github.com/datum-cloud/galactic/issues/989): positive isolation lacks per-direction negatives | Capture denied traffic separately in each direction with positive controls and a destination unique to the other tenant. | [#974](https://github.com/datum-cloud/galactic/issues/974), [#979](https://github.com/datum-cloud/galactic/issues/979) / M07 | +| A06 | [#989](https://github.com/datum-cloud/galactic/issues/989): TUN registered as InterfaceTypeVeth | Verify actual PacketAccess registration and both datapath directions; record why the selected registration is correct for the host link. | [#974](https://github.com/datum-cloud/galactic/issues/974) / M07, M16 | +| A07 | [#989](https://github.com/datum-cloud/galactic/issues/989): policy only on VPC A; no ICMP deny-counter assertion | Add denied VPC B traffic while A is allowed, plus ICMP counter and non-arrival checks. | [#979](https://github.com/datum-cloud/galactic/issues/979) / M06 | +| A08 | [#989](https://github.com/datum-cloud/galactic/issues/989): FilterList errors and opt-in CNI controls allow vacuous pass | Visibility errors fail fixture setup; assert attached CNI programs and a positive control before negative probes. | [#974](https://github.com/datum-cloud/galactic/issues/974), [#985](https://github.com/datum-cloud/galactic/issues/985) / M07, M20 | +| A09 | [#989](https://github.com/datum-cloud/galactic/issues/989): ignored cleanup errors / missing host cleanup assertions | Check links, tables, maps and unrelated resources after teardown; any error fails. | [#975](https://github.com/datum-cloud/galactic/issues/975), [#985](https://github.com/datum-cloud/galactic/issues/985) / M17, M20 | +| A10 | [#989](https://github.com/datum-cloud/galactic/issues/989): 10-minute sleep equals Go default timeout; no recorded real-node run | Use explicit test timeouts exceeding the workload plus teardown; publish actual target execution, never a skip. | [#985](https://github.com/datum-cloud/galactic/issues/985) / M20 | +| A11 | [#989](https://github.com/datum-cloud/galactic/issues/989): writer uniqueness, fence directory lifetime and unbounded mutate/Fence | MASQUE never runs these host operations; reuse [#996](https://github.com/datum-cloud/galactic/issues/996)/#1002 evidence of core fencing, persistent state and stalled-writer recovery. | [#975](https://github.com/datum-cloud/galactic/issues/975), [#985](https://github.com/datum-cloud/galactic/issues/985) / M17 | +| A12 | [#989](https://github.com/datum-cloud/galactic/issues/989): claim.Instance mutex held across store calls | One renewal worker per instance key; packet/session/route operations do not share its lock. Stall admission/release and prove renewal progress. | [#976](https://github.com/datum-cloud/galactic/issues/976), [#984](https://github.com/datum-cloud/galactic/issues/984) / M08, M14 | +| A13 | [#989](https://github.com/datum-cloud/galactic/issues/989): production timing values absent; store may reuse versions | Adopt [#993](https://github.com/datum-cloud/galactic/issues/993) parameters and provider UID/resourceVersion store. Recreated records and stale writes cannot reopen authority. | [#976](https://github.com/datum-cloud/galactic/issues/976) / M08–M10 | +| A14 | [#989](https://github.com/datum-cloud/galactic/issues/989): missing NetNsID indistinguishable from ID 0; weak ownership test | Reuse qualified hostattach fix evidence and add namespace-only impostor case with matching peer indices; no mutation of the impostor. | [#974](https://github.com/datum-cloud/galactic/issues/974), [#985](https://github.com/datum-cloud/galactic/issues/985) / M16, M17 | +| A15 | [#989](https://github.com/datum-cloud/galactic/issues/989): failed re-provision downs live attachment; recovery lacks traffic | Fault drift repair, check actual packet recovery and preservation of neighboring connectors; report the affected connector's outage. | [#975](https://github.com/datum-cloud/galactic/issues/975), [#985](https://github.com/datum-cloud/galactic/issues/985) / M17 | +| A16 | [#989](https://github.com/datum-cloud/galactic/issues/989): extra routes on pod link neither pruned nor reported | Packet device namespace is foundation-owned; inject route drift in fixture and require defined repair/refusal diagnostics. MASQUE does not silently add routes. | [#974](https://github.com/datum-cloud/galactic/issues/974), [#985](https://github.com/datum-cloud/galactic/issues/985) / M16, M17 | +| B01 | [#991](https://github.com/datum-cloud/galactic/issues/991): unlimited sessions/admission and QUIC limits | Enforce pre-auth connection/stream/global admission caps and queued-byte caps before exposure, on both transports. | [#980](https://github.com/datum-cloud/galactic/issues/980), [#981](https://github.com/datum-cloud/galactic/issues/981), [#984](https://github.com/datum-cloud/galactic/issues/984) / M14 | +| B02 | [#991](https://github.com/datum-cloud/galactic/issues/991): MTU probe before admission | Capture unauthenticated request path: no probe or tenant packet before successful authorization. | [#977](https://github.com/datum-cloud/galactic/issues/977), [#980](https://github.com/datum-cloud/galactic/issues/980), [#981](https://github.com/datum-cloud/galactic/issues/981) / M04 | +| B03 | [#991](https://github.com/datum-cloud/galactic/issues/991): EAGAIN terminates sessions; shared backend mutex | Count/drop bounded device backpressure, isolate connector queues and locks; unrelated sessions continue. | [#975](https://github.com/datum-cloud/galactic/issues/975), [#984](https://github.com/datum-cloud/galactic/issues/984) / M14 | +| B04 | [#991](https://github.com/datum-cloud/galactic/issues/991): routeRanges checks only mutual route overlap | Validate against authority destination grants and assigned /96; core handles foundation prefix conflicts. | [#977](https://github.com/datum-cloud/galactic/issues/977)–[#979](https://github.com/datum-cloud/galactic/issues/979) / M12, M13 | +| B05 | [#991](https://github.com/datum-cloud/galactic/issues/991): raw TUN only observes lease.Done cancellation | Production entry points require foundation-backed adapter, finite service ceiling and session-generation checks. Paused control workers cannot keep expired forwarding alive. | [#976](https://github.com/datum-cloud/galactic/issues/976), [#980](https://github.com/datum-cloud/galactic/issues/980), [#981](https://github.com/datum-cloud/galactic/issues/981) / M05, M08 | +| B06 | [#991](https://github.com/datum-cloud/galactic/issues/991): assumed library packet checks; IPv4 never demuxes | Verify actual pinned library checks; adapter rejects IPv4 explicitly and enforces tenant/attachment scope before /96 lookup. | [#979](https://github.com/datum-cloud/galactic/issues/979) / M07, M13 | +| C01 | [#986](https://github.com/datum-cloud/galactic/issues/986): missing service_peer_table after router-before-agent roll | MASQUE EdgeRoute publication does not select the private-service route path. Prove independence on selected workload path; if that path uses private services, document required map/agent-first rollout and test missing pins. | [#974](https://github.com/datum-cloud/galactic/issues/974), [#978](https://github.com/datum-cloud/galactic/issues/978) / M12, M17 | +| C02 | [#986](https://github.com/datum-cloud/galactic/issues/986): missing/family-mismatched next hop falls back silently | IPv4 private services are out of scope; normal IPv6 EdgeRoute path must have its own explicit next hop. Prove local/remote return delivery and negative missing-next-hop behavior. | [#978](https://github.com/datum-cloud/galactic/issues/978) / M12 | +| C03 | [#986](https://github.com/datum-cloud/galactic/issues/986): any other-link default suppresses VPC default | Potentially reachable through workload CNI, even with unprivileged provider. Test unicast, blackhole, unreachable and ECMP defaults on the workload; require a fix or an explicit supported-routing constraint. | [#974](https://github.com/datum-cloud/galactic/issues/974), [#978](https://github.com/datum-cloud/galactic/issues/978) / M12 | +| C04 | [#986](https://github.com/datum-cloud/galactic/issues/986): persistent ServiceRouteAuthorization bypasses deadline | Never used for connector publication. Assert no such authorization is created or required; provider lacks write permission. Record any separate workload-service control-plane trust boundary. | [#975](https://github.com/datum-cloud/galactic/issues/975), [#976](https://github.com/datum-cloud/galactic/issues/976) / M08, M16 | +| C05 | [#986](https://github.com/datum-cloud/galactic/issues/986): router restart with API down loses policy gate | Run on the selected workload path; closed policy may reduce availability, never isolation. Document recovery and prove traffic returns only after current policy/authority. | [#979](https://github.com/datum-cloud/galactic/issues/979), [#985](https://github.com/datum-cloud/galactic/issues/985) / M06, M08 | +| C06 | [#986](https://github.com/datum-cloud/galactic/issues/986): broken private-service alert runbook anchor | Outside connector publication; use exercised Edge/MASQUE diagnostics. If private-service alerts are part of the fixture, fix/link the applicable runbook separately. | [#983](https://github.com/datum-cloud/galactic/issues/983) / M15 | +| C07 | [#986](https://github.com/datum-cloud/galactic/issues/986): one missing veth disables policy globally | Workload churn is relevant: delete an unrelated pod while connectors carry traffic. No unrelated attachment/policy outage; route any discovered shared-path fix to its owner. | [#975](https://github.com/datum-cloud/galactic/issues/975), [#985](https://github.com/datum-cloud/galactic/issues/985) / M17 | +| C08 | [#986](https://github.com/datum-cloud/galactic/issues/986): cache-lag NotFound clears replacement attachment status | Recreate referenced objects under the same names; pin UIDs and verify replacement status/traffic survives delayed cleanup. | [#975](https://github.com/datum-cloud/galactic/issues/975), [#985](https://github.com/datum-cloud/galactic/issues/985) / M17 | + +## Review and completion + +Reviewers must explicitly examine M05 (a revoked connector beside a renewing neighbor), M09 (same-instance generation), M10 (partitioned owner still connected to clients/gVPC), M11 (prefix retention), M17 (late cleanup) and M18 (authority refresh), including HTTP/3-to-HTTP/2 transitions. Record the review link with the document revision; approving an implementation plan is not evidence that these design choices have been reviewed. + +[#972](https://github.com/datum-cloud/galactic/issues/972) is complete when this design and matrix are reviewed and their decisions accepted. [#973](https://github.com/datum-cloud/galactic/issues/973)/#974 feasibility failures revise the design before dependent implementation proceeds. [#985](https://github.com/datum-cloud/galactic/issues/985) cannot declare [#970](https://github.com/datum-cloud/galactic/issues/970) complete until every required result and finding disposition is present, all applicable foundation evidence is pinned, and capacity/clock assumptions match the tested platform. Keep [#986](https://github.com/datum-cloud/galactic/issues/986)'s unrelated legacy remediation open independently. diff --git a/docs/designs/masque/contract.md b/docs/designs/masque/contract.md new file mode 100644 index 00000000..95066e6b --- /dev/null +++ b/docs/designs/masque/contract.md @@ -0,0 +1,161 @@ +--- +status: provisional +stage: alpha +latest-milestone: "#972" +--- + +# MASQUE connector contract + +This is the implementation design for [#972](https://github.com/datum-cloud/galactic/issues/972), part of [MASQUE Connector for gVPC (#970)](https://github.com/datum-cloud/galactic/issues/970). It supersedes the [older gateway proposal](../masque-gateway.md) for this milestone. Decisions here are requirements for the follow-on issues, not claims that the service is implemented or qualified. The [acceptance matrix](acceptance.md) names their owners and evidence gates. + +The [Galactic Edge contract](../../edge/provider-contract.md) owns attachment admission, host networking, fencing, prefix arbitration, publication and gVPC policy. MASQUE owns credentials, connector allocations, session generations and transport. The service uses CONNECT-IP over HTTP/3 on UDP/443 and HTTP/2 over TLS on TCP/443. Both enter the same admission and packet adapter. Public APIs, portal/plugin integration, product-client changes and production rollout are outside this design. + +## Identity and resource mapping + +A connector is an authority-issued UUID bound immutably to a VPC UID and a cell. Names, user-selected VPC headers, credentials and IP addresses are not identities. Deleting and recreating a VPC cannot preserve the binding. Moving a connector to another VPC or cell requires a new identity and explicit allocation; there is no cross-cell claim store. + +**Use one EdgeService per connector authorization incarnation.** This deliberately gives each connector its own deadline and attachment, so another connector renewing the shared instance cannot keep it alive. A new authorization incarnation gets a new service name and UID. A refresh of the same credential revision within a running session may extend its current service only after trusted revalidation; credential rotation retires the old service incarnation and uses the release/recreate procedure below; a handoff to another provider pod uses a new service incarnation and the replacement procedure below. + +| Object | Identity and owner | Cardinality | +| -------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | +| EdgeProvider | Operator-approved provider name; record its UID in authority configuration; fixed ServiceAccounts and capability versions | One registered MASQUE provider per cell | +| Connector allocation | Authority-owned immutable record UID, connector UUID, VPC UID, cell, /96 and attachment pool | One retained record per connector | +| EdgeService | Authority-owned name derived from connector UUID and authorization incarnation; immutable VPC UID/provider; finite validUntil | One current grant per connector; retired grants are never renewed | +| EdgeInstance / Lease | provider.InstanceName(pod UID); Lease UID is its incarnation; new process holder ID on restart | One pair per provider pod, shared by that pod's connectors | +| EdgeAttachment | provider.AttachmentName(pod UID, service UID); pin object UID, service UID, VPC UID and instance | One per pod/service | +| EdgeRoute | Stable provider-wide hash of VPC UID and masked /96; pin route UID and accepted epoch | One contended aggregate route per delegated /96 | +| Local session | Process boot ID, connector UUID and monotonically increasing session generation | At most one forwarding session per connector | + +The route name follows the [reference provider](../../../examples/edge-provider/README.md#route-names). Never derive it from a session, service incarnation or transport: contenders must address the same object. The accepted route holder, its attachment UID and Lease incarnation/epoch must match before packet I/O starts. A successful claim write, Ready or a cached Accepted condition alone is insufficient. + +The service grants only its connector's /96 and `PacketAccess`/`PrefixReachability`. Its distinct attachment client pool is not delegated to the client. Core owns attachment address allocation in `EdgeService.status.allocations`; that is not the connector allocation database. Core alone writes BGPAdvertisements and the route index. MASQUE never accesses host maps, invokes hostattach, creates host VRFs, or operates a second route publisher or ownership lease store. + +## Authority and durable allocations + +The initial deployment uses existing Kubernetes Secret and ConfigMap kinds as internal storage, not new public CRDs. A separate **MASQUE grant authority** runs under an authority ServiceAccount. It validates operator-approved grants and writes EdgeServices. It has no provider token and no host access. The internet-facing provider cannot write EdgeServices, allocation records, grants or trust configuration. This authority is a MASQUE client-authorization component, not another attachment controller. + +The operator owns one internal grant record per connector: immutable connector UUID, VPC namespace/name/UID, cell, credential digest, credential expiry, enabled flag, allowed gVPC destination prefixes and policy references. The opaque bearer credential is 32 cryptographically random bytes, base64url encoded; only its SHA-256 digest is stored in the authority's restricted Secret. Compare digests in constant time. Credentials have a maximum 24-hour lifetime, no default unlimited lifetime, and are never written to logs or metrics. Rotation installs a new credential revision; an old credential cannot open a new session after rotation. Existing sessions are revalidated against the current revision and close. A request using the new credential gets a new service incarnation after retirement of the old one; it cannot extend the old session's ceiling. + +The provider sends credentials to this authority over an internal authenticated TLS channel. This is a private MASQUE interface delivered by [#977](https://github.com/datum-cloud/galactic/issues/977): `Authorize` takes the credential, requested VPC selector, authenticated pod UID and request nonce; `Refresh` takes the resulting authorization ID, credential revision and service UID; `Retire` conditionally ends that authorization ID. A successful response returns connector/allocation IDs, VPC UID, service namespace/name/UID, /96, allowed destinations, credential revision and absolute deadline. Invalid credentials return unauthenticated, wrong VPC or retired revision returns forbidden, and unavailable/stale authority returns retryable unavailable without a grant. Refresh never accepts a new credential revision implicitly. The authority validates every call against current records and binds authorization IDs to the calling pod; retries with the same nonce cannot create a second incarnation. Revocation notifications carry the authorization ID and revision and may accelerate closure, but delivery is not the safety backstop. The authority verifies the provider's pod-bound identity and uses uncached API reads of the grant and VPC; a caller-supplied VPC is only a selector that must match. Trust roots and server identity are operator-mounted, rotated with a bounded overlap of 24 hours; unknown or expired roots fail closed. TLS session resumption does not bypass a fresh admission decision. Neither the provider nor the client chooses its prefix, service, attachment identity or allowed destinations. + +The allocator holds one ConfigMap ledger per VPC/cell, at most 256 entries and 256 KiB. Entries contain connector UUID, allocation UUID, /96, distinct attachment /120 pool, creation time and retired state. Every insert or retirement uses the ledger UID and resourceVersion as preconditions. Concurrent allocations conflict and retry from a fresh read; a lost reply is resolved by reading the connector's allocation UUID. The ledger is authoritative even when no EdgeRoute exists. It is writable only by the authority; providers receive immutable allocation snapshots. A missing/recreated ledger is an operator recovery event: refuse allocation and restore from backup, never infer free space from absent routes. Reaching the ledger limit refuses new connectors; sharding is future design work. + +Retain an allocation until explicit operator retirement, across disconnects, credential rotation, expiration, pod restart and HTTP/2 fallback. Retirement revokes the grant and waits for the route to be released and its old gates fenced. For this milestone, retired /96s and attachment pools remain tombstoned and are **never reassigned** within that VPC/cell. This avoids an arbitrary time-based claim that delayed traffic has disappeared. Restore the ledger and grants together; rollback must not reuse allocation identities or credential revisions. + +### Address plan + +Follow the [gVPC tenant addressing plan](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/tenant.md): a /48 per ULA VPC, regional /64 space, and a /96 per endpoint. Within the cell's regional /64, the authority reserves a connector /80 and a separate attachment /80 through the platform's existing allocation authority. These are reserved address ranges, **not overlapping workload VPCAttachment podSubnet objects**: Edge rejects a delegated route overlapping a workload subnet. Neither the MASQUE provider nor this ledger allocates a VPC /48 or silently carves space out of an existing workload subnet. + +For example, for VPC `fd20:a1b:2c3d::/48`, reserve `fd20:a1b:2c3d:10:1::/80` for connector /96s and `fd20:a1b:2c3d:10:2::/80` for attachment pools within regional `fd20:a1b:2c3d:10::/64`. A connector can receive `fd20:a1b:2c3d:10:1:1::/96` and attachment pool `fd20:a1b:2c3d:10:2::100/120`. Its service grants the /96 and uses that /120 as clientPool. The pool lies outside every delegated /96, so no route contains an attachment's own address. Pools are distinct across connectors. A replacement reuses its retained pool only after the old service's attachments are released and the service is deleted; no two services hold it concurrently. + +Assign the /96 through CONNECT-IP address assignment. Clients may use its primary address or subdivide it locally without adding routes to core. Advertise only the aggregate /96. The client may not request routes beyond the authority's destination allowlist. Reject IPv4 for this milestone on both transports and in both packet directions; do not silently accept it into an IPv6-only return demultiplexer. + +## Client deadlines and revocation + +The authority renews each active connector's service every 5 s after a fresh grant read. It timestamps the **start** of that read and computes: + +```text +validUntil = min(credential expiry, + grant read-start wall time + 30 s - 2 s issuer clock allowance) +``` + +The response includes this absolute deadline, grant revision, service UID and allocation identity. A retry retains the original deadline; it cannot turn a delayed result into fresh authority. Requests have a 2 s API deadline and stale work is discarded. Service updates use UID/resourceVersion preconditions; retired authorization incarnations are never recreated or renewed by queued work. A new authority leader rereads all inputs before issuing anything. A provider never receives credentials capable of extending its own service. + +Operator revocation is a committed disabled/rotated grant revision. Stop refreshing that connector, shorten its service deadline to the past, and notify its active session. The provider closes its packet path promptly. If notification, API writes or a worker stall, the already-issued finite service ceiling is the backstop. Other connectors retain their own services and continue renewing the shared instance. + +| Event | Required bound | Assumptions and enforcement | +| ------------------------------------------------ | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | +| Instance renewal stops | 15.5 s holder-clock cutoff; 15.52 s real-time threshold | Foundation kernel gate, including API/provider/agent stalls | +| Connector credential expires | No packets after expiry + 1 s real time | Service ceiling no later than credential expiry; node subtracts MaxSkew; includes 500 ms foundation margin and drift allowance | +| Connector grant revoked or credential rotated | No packets after committed revision + 31 s real time | Issuer read-start TTL above; uncached linearizable read, issuer/node wall error at most 2 s, monotonic drift at most 0.001 | +| Revocation with all components connected | 2 s target | Authority notification and service revocation; measured, not a guarantee | +| Authority or its grant store becomes unavailable | No new admissions; existing grants expire within 31 s of their last successful read-start | Never refresh from cached grants; unrelated grants survive only while independently revalidated | +| Local route withdrawal | Within 1 s of gate expiry while router runs | Foundation map polling, independent of API | +| Router dies | Reflector withdrawal within 9 s of router loss | Negotiated hold time at most 9 s, configured on both ends | +| Remote convergence | 5 s target after local withdrawal | Measured separately; stale remote paths still meet a closed old gate | + +The 31 s connector bound is additional to, not a replacement for, the foundation's 15.52 s instance/revocation thresholds. A grant read that races revocation can issue at most its original 30 s window: subtracting the issuer's 2 s allowance prevents a fast issuer clock extending that window. Node ceiling conversion subtracts its own MaxSkew; 31 s leaves room for 500 ms margin and monotonic drift. Finite credential expiry is an absolute trusted timestamp, not a provider-supplied duration. Clock excursions outside these assumptions require admission to stop; no real-time guarantee is claimed for a VM whose clock stops during pause/live migration. Use the foundation's physical-node or no-pause deployment assumption. + +Never set the shared instance lease ceiling from one connector's credential: the per-service gate ceiling is what isolates its expiration from its neighbors. + +The existing implementation mechanism is `EdgeService.spec.validUntil` and [the agent's per-route ceiling](../../../internal/edge/agent/gate.go); `TestGateCeiling` covers the primitive. The **multiple connectors sharing a live instance** scenario remains a required integration check. No foundation extension is selected by this design. If refresh, service replacement or ceiling qualification cannot meet these requirements, [#976](https://github.com/datum-cloud/galactic/issues/976) must track a foundation change under [#992](https://github.com/datum-cloud/galactic/issues/992) and qualify it before integrated acceptance. Falling back to cancellation-only expiry or sharing one uncapped service is forbidden. + +## Interfaces and session transitions + +Use the existing [connectip interfaces](../../../internal/edge/connectip/backend.go): `Admission.Admit`, `Lease.Info/Routes/Done/Close`, and `PacketBackend.Open`. Their production adapter is follow-on work. `SessionInfo.ID` is the request generation, `Connector` is the connector UUID, `VPC` carries the immutable VPC identity, and `Epoch` is the Edge route epoch. Do not overload one epoch for both session and foundation authority. The adapter keeps the full attachment/service/route UIDs, instance incarnation/epoch and process boot ID beside this snapshot; string names alone cannot authorize I/O. + +1. **Pending:** reserve bounded pre-admission capacity before credential parsing or API work. Authenticate and authorize with the authority. No TUN write or MTU probe yet. +2. **Allocated:** read the retained /96 and finite service grant; reserve one connector session transition. A failed admission releases provisional request resources, not the durable allocation. +3. **Attached:** obtain the pod's shared instance lease through `internal/edge/provider`; request the derived attachment and verify its immutable references. Wait for Programmed, open only the device named in status, then set Ready. +4. **Claiming:** contend for the stable EdgeRoute. Wait for the accepted holder/epochs and observed open gate/local publication for that exact incarnation. Provider Ready is not permission to forward. +5. **Active:** install the current generation in the packet adapter, then accept CONNECT-IP and perform any authenticated MTU probe. Check authority, generation, IPv6 source ownership and destination allowance at every I/O boundary, including dequeue. Return traffic is scoped by attachment/VPC identity before destination /96 lookup. +6. **Closing:** mark generation closed, unblock and drain its I/O, discard its buffers, then conditionally release only the claim epoch it still owns. A stale Close is a no-op for its successor. Release an attachment only when no active/provisional session reference remains; release the instance only when the pod drains all its connectors. + +The packet adapter accepts only complete IPv6 packets within the device MTU, walks at most 8 extension headers and 256 extension-header bytes, and rejects fragments (including atomic fragments), malformed lengths and unevaluable policy with a counted reason. This milestone does not promise fragmented-inner-packet support. Required ICMPv6 errors and control traffic follow the foundation policy and CONNECT-IP rules; locally generated errors are associated with the same tenant/session and cannot bypass destination/source checks. [#979](https://github.com/datum-cloud/galactic/issues/979) tests this profile against the pinned library. + +Do not hold a shared lock across an API call, blocking packet write or queue wait. One renewal worker owns the pod's one instance lease; route acquisition/release cannot hold its mutex. Separate bounded control/renewal workers from transport workers. Request cancellation must cancel its own work without cancelling another connector's renewal. + +### Same-instance replacement + +A per-connector transition lock admits one replacement at a time; a concurrent loser receives a retryable conflict. Authenticate the replacement before disturbing the active session. Allocate a new process-local generation, mark the predecessor closed, prevent new writes, cancel and join its workers, discard its queued packets and flush the packet reader's pending delivery before activating the replacement. Packets already queued in userspace carry their arrival generation and cannot be relabeled. If an old worker cannot quiesce within 1 s, fail the replacement and revoke this connector's service; never activate a second writer beside it. A process restart uses a fresh boot ID and the foundation instance takeover rules. + +The same attachment and EdgeRoute can remain held during a successful same-process transition, but the I/O barrier must finish before the successor becomes Active. Cancellation, release and callbacks compare the entire generation/UID tuple. An old callback cannot delete the successor's registration, close its device or release its claim. Incoming packets read after the barrier belong to the current retained address; IP itself has no session generation, so this does not promise to identify an old packet delayed in the remote network. Transport sequence state and already-buffered application bytes are never transferred. + +HTTP/3-to-HTTP/2 fallback is a newly authenticated CONNECT-IP request using this exact procedure. It retains the /96 but not the QUIC connection or a lossless packet stream. Inner flows may survive if endpoints tolerate the interruption; session continuity is not guaranteed. + +### Cross-instance replacement and stale cleanup + +A replacement cannot force transfer from a live holder merely by writing a higher route epoch. Ask the authority to retire the old connector service, stop refreshing it, and drain/delete its attachments. Persist this transition in the connector's authority record with UID/resourceVersion preconditions so concurrent replacements choose one successor. Keep the allocation reserved throughout. + +Wait for core's fencing evidence and old attachment release, then delete the old service and create the successor service with the same allocation/pool and a new incarnation. Refresh only the successor. The new pod requests its own attachment and takes the **same** EdgeRoute through core's acceptance. In particular, do not refresh the old service to make a new claimant ready: that could reopen the old holder. The old instance may continue renewing for unrelated connectors; revoking its service closes only this connector's gates. + +If the old node cannot report Released, replacement waits. No bounded availability is promised for that case, even after packets are proven stopped. This is the deliberate cost of reusing the retained attachment pool without bypassing foundation cleanup. Core's cross-attachment acceptance still requires confirmed old closure or the 15.53 s TakeoverWait and its fencing rules; MASQUE cannot substitute a timer or its own lease. API partition with client/gVPC connectivity intact must produce packet cutoff, not dual forwarding. + +Late cleanup uses UID, route epoch and session generation preconditions and cannot remove a replacement's objects. After a restart, reconcile from authority/API state and reacquire; do not restore forwarding permission from a local cache. Fencing state stays in the foundation's node directory for the host lifetime, and MASQUE never deletes it. Router GC and CNI maintenance remain foundation responsibilities. + +## Deployment and test provisioning + +Request `v1alpha1`, `PacketAccess/v1` and `PrefixReachability/v1`. Reject unsupported capabilities before allocation. Use a pod-bound ServiceAccount token with the API audience and the foundation's admission policies; verify the Kubernetes token extras required by that contract. The provider gets no hostNetwork, hostPID, hostPath, NET_ADMIN or BPF privileges. A device plugin supplies `/dev/net/tun`; core creates the TUN in the pod namespace. Bindings grant only provider-owned resources/Ready, not service grants, accepted status, route indexes or BGPAdvertisements. + +The operator provisions the VPC, address reservations, policies in both directions, approved EdgeProvider, authority ledger/grant Secrets, and TLS trust. The authority's access to tenant namespaces is explicitly scoped by RoleBindings; registration alone grants no tenant authority. The provider receives service namespace/name/UID and VPC UID through the authenticated authority response, not broad tenant read access. Deny service/status and privilege-escalating writes in negative RBAC tests. + +For authorized test connectors, an operator tool delivered by [#977](https://github.com/datum-cloud/galactic/issues/977) creates the ledger entry and a restricted grant Secret, emits the random bearer once to the test runner, and records only its digest. It validates the selected VPC UID and reserved pools. The authority creates the finite EdgeService; the runner supplies the bearer and VPC selector to an independent CONNECT-IP client. Revocation disables that grant; cleanup waits for Released and asserts removal of test host links, gates and routes while retaining/tombstoning the allocation according to policy. No portal, public enrollment endpoint or provider-minted EdgeService is needed. + +Startup requires the foundation components, device plugin and an existing canonical host VRF (a workload of the VPC on each provider node, as the current contract requires). Until policy, attachment and exact route incarnation are ready, refuse traffic. Rollout first qualifies compatible core artifacts, then authority, then providers. SIGTERM stops admission, closes sessions and releases their route epochs and attachments, and finally releases the instance. Graceful drain has a 30 s budget; security deadlines always take precedence. Provider crash, core/node restart, API outage and delayed cleanup are tested separately. + +## Resource budgets and benchmark contract + +These are initial **qualification targets**, not measured capacity. [#984](https://github.com/datum-cloud/galactic/issues/984) must publish results for HTTP/3 and HTTP/2 separately and obtain review of any target change before closure. Both transports use identical session/foundation limits. + +| Resource | Initial limit or target | +| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Provider size | 4 dedicated CPU cores, 8 GiB memory allocation; provider RSS at most 1 GiB and average CPU at most 3.2 cores at target load | +| Active connectors | 32 per pod; 128 per cell across 4 pods; at most 256 retained allocations per VPC/cell | +| Replacement headroom | At most 32 additional closing/provisional sessions per pod; 256 total attachments and 8 instances during a cell roll | +| Foundation routes | At most 128 active connector routes per cell; refuse before the existing 1024-entry per-VPC index limit; reserve at least 25% measured map capacity | +| Service refresh work | 25.6 successful EdgeService writes/s at 128 active connectors, plus retries and status fanout; measure total core/API cost | +| Lease renewal work | 0.8 successful Lease writes/s for 4 instances, 1.6/s during an 8-instance roll; reserved independent workers | +| Authority/API concurrency | 8 in-flight operations per pod, 2 s per operation; aggregate authority client budget 150 requests/s, burst 100, excluding separately measured core fanout | +| Unauthenticated transport | 128 connections per pod; 8 per source IP; source-IP cap is an initial test-deployment policy that may constrain clients behind one NAT; 5 s handshake deadline; 8 KiB header limit; 2 CONNECT streams per connection | +| Admission | 16 in-flight per pod, 2 per credential digest, 3 s request deadline; connection and global limits apply before digest lookup | +| Session buffers | 256 KiB queued bytes and 128 packets per direction; whichever limit comes first; 32 MiB aggregate packet queues per pod | +| Capsule handling | 64 KiB maximum known capsule payload; incremental parsing; unknown capsules skipped with bounded scratch storage and a 5 s progress timeout | +| Slow device/client | Nonblocking I/O; EAGAIN is a counted packet drop; 5 s blocked-delivery timeout closes only that session | +| HTTP/3 throughput | At least 250 Mbit/s aggregate bidirectional inner-IP goodput per pod | +| HTTP/2 throughput | At least 100 Mbit/s aggregate bidirectional inner-IP goodput per pod | + +Benchmark on two physical edge nodes and a separate client/load host connected at 10 Gbit/s, with a local workload and a workload on a remote node. Pin OS/kernel, Talos/Cilium, CPU model, NIC/MTU, core/provider images, library and client versions. Run each transport with 1, 8 and 32 connectors per pod and 128 per cell; include two tenants deliberately sharing inner address values. Use 1280-byte inner IPv6 packets for throughput, a separate 64-byte packet-rate run, and a 50/50 byte split between UDP and TCP, evenly in both directions. Warm up 2 minutes and measure 10 minutes, three runs each. Report all runs, p50/p99 RTT, loss, CPU, RSS, queue peaks, API requests/status fanout and map occupancy. Throughput thresholds apply to the loss-free baseline; require less than 1% UDP loss and p99 added RTT at most 10 ms above the untunneled path. + +Repeat with 1% packet loss, 50 ms added RTT, 10 reconnects/s, stalled readers/devices, and offered load at twice every admission/session limit. Overload must reject bounded work, preserve expiry bounds, and keep existing unrelated sessions at at least 80% of their baseline goodput under admission floods. Queue/RSS caps continue to hold; queues and active session counts return to baseline within 30 s of load removal. Report process/core/API budgets separately; adding privileges or bypassing policy is not a performance optimization. + +## Alternatives and consequences + +A shared EdgeService per VPC would reduce attachment count and grant writes, but its deadline would cover every connector. Session cancellation alone cannot enforce individual expiry while a control worker is stalled. A new independently expiring per-route client grant could support shared attachments more efficiently, but would require a versioned foundation extension and fresh qualification under [#992](https://github.com/datum-cloud/galactic/issues/992). This design instead uses existing per-service ceilings, accepts the attachment/API overhead, and makes M05/M18 qualification gates explicit. + +One instance lease per connector would separate failure domains but conflicts with the foundation's pod-bound instance identity; inventing another lease store would not make core honor it. Immediate cross-node pool reuse before Released would improve availability but bypass allocation cleanup. Retaining tombstones consumes ledger space deliberately; exhaustion refuses admission rather than risking address reassignment. The initial 128-connector capacity target includes those costs and does not imply that the foundation's 1024-route ceiling is a supported MASQUE throughput level. + +## Protocol references and scope + +[CONNECT-IP (RFC 9484)](https://datatracker.ietf.org/doc/html/rfc9484) defines the tunnel, address assignment and route advertisements. [RFC 9297](https://datatracker.ietf.org/doc/html/rfc9297) defines HTTP Datagrams and capsules: the HTTP/2 fallback carries datagrams in capsules on a new request. HTTP/3 uses [RFC 9114](https://datatracker.ietf.org/doc/html/rfc9114), [Extended CONNECT, RFC 9220](https://datatracker.ietf.org/doc/html/rfc9220), [QUIC DATAGRAM, RFC 9221](https://datatracker.ietf.org/doc/html/rfc9221), and [QUIC transport, RFC 9000](https://datatracker.ietf.org/doc/html/rfc9000). HTTP/2 uses [RFC 9113](https://datatracker.ietf.org/doc/html/rfc9113) and [Extended CONNECT, RFC 8441](https://datatracker.ietf.org/doc/html/rfc8441). TLS follows [RFC 8446](https://datatracker.ietf.org/doc/html/rfc8446) and QUIC's binding in [RFC 9001](https://datatracker.ietf.org/doc/html/rfc9001). + +[#973](https://github.com/datum-cloud/galactic/issues/973) selects and proves libraries and independent clients; this contract does not assume HTTP/2 support in the current HTTP/3 server. [#982](https://github.com/datum-cloud/galactic/issues/982) must prove 1280-byte IPv6 packets in both directions and correct ICMPv6 Packet Too Big or explicit tunnel failure when the path cannot carry them. Packet-size claims include encapsulation overhead; an attachment MTU floor is not evidence of end-to-end MTU support.