Repository navigation
feat(edge): Let a MASQUE connector reconnect and refresh its grant - #1162
Conversation
The MASQUE adapter refused a connector's second request while its first was open, so a reconnect or an HTTP/3 to HTTP/2 fallback failed until the old stream closed. A grant's deadline was also fixed at admission. The connector design caps a grant at 30 s and expects the authority to revalidate it, so every request would end at that ceiling, and nothing could end one connector's request early. A connector's new request now replaces the open one: 1. The new request is authorized and prepared while the old one keeps forwarding. For the same service, the attachment and route claim stay held, so the claim epoch does not change. 2. The old request ends, and the adapter waits up to 1 s for it to close. Closing stops its packet I/O and discards its device registration and queued packets. The wait has its own bound, not the request's preparation budget. 3. Only then is the new request registered. If the old request does not close in time, the new one is refused and the connector's grant is retired, so neither forwards. A request arriving while another of the same connector is being admitted is refused as unavailable. A replaced request's late Close drops only its own references. The adapter now closes a request's packet registration itself on Close, and a sentinel stops Open from registering one after that. Authorizer gains Refresh and Retire. Each open request refreshes its grant every 5 s with a 2 s deadline: - A refreshed grant with the same connector, VPC, service, /96 and routes moves the deadline. - An unauthenticated or forbidden answer ends only that connector's request. - A changed grant ends the request. - Any other failure keeps the old deadline. Adapter.Revoke ends a connector's request at once and refuses its admissions already under way. A refreshed grant must also pass the admission checks and keep the same principal. The foundation's per-service validUntil gate stays the backstop, so other connectors renewing the shared instance cannot extend a revoked one. The conformance fixture's authorizer implements Refresh and Retire over its static connectors. The connectip package doc now names the adapter as the wrapper that fences TUNBackend.
TestMASQUE had no rows for connector replacement, for a stalled or partitioned provider, or for revoking one connector while others share the instance. These rows add them, each with thresholds fixed in advance: - same-pod-replacement: a connector's second request replaces its first. The replaced request's last reply comes before the replacement's first, the handoff takes at most 3 s, and the route keeps its UID and claim epoch. - provider-stall, agent-stall and api-partition: the fixture is stopped, galactic-edge is stopped, or the fixture pod's traffic to the API server is dropped while a request holds the route. A capture on the fixture's packet device shows the gate open after the last renewal start, closing within 15.52 s of it but no earlier than the 15 s lease allows, and staying closed until the fault clears. The holder router's GateClosed transition on the route's advertisement gives local withdrawal as a separate measurement (bound 1 s, plus 1 s of API time resolution). The partition row also requires the client's request to end within the same bound. - connector-revocation: VPC 20's service gets a validUntil while VPC 10's connector keeps the shared instance renewing. VPC 20 stops forwarding by validUntil plus 1 s. VPC 10 keeps at least 90% of its replies with no gap over 1 s, and the instance keeps renewing at the same epoch. Remote convergence is not measured. The compute node's tenant routes are pinned eBPF map entries, and the lab has no reader for them.
|
Both first-pass reviews returned merge on head dbe8191. pr-conventions-reviewer: "VERDICT: merge. Nothing here ships on reconcile. The PR changes only the MASQUE adapter, its tests and docs, nothing under config/ changes, and no open PR touches the same files." pr-adversary: "VERDICT: merge. No correctness defect in the replace, refresh and retire logic. The only open item is the unrun lab rows, which the author disclosed." The five new lab rows in tests/edgeconformance/masque_ownership_test.go have not run yet. The lab could not be run during review, so this merge rests on unit and envtest evidence. The user chose to mark the PR ready on that basis. No findings were applied. The nits were left as they are: the commit subjects are within the 72-character limit, and rewriting pushed commits is not worth it. The title is kept. Adapter.Revoke is kept as the API the grant authority in #977 will call. CI on the head: Build, Lint, Unit Tests, Unit Tests (root), E2E Tests and the publish jobs all passed. publish-service-api was skipped by its workflow condition. Auto-merge is enabled with a merge commit, the only method the base branch ruleset allows. The ruleset requires one code-owner approval and dismisses approvals on push, so auto-merge waits for a human approval. |
ecv
left a comment
There was a problem hiding this comment.
Approve.
Replacement is ordered correctly: replace() ends the old request and blocks on s.closed, which is closed only after the packet registration is closed and the route and attachment references dropped, and only then does newSession register the new one. A stalled old request fails the replacement after 1s and retires the grant, so there is no window with two forwarding requests. It fails closed.
- Grant refresh cannot cross VPCs or change authority:
sameAuthorityrequires the same connector, principal, VPC, VPC UID and ID, service, /96 and routes, andvalidateis re-run against the original VPC. Only the deadline moves. - Revoke is per connector, via
revokedAt/revokeSeq, and also refuses an admission that began before the revoke (TestRevokeDuringAdmission). A refresh result arriving after the request ended cannot rearm the timer (setDeadlinechecksended), andforwarding()checksendedfirst. - Closes from #1151: one-request-per-connector blocking reconnects (replacement), and no Refresh/Retire (both added to
Authorizer). The route claim stays at the same epoch across the handoff. - Retire is only called on the stalled-replacement path, and nothing in the adapter retires a grant on normal teardown, so teardown relies on the authority's own expiry.
Adapter.Revokehas no non-test caller in this diff, so nothing triggers revocation yet. The revocation bound is still the authority's NotAfter plus the 5s refresh interval.
Follow-ups:
- Still open from #1151: the adapter trusts
Grant.NotAfterwith no cap (for example now plus a maximum grant lifetime) and no MaxSkew margin. A refresh returning a distant NotAfter is accepted as long asvalidatepasses, so the 30s ceiling lives only in the authorizer implementation, which is not in this repo. - Still open from #1151: two connectors with one /96 are only handled by the existing claim-moving behavior (
TestClaimMovingAttachmentEndsRequests), and this PR adds no check that refuses the second. - A refresh failing for a whole partition keeps the request alive to its last NotAfter. The new lab rows that measure this have not run (PR body), so the bound is unverified.
replacewaits for the transport to callCloseafterend. Document that contract onconnectip.Leasesince a transport that never closes turns every reconnect into a 1s failure plus a retired grant.
Summary
The MASQUE provider refused a connector's second request while its first was open, and it could not refresh or revoke one connector's grant. A reconnect or an HTTP/2 fallback failed until the old stream closed, and every request would have ended at the grant's 30 second ceiling.
A new request now replaces the open one only after the old one stops, and the route claim is held at the same epoch throughout.
Open grants are refreshed every 5 seconds, and revoking one connector ends only its request.
Compatibility
The provider's authorizer interface gains a refresh call and a retire call. The test fixture implements both, and nothing else implements the interface yet. No core component, CRD or manifest changes, so rollback is a revert.
Not covered
Test plan
Related to #976