Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/designs/masque/acceptance.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Packet cases use sequence-numbered traffic in both directions, capture at sender

[#985](https://github.com/datum-cloud/galactic/issues/985) packages setup, fault injection, collection and cleanup commands in a clean-checkout fixture. It asserts removal of test-created links, routes, maps, namespaces and API resources and preservation of unrelated resources. The ledger's retained/tombstoned allocations are expected state, identified explicitly rather than called a leak. Cleanup errors fail the run. Tests never deploy to production as part of this milestone.

`task verify:masque-conformance`, run from `deploy/containerlab`, runs the [#974](https://github.com/datum-cloud/galactic/issues/974) packet-access checks on the lab's dfw cluster. It covers M16 and M07, NAT egress from the attachment node, pod recreation, a partly set-up instance killed without shutdown, and asserted teardown. The provider is the test-only fixture in `tests/edgeconformance/masqueprovider`. It holds an instance lease, `PacketAccess` attachments and per-/96 routes through the provider contract, carries packets with `internal/edge/connectip`, and admits static test credentials in place of the [#977](https://github.com/datum-cloud/galactic/issues/977) authority. A lab result does not satisfy a row that requires the target platform.
`task verify:masque-conformance`, run from `deploy/containerlab`, runs the [#974](https://github.com/datum-cloud/galactic/issues/974) packet-access checks on the lab's dfw cluster. It covers M16 and M07, NAT egress from the attachment node, two connectors sharing one attachment while one request ends, router garbage collection while connectors forward, a provider process killed and restarted in place, pod recreation, a partly set-up instance killed without shutdown, and asserted teardown. The provider is the test-only fixture in `tests/edgeconformance/masqueprovider`. It runs the [MASQUE provider adapter](../../edge/masque-provider.md) with `internal/edge/connectip`, and admits static test credentials in place of the [#977](https://github.com/datum-cloud/galactic/issues/977) authority. A lab result does not satisfy a row that requires the target platform.

## Epic and implementation checks

Expand Down
2 changes: 1 addition & 1 deletion docs/designs/masque/contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ The existing implementation mechanism is `EdgeService.spec.validUntil` and [the

## Interfaces and session transitions

Use the existing [connectip interfaces](../../../internal/edge/connectip/backend.go): `Admission.Admit`, `Lease.Info/Routes/Done/Close`, and `PacketBackend.Open`. Their production adapter is follow-on work. `SessionInfo.ID` is the request generation, `Connector` is the connector UUID, `VPC` carries the immutable VPC identity, and `Epoch` is the Edge route epoch. Do not overload one epoch for both session and foundation authority. The adapter keeps the full attachment/service/route UIDs, instance incarnation/epoch and process boot ID beside this snapshot; string names alone cannot authorize I/O.
Use the existing [connectip interfaces](../../../internal/edge/connectip/backend.go): `Admission.Admit`, `Lease.Info/Routes/Done/Close`, and `PacketBackend.Open`. [`internal/edge/masque`](../../edge/masque-provider.md) implements them over the provider contract. `SessionInfo.ID` is the request generation, `Connector` is the connector UUID, `VPC` carries the immutable VPC identity, and `Epoch` is the Edge route epoch. Do not overload one epoch for both session and foundation authority. The adapter keeps the full attachment/service/route UIDs, instance incarnation/epoch and process boot ID beside this snapshot; string names alone cannot authorize I/O.

1. **Pending:** reserve bounded pre-admission capacity before credential parsing or API work. Authenticate and authorize with the authority. No TUN write or MTU probe yet.
2. **Allocated:** read the retained /96 and finite service grant; reserve one connector session transition. A failed admission releases provisional request resources, not the durable allocation.
Expand Down
1 change: 1 addition & 0 deletions docs/edge/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ Galactic Edge gives providers authorized attachments to tenant VPCs and controls

- [Provider contract](provider-contract.md): resources, writers, lifecycle, timing assumptions and required failure evidence.
- [Troubleshooting](troubleshooting.md): conditions, Events, metrics and locating the current prefix holder.
- [MASQUE provider adapter](masque-provider.md): setup, readiness, drain and cleanup of the CONNECT-IP provider's attachments and routes.
- [CONNECT-IP interoperability](connectip-interop.md): HTTP/3 and HTTP/2 protocol proof with an independent client, selections and remaining HTTP/2 listener work.
- [Foundation metrics](../observability/edge.md): lifecycle and forwarding-gate measurements.
- [Design record](../designs/galactic-edge.md): motivation and design alternatives.
Expand Down
53 changes: 53 additions & 0 deletions docs/edge/masque-provider.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# MASQUE provider adapter

`internal/edge/masque` connects a CONNECT-IP server (`internal/edge/connectip`) to a tenant VPC through the [provider contract](provider-contract.md). It implements `connectip.Admission` and `connectip.PacketBackend`, so every transport uses the same attachment, route and packet checks. It has no command of its own: the service binary that embeds it, and the grant authority that issues its grants, are separate deliverables of the [MASQUE connector design](../designs/masque/contract.md). The conformance fixture in `tests/edgeconformance/masqueprovider` runs it with static test credentials.

The adapter never touches host networking. galactic-edge programs and releases each attachment's host state, core accepts or refuses each route, and the adapter only writes what the contract assigns to a provider: its instance and Lease, its attachments and their `Ready` condition, and its route claims.

## Setup

The cell must run galactic-controller with `--edge` and the galactic-edge DaemonSet on the provider's node ([deployment](provider-contract.md)). The provider's node also needs a TUN device plugin, and the VPC needs a workload on that node, because an attachment joins the VPC's existing host VRF.

1. Register the provider: an `EdgeProvider` that lists contract version `v1alpha1`, `PacketAccess/v1` and `PrefixReachability/v1`, and names the provider's ServiceAccount. An operator approves it (`status.approved`). galactic-controller then binds the provider role and admission policy to that ServiceAccount.
2. Grant each connector an `EdgeService` in its VPC's namespace. It grants `PacketAccess` and `PrefixReachability`, lists the connector's /96 in `prefixes`, and sets a `clientPool` outside every connector /96. A service that lacks `PrefixReachability` gets an attachment, but core refuses every route on it.
3. Run the provider pod as that ServiceAccount, with a pod-bound token, the downward-API pod name, namespace, UID and node name, a device-plugin TUN, and no host namespace, hostPath, `NET_ADMIN` or BPF access. `tests/edgeconformance/masqueprovider/deploy/` is a worked example.

The adapter's `Authorizer` turns a request's credential and VPC selector into a `Grant`: connector ID, VPC UID and base62 identifier, the `EdgeService` reference, the connector /96, the routes to advertise and an absolute deadline. A grant the adapter cannot use (wrong VPC, no service, not a masked IPv6 /96, already expired) is refused.

## Readiness

The adapter registers its instance at start and admits nothing until it holds the instance's Lease. It prepares a connector on the connector's first request:

1. It requests the pod's attachment to the grant's service (`ea-<pod UID>-<service UID>`), requesting `PacketAccess` only, and verifies the stored object names this pod, instance, provider and service.
2. It waits until core admits the attachment for the grant's VPC on this node and galactic-edge reports it `Programmed` with a packet device. It then opens that device and sets `Ready=True` with reason `Serving`.
3. It claims the connector's route (`er-` and a hash of the VPC UID and /96) and waits until core accepts that claim for this attachment's UID and this Lease's incarnation and epoch.

Only then does the request get its packet path. The request waits up to 30 seconds for these steps (`Config.Prepare`), including any wait behind another request preparing the same attachment or route, and is answered 503 if they do not complete. It fails at once, also with 503, when core refuses the attachment, the attachment is draining, its status names another VPC or node or no packet device, the device cannot be opened, or core refuses the route outright (for example `CapabilityNotGranted`, `PrefixNotGranted` or `PrefixConflict`). On a refusal after the device is known, the attachment reports `Ready=False` with reason `DataPlaneFailed`.

Connectors of one service share its attachment and device. A connector has at most one open request: another request for it is answered 503 before any preparation, and the open request is left in place.

## Forwarding

A request forwards a packet, in either direction, only while its instance Lease is held at the epoch it was admitted under and before the Lease's deadline, before its grant's deadline, and while the request is open. Every second the adapter also checks each attachment and route it uses. It ends the requests on an attachment that is deleted, replaced, draining, no longer programmed or reprogrammed, and the requests on a route whose claim core no longer accepts, whose gate the node closed, or whose claim moved to another attachment. Losing the lease ends every request. A full device drops the packet and counts it instead of ending the request. These checks are in addition to the foundation's: galactic-edge closes a route's gate at the Lease deadline even if the provider stops.

## Drain and shutdown

When core drains an attachment, after its service is deleted or its authority ends, the adapter ends that attachment's requests at its next check and serves no new request on it. Core deletes the attachment after the service's `drainSeconds`.

On shutdown, stop the transports first so every request ends, then call `Shutdown`. It stops admitting, releases each route claim the adapter still holds, deletes its attachments and releases the instance Lease. A replacement can therefore take nothing over while this pod still forwards. Shutdown is bounded by the caller's context; the fixture gives it 20 seconds.

## Cleanup

- When a connector's last request ends, the adapter releases that connector's route claim. Core deletes the route once its attachments are gone.
- An attachment with no request for 30 seconds (`Config.Idle`) is deleted. A reconnect within that time, such as an HTTP/3 to HTTP/2 fallback, reuses it.
- Every delete carries the attachment's UID as a precondition, so a late delete never removes a replacement under the same name.
- Core holds a deleted attachment with its finalizer until the node reports `Released`. A new request for the same service waits for that before requesting a new attachment.
- A process that restarts in the same pod adopts the pod's attachments and reports them `Ready=False` until it serves them again on request. It deletes the unused ones once idle. It restores no forwarding permission from the previous process; each route is claimed again under the Lease it acquires.
- When the pod is deleted without shutdown, core reaps its instance, Lease, attachments and routes ([contract](provider-contract.md)).

`kubectl get edgeattachments,edgeroutes,leases -n <provider namespace>` shows what remains, and [troubleshooting](troubleshooting.md) explains each condition.

## Verification

- `go test ./internal/edge/masque/` runs the adapter's unit tests and, with `KUBEBUILDER_ASSETS` set, `TestAgainstCore`. That test runs the adapter as a provider pod's pod-bound identity against core's attachment and route controllers and the shipped RBAC and admission policies.
- `task verify:masque-conformance`, run from `deploy/containerlab`, runs the fixture in the lab: packet access, isolation, NAT egress, shared-attachment cancellation, maintenance, process restart, partial setup, recreation and teardown. A lab result is not target-platform qualification.
19 changes: 18 additions & 1 deletion tests/edgeconformance/fixtures_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,23 @@ func (l *lab) vpcUID(t *testing.T, tn tenant) string {
return string(v.UID)
}

// appLabel selects a tenant's edge-client pods.
const appLabel = "app"

// vpcID returns the base62 identifier of tn's VPC, as core reports it on
// an admitted attachment.
func (l *lab) vpcID(t *testing.T, tn tenant) string {
t.Helper()
var v cloudv1alpha1.VPC
if err := l.c.Get(context.Background(), client.ObjectKey{Namespace: tn.vpcNamespace, Name: tn.vpc}, &v); err != nil {
t.Fatal(err)
}
if v.Status.VPC == "" {
t.Fatalf("VPC %s/%s has no identifier yet", tn.vpcNamespace, tn.vpc)
}
return v.Status.VPC
}

// grantService creates or replaces the platform authority's grant of tn's
// VPC to tn's provider, and returns it as stored.
func (l *lab) grantService(t *testing.T, tn tenant, name string, mutate func(*galacticv1alpha1.EdgeService),
Expand Down Expand Up @@ -396,7 +413,7 @@ func (l *lab) clientPID(t *testing.T, tn tenant, node string) int {
i := map[string]string{computeNode: "1", edgeNode1: "2", edgeNode2: "3"}[node]
var pods corev1.PodList
if err := l.c.List(context.Background(), &pods, client.InNamespace(tn.vpcNamespace),
client.MatchingLabels{"app": "edge-client-" + i}); err != nil {
client.MatchingLabels{appLabel: "edge-client-" + i}); err != nil {
t.Fatal(err)
}
for _, p := range pods.Items {
Expand Down
Loading
Loading