[privateip]

Summary

A route added through an Edge attachment's pod link stayed in place and went unreported, so a provider pod could steer its VPC traffic past what the node checks.

The node now removes any IPv6 route through an attachment's links that it did not install, and any global address it did not assign on the links inside the pod, at its next pass at most a minute later. It reports one it cannot remove.

This also adds the last lab checks for MASQUE packet access: the fixture's refused requests and permissions, a pod with no packet device, how its host links are registered, and drift repair with a client connected.

Compatibility and rollback

Nodes running this release remove IPv6 routes through an attachment's host link, pod link or packet device that differ from the ones the node installs, including routes written as kernel routes, and global addresses it did not assign on the pod link and packet device. The node never created such routes or addresses, so only hand-added or foreign ones go. Revert the commits to roll back. Merging deploys the change to staging, and it reaches production only in a later release.

Test plan

• Lint, build, unit, real-kernel and end-to-end tests pass locally
• Every lab row this change adds passes, and the drift row fails against an agent without the fix
• CI passes

Three existing MASQUE lab rows fail on main without this change, so they are not covered here: maintenance during use, partial setup and recreation. The connector revocation row does not run, because recreation stops the suite first. Running the checks on the target platform moves to #985.

Fixes #974
Related to #989

[privateip]

Validation run locally. The lab run is at 9f0b72d, this branch on main at 74b938e with the address fix from review. The repository checks ran at 2ea8de3, and the rebase changed none of this branch's files.

Real-kernel tests in internal/edge/hostattach:

• TestPodLinkRouteDrift adds three routes through the pod link: a stray prefix, a second default at metric 1, and a stray prefix written as proto kernel. Readback reports each, and Provision removes it.
• TestAddressDrift adds a global address to the pod link and to the packet device. Readback reports each, and Provision removes the address and its connected route.
• TestPeerNamespaceOnlyImpostor offers a peer whose names and indexes all match the real one and whose namespace alone differs. Provision and Release refuse it and change nothing.
• Each of the five conditions these tests rely on was removed in turn, and the matching test failed each time: the pod-link prune and readback, the gateway and metric match, the connected-route rule for kernel routes, the address prune and readback, and the peer-namespace check. The full package passes as root.

Lab run, task verify:masque-conformance on the containerlab lab's dfw cluster. The lab was rebuilt from #1140's merge, and galactic-edge and the fixture were loaded from this branch.

Row	Result
A16 before the fix	The stray route was still there after 150 s
A16 with the fix	Removed after 27 s while a connected client answered 4500 of 4500 probes, with a 23 ms largest gap
M16 refused requests	An attachment for another pod and PacketAccess without a device owner were refused by the API. An unknown service, contract version and capability version were refused by core (GrantRevoked, UnsupportedContractVersion, UnsupportedCapability). EdgeService, BGPAdvertisement, route index, route status and attachment host link writes were denied. The token carries the pod UID and node, and the only binding is core's galactic-edge-provider RoleBinding
M16 missing device	The attachment reported DataPlaneFailed, the requests were refused and no route was claimed
A06	Both host links are veths, registered for delivery as kind 0
M07, NAT egress, shared attachment, process restart, same-pod replacement, provider stall, agent stall, API partition	Passed. In the API partition the gate closed 15.0 s after the provider's last renewal, inside the 15.52 s bound

maintenance-during-use, partial-setup and recreation fail here and on main at d2ef1ad, with main's agent, fixture and tests. On main, maintenance-during-use also panics on a missing attachment and ends the run. The provider adapter deletes attachments idle for 30 s, and this PR's inUse helper reconnects both tenants before a row reads its attachments, which removes that row's panic. recreation still panics in waitReaped, here and on main, so connector-revocation does not run.

Repository checks at 2ea8de3: task build, task test:unit and task test:e2e pass. golangci-lint reports 0 issues. yamlfmt passes with the repository config minus its .git/** exclude, because .git is a file in a worktree checkout.

Not run: the target-platform rows, which move to #985.

[privateip]

Review record for head 6a82abf. Both review passes returned merge, and the fixes from the first pass were confirmed on the second.

First pass at 2ea8de3: the adversarial review returned merge. It noted that it could not run the real-kernel tests or the lab, and that the contract said "every unicast route" where the check is IPv6-only. The conventions review held on three commit subjects over 72 characters, and noted the overlap with #1162, the staging and release path, and a narrating test comment.

Fixes at fe80f02: the three subjects were reworded, the contract now says IPv6, the narrating comment was removed, and the body states the staging and release path.

Second pass at fe80f02: the re-review returned merge and confirmed the fixes against the archived lab runs. The conventions review returned merge.

Since then the branch was rebased onto main with #1162 and #1171, with the commit subjects unchanged. One further commit names nsenter once, to fix a goconst lint failure the rebase caused. A narrow re-review of that delta returned merge.

Checks on 6a82abf: CI's root unit test job ran the new real-kernel tests and passed. A full lab run on the rebased branch passed every row this PR adds. Three rows that fail on main still fail, as the body says. Build, Documentation, E2E Tests, Lint, Manifests, Service API bundle, Unit Tests, Unit Tests (root) and the publish jobs are green. publish-galactic-debug-image and publish-service-api were skipped.

Reviewers: none requested beyond the CODEOWNERS team, since the repository sets no prReview.humanReviewer. Auto-merge is enabled with the merge method, and the main ruleset requires one approval with code owner review, so it waits for a human approval.

[ecv]

Request changes.

The stray-route removal is exempt for any "proto kernel" route whose prefix matches an address on the link, and the node never prunes addresses on the pod link, so a provider pod can still steer traffic past the node with one extra ip addr add.

• internal/edge/hostattach/inventory.go extraRoutes/connectedPrefixes: the connected-route exemption is built from whatever IPv6 addresses the link carries now, not from the addresses the node gave it (fe80::2 and the attachment address). The pod owns its namespace, so it adds, for example, fd00:dead::1/64 to the pod link. The kernel adds the connected route fd00:dead::/64 (proto kernel) in the pod VRF table. That prefix is in connected, so extraRoutes skips it and Readback stays green. A /64 or /128 connected route is more specific than a tunRoutes prefix and wins over it. Nothing in configurePodLink/ensureAddr removes the address, and checkAddr only checks that the wanted ones are present. Fix: build connected from the node's own addresses only, or prune and report extra addresses on the pod link in the same pass. This is from reading the code, not a run. TestPodLinkRouteDrift covers a stray prefix, a second default and a forged proto kernel route, but not an added address.
• Removal is only for routes that match RT_FILTER_TABLE|RT_FILTER_OIF in the one table the node owns (host table or PodTable), IPv6 only. IPv4 routes, routes in the pod's main or other tables, ip rule entries, and multipath routes whose LinkIndex is 0 are not looked at. The doc says "IPv6 unicast route", so this is stated scope, but it is narrower than the PR title.
• Window: polling only. agent.Resync is one minute and there is no netlink route subscription, so a stray route can carry traffic for up to a minute before the next pass removes it. A route added after pruneRoutes in the same pass is caught by Readback, which fails the attachment out of Programmed rather than removing it.
• No race with the node's own routes: pruneRoutes runs before the RouteReplace calls, and matches accepts metric 0 or 1024 with the same destination and gateway, so a just-installed route is not seen as stray. The kernel's fe80::/64, local-table and ff00::/8 routes are skipped by route type or by the address check, so they are not flapped. RA routes are not an issue while the pod forwards (podForwarding=1).
• Route listing and the address listing go through nldump.On(h) (the #1132 retry wrapper), not single-attempt netlink calls.
• Reporting: an unremovable route returns an error naming the link, and Readback adds unexpected route <dst> via <gw> metric <n>. agent.go reports that as a failure with diag.StageProgramming and a condition, and it clears when Readback next passes. No metric is added.

Follow-ups:

• Extend the sweep to IPv4, to every table the pod link can carry routes in, to multipath routes, and to ip rule steering through the VRF.
• A route event subscription (or a shorter pass for the pod link) would shrink the one-minute window.
• Add a drift test for an extra address on the pod link once the exemption is fixed.

[privateip]

Thanks. Taking the points in order.

Connected-route exemption. Fixed in 9f0b72d, both ways you suggested:

• The kernel routes the node leaves on a link are now only fe80::/64 and the connected routes of the addresses it assigns itself: fe80::2 and the attachment address on the pod link, and none beyond link-local on the host link or the packet device. A connected route for an address the node did not assign is removed like any stray route.
• Each pass also removes every global address the node did not assign from the pod link and the packet device, and Readback reports any it finds.
• TestAddressDrift adds a global address to the pod link and to the packet device. Readback reports each address, and Provision removes the address and its route. The test fails with the address check removed.

Scope. IPv4, tables other than the one the node owns, ip rule entries and multipath routes stay outside this repair. The contract sentence now names those limits, and I narrowed the title to IPv6 routes and addresses.

Window. The repair runs on the one-minute resync, and the contract says so. A route or address added between passes, or after the prune in the same pass, fails Readback, and the attachment leaves Programmed until the next pass removes it. Shrinking the window with a route event subscription is a follow-up, along with the IPv4, other-table, rule and multipath sweeps.

No change for the points on ordering, nldump and reporting.

I'll post the lab and test results for the new head here once they finish.

[privateip]

Results for the fix. cb5bfc2 differs from the tested 9f0b72d only in the comment above the pod link's prune, which now covers addresses too.

• TestAddressDrift, TestPodLinkRouteDrift and the rest of the hostattach suite pass as root. With the address check stubbed out, TestAddressDrift fails on both links. Readback still reports the connected route as an unexpected route, so the route exemption alone already closes the bypass you described.
• Full lab run at 9f0b72d with this agent: every row this PR adds passes. The drift row removed the stray route after 27 s with 4500 of 4500 replies. The stall and partition rows cut forwarding 15.0 s after the last renewal. maintenance-during-use, partial-setup and recreation fail here and on main, as the PR body says. No lab row adds an address, so the unit test is the only coverage for that case.

[ecv]

Approve.

The hold is closed. The connected-route exemption is now a fixed list (fe80::/64 plus the node's own address) passed in by each caller, no longer read from the link's current addresses, and extra global addresses are pruned and reported, so ip addr add fd00:dead::1/64 on the pod link no longer buys a skipped route.

• Host link: exemption is fe80::/64 only, and the node assigns only fe80::1 there. Pod link: fe80::/64 plus a.Address, a /128 that carries no prefix route. Packet device: fe80::/64 only, and the node assigns no address to it, so pruneAddrs(h, tun, nil) is correct.
• pruneAddrs runs after the node adds fe80::2 and a.Address and leaves link-local and own addresses alone, so it cannot remove the node's own addresses. Local-table and ff00::/8 routes are still skipped by the RTN_UNICAST filter, and the fe80::/64 route is still exempt.
• TestAddressDrift adds a global address to the pod link and to the packet device, expects ErrDrift with "unexpected address", then repairs and re-reads clean. The old code built the exemption from the link's addresses and had no such message, so the test fails there. It is a root test, and the root CI job runs it.

Follow-ups:

• extraAddrs skips all of fe80::/10, not just the node's fe80::2. A provider adding fe80:1::1/64 keeps the address while its connected route is pruned. Narrow the skip to the node's link-local addresses.
• The host link has no pruneAddrs/checkExtraAddrs. Only the pod namespace is provider-reachable today, so this is a hardening gap.
• Earlier follow-ups (IPv4, other tables, ip rule, polling window) stand.