Repository navigation
fix: Run the current egress datapath on existing VMs and pods - #776
Merged
Merged
Conversation
CNI ADD attaches usid_egress to an attachment's host-side veth or tap once, and the tc filter keeps its own reference to that program. When galactic-cni starts a new build, attach.Load pins a fresh usid_egress, but nothing moved existing attachments onto it. Every VM and pod kept running the previous build's egress program, bound to the previous copies of its maps. So a fix to usid_egress never reached a running workload until it was recreated. And when a map is recreated because its layout changed, as vip_xlat_table was in #768, existing attachments kept reading the old copy: rows galactic-router, a later CNI ADD or the egress refresh sweep wrote into the new map never reached them. In staging us-central-1 one VM was still on an older program after the 2026-10-07 rollout. attach.Start now calls ReattachEgress after attaching usid_ingress. It scans only veths and taps, the only links CNI ADD attaches to, finds the filter named galactic_usid_egress on each one's ingress hook, and replaces it in place with the new program, keeping its handle and priority so a changed priority setting cannot leave a duplicate. A filter already on the new program is skipped, and a link deleted during the scan, a pod or VM being torn down, is skipped rather than reported. A failure is logged, not fatal: an attachment left on the previous program still forwards, and failing startup would drop the node's whole ingress datapath. The new root test attaches a previous build's usid_egress to a veth and its usid_ingress to an uplink in a test namespace, loads a second build, and checks that only the egress filter moves to the new program and that a second pass changes nothing. Related to #774 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
privateip
force-pushed
the
fix/reattach-usid-egress
branch
from
October 7, 2026 02:55
e4fc179 to
2ccd4c6
Compare
privateip
marked this pull request as ready for review
October 7, 2026 03:07
privateip
enabled auto-merge
October 7, 2026 03:07
Collaborator
Author
|
First pass: the adversary reviewer returned merge, gated on CI including Unit Tests (root), with a staging check after merge. The conventions reviewer held only over the overlap with draft PR #634; the decision is that #776 lands first, and a comment on #634 asks it to build on #776 when it rebases. Second pass at 2ccd4c6: the pr-rereviewer returned merge (squash, subject length, docstring and stale body fixed). The conventions reviewer returned merge with two nits and no change needed. The reviewer reports' RAN: lists were not supplied to this phase, so they are not repeated here. No further findings were applied in this phase. CI is green on 2ccd4c6. Unit Tests (root) first failed on TestPMTU_VethPodLearnsPathMTU, a known flaky test this PR does not touch (it failed once on #753 as well), and passed on rerun of run 37564349761. TestReattachEgress_MovesAttachmentsToTheNewProgram passed in both attempts. Auto-merge is enabled with the merge method, the only one the ruleset allows. The ruleset requires one approving review, code owner review, and approval of the last push, so the PR waits for a human approval. |
ecv
approved these changes
Oct 7, 2026
privateip
added a commit
that referenced
this pull request
Oct 7, 2026
On a compute node CNI ADD is the only writer of each attachment's vrf_table, vpc_attribution_table, ifindex_vrf_table, ifindex_egress_kind_table, tenant_gw_table and egress_route_table rows, and of the node's own locator_table, function_table, node_src_addr_table and public_uplink_table rows. When a pinned map is recreated, by a layout change or a lost pin, its rows stay missing until every workload re-attaches. Recreating every map on 2026-10-07 left both staging compute nodes forwarding nothing until 44 VMs were recreated. galactic-cni now rebuilds every missing row itself, with gc.RepairAttachmentDatapath. It runs once right after the datapath loads, off the main loop and retried with backoff from 15s until a pass completes without error, and again on every GC tick before the vrf_table sweep. A size-1 semaphore keeps the two from overlapping. A pass finds each attachment in the kernel: an interface named like an attachment's host side and enslaved to a VRF. The VRF master gives the routing table, which also covers the legacy per-attachment VRFs, and the link type gives the egress kind. The guest prefixes come from the pod-subnet routes out that interface in the VRF's table, and the gateways from its own non-link-local addresses, which is what the master plugin installs from the IPAM result; a tap with no IPAM has neither and gets no tenant_gw_table row, as at ADD. The Block and Node-ID come from the node's BGPRouter and the Argument from the VPC's BGPVRFInstance, and the shard routes from the conflist's egress configuration the daemon already loads. Every value goes through the attachreg helpers ADD uses. A pass writes only rows that are missing. It skips an interface with no usable BGPVRFInstance, anything under the ingress sidecar's Block, and VRF tables the sidecar or the sidecar return path own. If an interface is deleted or re-enslaved while its rows are written, as a concurrent CNI DEL does, the pass removes the ifindex-keyed rows it just wrote. One summary line reports the rows rebuilt per map, at Info only when it rebuilt something. A public uplink whose neighbor has not resolved is reported but does not fail the pass. egress_route_table is a longest-prefix-match trie, so a lookup for a prefix returns a covering ::/0 entry and cannot say whether the prefix itself is present. EgressRouteTable.Prefixes walks the keys once per pass for an exact answer; Lookup's doc comment now says what it actually returns. The new root test caught this: it registers attachments through the ADD-side helpers, empties every map, and requires one pass to restore every row exactly. Re-attaching usid_egress is not part of this; #776 does that at startup. The repair also stays off the installer's main loop: the GC-tick pass runs in a goroutine behind the size-1 semaphore and is skipped while one is still running, and a shard that cannot be resolved is tried once per pass instead of once per VRF. The startup pass retries for up to 10 minutes while attachments wait on a BGPRouter or BGPVRFInstance. Rows written for a link that vanishes mid-pass are rolled back, including the local pass-through routes and the shared vrf_table and vpc_attribution_table rows no other attachment needs. Related to #774 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When galactic-cni restarts with a new build, every existing VM and pod keeps running the previous build's egress program, bound to the previous copies of its maps, until the workload is recreated.
Egress fixes never reach running workloads, and rows written to a recreated map never reach their egress path. In staging us-central-1 one VM is on an older program right now.
At each startup, galactic-cni now moves every existing attachment onto the egress program it loaded.
Test plan
Related to #774
🤖 Generated with Claude Code