Repository navigation
fix: Rebuild lost attachment datapath rows without recreating workloads - #777
Merged
Merged
Conversation
CNI ADD is the only writer of an attachment's uSID datapath rows, and of this node's own locator, function, node source and public uplink rows. Rebuilding them after a map is recreated has to happen in the galactic-cni daemon, and it has to write exactly what ADD writes. internal/gc cannot import internal/cnibgp, which imports it, so the registration code moves to a new internal/plumbing/ebpf/attachreg package that both can use. The move keeps ADD's behavior. registerEBPFDatapath and registerTenantGateway stay in cnibgp as thin wrappers that pass the router's identity, the conflist's egress shard configuration and the IPAM gateways to attachreg.RegisterDatapath and attachreg.RegisterTenantGateway. The helpers they call move with them, and so do their tests. Two small changes make the moved code usable against an already-open map. srv6.EgressPrefixRouteAddTo runs the existing shard-selection loop against a table the caller opened, and EgressPrefixRouteAdd now delegates to it. The shard routes ADD installs are written to the pin directory ADD was given rather than srv6's package default; both are attach.PinDir in production. The ::/0 route and the NAT64 routes now go through one function that can skip a prefix already present, which the repair needs. Related to #774 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a compute node CNI ADD is the only writer of each attachment's vrf_table, vpc_attribution_table, ifindex_vrf_table, ifindex_egress_kind_table, tenant_gw_table and egress_route_table rows, and of the node's own locator_table, function_table, node_src_addr_table and public_uplink_table rows. When a pinned map is recreated, by a layout change or a lost pin, its rows stay missing until every workload re-attaches. Recreating every map on 2026-10-07 left both staging compute nodes forwarding nothing until 44 VMs were recreated. galactic-cni now rebuilds every missing row itself, with gc.RepairAttachmentDatapath. It runs once right after the datapath loads, off the main loop and retried with backoff from 15s until a pass completes without error, and again on every GC tick before the vrf_table sweep. A size-1 semaphore keeps the two from overlapping. A pass finds each attachment in the kernel: an interface named like an attachment's host side and enslaved to a VRF. The VRF master gives the routing table, which also covers the legacy per-attachment VRFs, and the link type gives the egress kind. The guest prefixes come from the pod-subnet routes out that interface in the VRF's table, and the gateways from its own non-link-local addresses, which is what the master plugin installs from the IPAM result; a tap with no IPAM has neither and gets no tenant_gw_table row, as at ADD. The Block and Node-ID come from the node's BGPRouter and the Argument from the VPC's BGPVRFInstance, and the shard routes from the conflist's egress configuration the daemon already loads. Every value goes through the attachreg helpers ADD uses. A pass writes only rows that are missing. It skips an interface with no usable BGPVRFInstance, anything under the ingress sidecar's Block, and VRF tables the sidecar or the sidecar return path own. If an interface is deleted or re-enslaved while its rows are written, as a concurrent CNI DEL does, the pass removes the ifindex-keyed rows it just wrote. One summary line reports the rows rebuilt per map, at Info only when it rebuilt something. A public uplink whose neighbor has not resolved is reported but does not fail the pass. egress_route_table is a longest-prefix-match trie, so a lookup for a prefix returns a covering ::/0 entry and cannot say whether the prefix itself is present. EgressRouteTable.Prefixes walks the keys once per pass for an exact answer; Lookup's doc comment now says what it actually returns. The new root test caught this: it registers attachments through the ADD-side helpers, empties every map, and requires one pass to restore every row exactly. Re-attaching usid_egress is not part of this; #776 does that at startup. The repair also stays off the installer's main loop: the GC-tick pass runs in a goroutine behind the size-1 semaphore and is skipped while one is still running, and a shard that cannot be resolved is tried once per pass instead of once per VRF. The startup pass retries for up to 10 minutes while attachments wait on a BGPRouter or BGPVRFInstance. Rows written for a link that vanishes mid-pass are rolled back, including the local pass-through routes and the shared vrf_table and vpc_attribution_table rows no other attachment needs. Related to #774 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
privateip
force-pushed
the
fix/reseed-attachment-datapath
branch
from
October 7, 2026 11:39
0353715 to
99acea5
Compare
privateip
marked this pull request as ready for review
October 7, 2026 11:47
privateip
enabled auto-merge
October 7, 2026 11:47
Collaborator
Author
|
Second-pass verdicts on head 99acea5: pr-rereviewer VERDICT merge (all three findings applied after the first pass confirmed). pr-conventions-reviewer VERDICT merge, no findings. First pass: pr-adversary merge; pr-conventions-reviewer hold with one warning and two nits, all fixed in the pushed commits. The affected root-privilege test suites were run as root and passed. Two decisions were settled by the author: the PR lands without waiting on #580, #598, #613 and #634, and the merge to staging is the soak before release. CI: every check on the head passed (Build, Lint, Unit Tests, Unit Tests (root), E2E Tests, the image publish jobs, kustomize bundle, CLA). Auto-merge is enabled with the merge method, the only one the base branch ruleset allows. The ruleset requires one approving review, code owner review, and approval of the last push, so the PR waits for a human approval. |
scotwells
approved these changes
Oct 7, 2026
This was referenced Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Some datapath tables are written only when a workload attaches, so a node that loses them blackholes every workload on it until each one is recreated, as in the 2026-10-07 staging outage.
The CNI installer now rebuilds any missing rows for every live attachment at startup and on each garbage-collection pass, using the same registration code that attach uses. It fills only what is missing and leaves the edge sidecar's rows alone.
Static-address attachments with no gateway still miss their pass-through row.
Test plan
Related to #774
🤖 Generated with Claude Code