Netlink listings fail during concurrent link changes #1133

Activity

1. privateip

   added

   enhancementNew feature or requestNew feature or request

   tech-debtInternal quality/maintenance: unwind shortcuts, config drift, overdue upgrades, fork/dep hygieneInternal quality/maintenance: unwind shortcuts, config drift, overdue upgrades, fork/dep hygiene

   on Oct 27, 2026
2. privateip commented on Oct 27, 2026

   

   privateip

   on Oct 27, 2026

   CollaboratorAuthor

   More actions

   Remaining single-attempt listings on main at 20d8313, excluding the sidecar and VRF call sites that #1132 moves:

   • internal/gc/gc.go (listAllLinksFn) and internal/gc/datapath_repair.go (route and address listings)
   • internal/hostgw/hostgw.go (route listing) and internal/hostgw/tapneigh.go (link, route and neighbour listings)
   • internal/hostconf/hostconf.go, internal/installer/installer.go, internal/installer/locatorroute.go, internal/installer/sidecarreturn.go
   • internal/plumbing/ebpf/attach/interfaces.go, internal/plumbing/ebpf/attach/egress_reattach.go, and the linkListFn defaults in edgeattach, natattach and xdpattach
   • internal/plumbing/ebpf/egressroutemap/egressroute.go and internal/plumbing/srv6/egress.go (neighbour listings)
   • internal/plumbing/srv6/plainroute.go (rule and route listings)
   • internal/plumbing/vip/vip.go and internal/plumbing/loaddr/loaddr.go (address listings)
3. privateip

   self-assigned this

   on Oct 27, 2026
4. privateip commented on Oct 27, 2026

   

   privateip

   on Oct 27, 2026

   CollaboratorAuthor

   More actions

   Confirmed on main at 1a9d0f1.

   Reproduction: a root test ran garbage collection's VRF lookup (resolveVRFKernelState) in a network namespace while two goroutines added and removed links. 139 of 300 lookups failed with execute netlink request after 1 attempts: results may be incomplete or inconsistent, from the single-attempt link listing.

   Two findings that go beyond the summary above:

   • The handle-based listings in the CNI plugins (internal/cni, internal/cniroute) and in the edge agent (internal/edge/hostattach) make single attempts too. The fix covers them through a nldump form that retries over a handle the caller already holds.
   • On a 7.1 kernel, only link and address dumps were reported interrupted under churn. Route, neighbor and rule dumps of up to 4,000 entries were not. The fix still retries all five kinds, since netlink documents that each can return an interrupted dump. The churn tests use link and address churn, which is what makes garbage collection, datapath repair and the host gateway fail today.

   Plan:

   1. Add neighbor and rule listings to nldump, counted under new kind values on the existing galactic_netlink_dump_* counters.
   2. Move every remaining listing in the node binaries onto nldump. hack/returnpath-lab is a lab tool and stays as it is.
   3. Add root churn tests in which garbage collection, datapath repair and the host gateway each complete passes while links and addresses change.
5. privateip commented on Oct 27, 2026

   

   privateip

   on Oct 27, 2026

   CollaboratorAuthor

   More actions

   Fix is up for review in #1137. It also moves the CNI plugins and the edge agent onto the shared retry, and adds root churn tests for garbage collection, datapath repair and the host gateway that fail on the previous main.
6. privateip

   closed this as completedin #1137on Oct 27, 2026