Skip to content

fix: Retry netlink listings that another process interrupts #1137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
privateip merged 1 commit into from
Oct 27, 2026
Merged

fix: Retry netlink listings that another process interrupts #1137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 18 additions & 13 deletions docs/observability/netlink.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,22 @@

See [scraping and component ports](README.md#where-the-metrics-come-from).

The kernel interrupts a listing of links, addresses or routes when another
process changes them while it runs. The listing is then discarded and
retried: up to 5 rounds of 10 back-to-back attempts, with a pause of 20, 40,
60 and 80 ms between rounds. These counters show how often that happens on a
node, so sustained churn is visible before it makes a component fail.

| Metric | Type | Labels | Meaning |
| ----------------------------------------- | ------- | ----------------------------------- | ---------------------------------------------------------------------------------------------- |
| `galactic_netlink_dump_interrupted_total` | counter | `kind` (`link`, `address`, `route`) | Listing attempts the kernel interrupted. Each one is discarded and retried |
| `galactic_netlink_dump_failures_total` | counter | `kind` (`link`, `address`, `route`) | Listings still interrupted on their last attempt. The caller got an error and acted on nothing |
The kernel interrupts a listing of links, addresses, routes, neighbors or
rules when another process changes them while it runs. Every such listing in
the node binaries discards an interrupted result and retries: up to 5 rounds
of 10 back-to-back attempts, with a pause of 20, 40, 60 and 80 ms between
rounds. These counters show how often that happens on a node, so sustained
churn is visible before it makes a component fail.

On Linux 7.1, listings of thousands of routes, neighbors or rules were never
interrupted under churn, while link and address listings were, so the
`route`, `neighbor` and `rule` series can stay at 0. A non-zero value there
is not a fault.

| Metric | Type | Labels | Meaning |
| ----------------------------------------- | ------- | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| `galactic_netlink_dump_interrupted_total` | counter | `kind` (`link`, `address`, `route`, `neighbor`, `rule`) | Listing attempts the kernel interrupted. Each one is discarded and retried |
| `galactic_netlink_dump_failures_total` | counter | `kind` (`link`, `address`, `route`, `neighbor`, `rule`) | Listings still interrupted on their last attempt. The caller got an error and acted on nothing |

Each process counts only its own listings, on its existing metrics port:

Expand All @@ -29,9 +35,8 @@ Until one exists, the queries below show nothing for them, which is not the
same as no churn.

The CNI plugins run once per attach and expose no metrics, so their
listings are not counted; an exhausted retry there fails the attach or
detach with the interruption error. Listings that do not go through the
shared retry are not counted either.
retries are not counted; an exhausted retry there fails the attach or
detach with the interruption error.

Interruptions are normal while pods start and stop. A failure is not: the
component returned an error for that pass and retries it later.
Expand Down
12 changes: 7 additions & 5 deletions internal/cni/netns.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import (

"github.com/containernetworking/plugins/pkg/ns"
"github.com/vishvananda/netlink"

"go.datum.net/galactic/internal/plumbing/nldump"
)

// configureInterfaceInNetns applies addresses and default routes to the guest
Expand Down Expand Up @@ -101,7 +103,7 @@ func addrFamily(ip net.IP) int {
func addAddrIfMissing(handle *netlink.Handle, link netlink.Link, ifName string, ipNet *net.IPNet) error {
want := netlink.Addr{IPNet: ipNet}

existing, err := handle.AddrList(link, addrFamily(ipNet.IP))
existing, err := nldump.On(handle).AddrList(link, addrFamily(ipNet.IP))
if err != nil {
return fmt.Errorf("list addresses on %q: %w", ifName, err)
}
Expand All @@ -122,7 +124,7 @@ func addAddrIfMissing(handle *netlink.Handle, link netlink.Link, ifName string,
// gateway is a real misconfiguration rather than something a retried ADD should
// paper over, so it returns an error.
func addDefaultRouteIfMissing(handle *netlink.Handle, link netlink.Link, ifName string, gateway net.IP) error {
existing, err := handle.RouteList(link, addrFamily(gateway))
existing, err := nldump.On(handle).RouteList(link, addrFamily(gateway))
if err != nil {
return fmt.Errorf("list routes on %q: %w", ifName, err)
}
Expand All @@ -136,7 +138,7 @@ func addDefaultRouteIfMissing(handle *netlink.Handle, link netlink.Link, ifName
return fmt.Errorf("default route on %q already points via %s, refusing to add conflicting route via %s",
ifName, r.Gw, gateway)
}
routes, err := handle.RouteList(nil, addrFamily(gateway))
routes, err := nldump.On(handle).RouteList(nil, addrFamily(gateway))
if err != nil {
return fmt.Errorf("list namespace routes: %w", err)
}
Expand Down Expand Up @@ -208,7 +210,7 @@ func flushGuestNetnsConfig(netnsPath, ifName string) error {
}

for _, family := range []int{netlink.FAMILY_V6, netlink.FAMILY_V4} {
routes, err := handle.RouteList(link, family)
routes, err := nldump.On(handle).RouteList(link, family)
if err != nil {
return fmt.Errorf("list routes on %q: %w", ifName, err)
}
Expand All @@ -221,7 +223,7 @@ func flushGuestNetnsConfig(netnsPath, ifName string) error {
}
}

addrs, err := handle.AddrList(link, family)
addrs, err := nldump.On(handle).AddrList(link, family)
if err != nil {
return fmt.Errorf("list addresses on %q: %w", ifName, err)
}
Expand Down
3 changes: 2 additions & 1 deletion internal/cni/ops_check.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import (

"go.datum.net/galactic/internal/cnimaster"
"go.datum.net/galactic/internal/plumbing/intf"
"go.datum.net/galactic/internal/plumbing/nldump"
)

// cmdCheck validates that the container's network state matches what ADD
Expand Down Expand Up @@ -215,7 +216,7 @@ func validateIPOnInterface(ip net.IP, name, netns string) error {
return fmt.Errorf("find link: %w", err)
}

addrs, err := handle.AddrList(link, netlink.FAMILY_V6)
addrs, err := nldump.On(handle).AddrList(link, netlink.FAMILY_V6)
if err != nil {
return fmt.Errorf("list addresses: %w", err)
}
Expand Down
3 changes: 2 additions & 1 deletion internal/cniroute/ops_check.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import (
"github.com/vishvananda/netlink"

"go.datum.net/galactic/internal/plumbing/intf"
"go.datum.net/galactic/internal/plumbing/nldump"
"go.datum.net/galactic/internal/plumbing/vrf"
)

Expand Down Expand Up @@ -63,7 +64,7 @@ func checkTerminationRoutes(vpc, vpcAttachment string, terminations []Terminatio
}
defer handle.Close() //nolint:errcheck // netlink cleanup on teardown

routes, err := handle.RouteListFiltered(
routes, err := nldump.On(handle).RouteListFiltered(
netlink.FAMILY_V6,
&netlink.Route{Table: int(tableID)},
netlink.RT_FILTER_TABLE,
Expand Down
6 changes: 4 additions & 2 deletions internal/edge/hostattach/hostattach.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ import (

"github.com/vishvananda/netlink"
"golang.org/x/sys/unix"

"go.datum.net/galactic/internal/plumbing/nldump"
)

// MinMTU is the smallest MTU an attachment link or packet device may have:
Expand Down Expand Up @@ -459,7 +461,7 @@ func ensureMTU(h *netlink.Handle, link netlink.Link, mtu int) error {
// route for it from the VRF table, after which the address no longer accepts
// traffic.
func ensureAddr(h *netlink.Handle, link netlink.Link, addr *netlink.Addr) error {
addrs, err := h.AddrList(link, unix.AF_INET6)
addrs, err := nldump.On(h).AddrList(link, unix.AF_INET6)
if err != nil {
return err
}
Expand Down Expand Up @@ -535,7 +537,7 @@ func configureHostLink(
// attachment or to a workload, and choosing between them is not the
// provisioner's decision.
func checkRouteOwner(h *netlink.Handle, r *netlink.Route) error {
existing, err := h.RouteListFiltered(unix.AF_INET6, &netlink.Route{Table: r.Table, Dst: r.Dst},
existing, err := nldump.On(h).RouteListFiltered(unix.AF_INET6, &netlink.Route{Table: r.Table, Dst: r.Dst},
netlink.RT_FILTER_TABLE|netlink.RT_FILTER_DST)
if err != nil {
return fmt.Errorf("hostattach: list routes for %s in table %d: %w", r.Dst, r.Table, err)
Expand Down
6 changes: 4 additions & 2 deletions internal/edge/hostattach/inventory.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import (

"github.com/vishvananda/netlink"
"golang.org/x/sys/unix"

"go.datum.net/galactic/internal/plumbing/nldump"
)

const aliasPrefix = "galactic-edge:hostattach:"
Expand Down Expand Up @@ -69,7 +71,7 @@ func ListHost() ([]ObservedAttachment, error) {
}

func listOwned(h *netlink.Handle) ([]ObservedAttachment, error) {
links, err := h.LinkList()
links, err := nldump.On(h).LinkList()
if err != nil {
return nil, fmt.Errorf("hostattach: list owned links: %w", err)
}
Expand All @@ -95,7 +97,7 @@ func hostPrefixes(a Attachment) []netip.Prefix {
// no longer appear in desired state. Kernel-derived local/link-local routes
// and routes using other links are never candidates for deletion.
func extraRoutes(h *netlink.Handle, index int, table uint32, want []netip.Prefix) ([]netlink.Route, error) {
routes, err := h.RouteListFiltered(unix.AF_INET6, &netlink.Route{Table: int(table), LinkIndex: index},
routes, err := nldump.On(h).RouteListFiltered(unix.AF_INET6, &netlink.Route{Table: int(table), LinkIndex: index},
netlink.RT_FILTER_TABLE|netlink.RT_FILTER_OIF)
if err != nil {
return nil, fmt.Errorf("hostattach: list attachment routes: %w", err)
Expand Down
14 changes: 9 additions & 5 deletions internal/edge/hostattach/readback.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ import (

"github.com/vishvananda/netlink"
"golang.org/x/sys/unix"

"go.datum.net/galactic/internal/plumbing/nldump"
)

// ErrDrift is wrapped by Readback when the kernel does not hold the state the
Expand Down Expand Up @@ -138,7 +140,7 @@ func checkLink(h *netlink.Handle, name, master string, add func(string, ...any))
}

func checkAddr(h *netlink.Handle, link netlink.Link, want netip.Addr, add func(string, ...any)) {
addrs, err := h.AddrList(link, unix.AF_INET6)
addrs, err := nldump.On(h).AddrList(link, unix.AF_INET6)
if err != nil {
add("list addresses on %q: %v", link.Attrs().Name, err)
return
Expand All @@ -155,7 +157,7 @@ func checkAddr(h *netlink.Handle, link netlink.Link, want netip.Addr, add func(s
}

func checkNeigh(h *netlink.Handle, link netlink.Link, ip netip.Addr, mac net.HardwareAddr, add func(string, ...any)) {
neighs, err := h.NeighList(link.Attrs().Index, unix.AF_INET6)
neighs, err := nldump.On(h).NeighList(link.Attrs().Index, unix.AF_INET6)
if err != nil {
add("list neighbors on %q: %v", link.Attrs().Name, err)
return
Expand All @@ -177,7 +179,8 @@ func checkNeigh(h *netlink.Handle, link netlink.Link, ip netip.Addr, mac net.Har
func checkRoute(
h *netlink.Handle, table uint32, dst netip.Prefix, ifindex int, gw netip.Addr, add func(string, ...any),
) {
routes, err := h.RouteListFiltered(unix.AF_INET6, &netlink.Route{Table: int(table)}, netlink.RT_FILTER_TABLE)
routes, err := nldump.On(h).RouteListFiltered(
unix.AF_INET6, &netlink.Route{Table: int(table)}, netlink.RT_FILTER_TABLE)
if err != nil {
add("list routes in table %d: %v", table, err)
return
Expand All @@ -198,7 +201,8 @@ func checkRoute(
}

func checkUnreachableDefault(h *netlink.Handle, table uint32, add func(string, ...any)) {
routes, err := h.RouteListFiltered(unix.AF_INET6, &netlink.Route{Table: int(table)}, netlink.RT_FILTER_TABLE)
routes, err := nldump.On(h).RouteListFiltered(
unix.AF_INET6, &netlink.Route{Table: int(table)}, netlink.RT_FILTER_TABLE)
if err != nil {
add("list routes in table %d: %v", table, err)
return
Expand All @@ -214,7 +218,7 @@ func checkUnreachableDefault(h *netlink.Handle, table uint32, add func(string, .
// checkLocalRoute requires the kernel's local route for addr in table, without
// which the pod VRF does not deliver traffic addressed to the attachment.
func checkLocalRoute(h *netlink.Handle, table uint32, addr netip.Addr, add func(string, ...any)) {
routes, err := h.RouteListFiltered(unix.AF_INET6,
routes, err := nldump.On(h).RouteListFiltered(unix.AF_INET6,
&netlink.Route{Table: int(table), Type: unix.RTN_LOCAL}, netlink.RT_FILTER_TABLE|netlink.RT_FILTER_TYPE)
if err != nil {
add("list local routes in table %d: %v", table, err)
Expand Down
Loading
Loading