Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion docs/agents/ARCHITECTURE-CNI.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,8 @@ CNI config and acts on them.

Each attachment endpoint is assigned a /128 USID (Unique Local SID, RFC 8986
Section 3.2). There is no companion-operator-injected `srv6_sid` NAD/config
field: `galactic-bgp` (`internal/cnibgp/bgp.go`'s `registerEBPFDatapath`)
field: `galactic-bgp` (`internal/cnibgp/bgp.go`'s `registerEBPFDatapath`,
which calls `internal/plumbing/ebpf/attachreg`'s `RegisterDatapath`)
derives the uSID `Block` from the node's `BGPRouter.spec.srv6Locator` via
`uformat.Block`, and registers `locator_table`/`function_table`/`vrf_table`
entries keyed on that `Block` plus this attachment's `Argument` (its
Expand All @@ -71,6 +72,30 @@ history). If the router lacks either `srv6Locator` or `nodeID`, eBPF
registration is skipped entirely for that attachment (`registerEBPFDatapath`
returns `registered=false`, not an error).

CNI ADD is the only writer of an attachment's rows, and of this node's own
`locator_table`, `function_table`, `node_src_addr_table` and
`public_uplink_table` rows, so a map recreated empty by a layout change or a
lost pin would otherwise stay empty until every workload re-attaches.
`galactic-cni run` rebuilds every missing row
(`internal/gc/datapath_repair.go`'s `RepairAttachmentDatapath`, through the
same `attachreg` helpers ADD uses), always off the daemon's main loop: once
right after the datapath loads, retried while a pass fails or finds
attachments whose `BGPRouter` or `BGPVRFInstance` has not appeared yet (the
latter for up to ten minutes), and again on every eBPF GC tick. It finds
attachments by their host interface name and VRF master, takes the routing
table and egress kind from the kernel, the guest prefixes from the pod-subnet
routes out that interface and the gateways from its own addresses, and the
Block and Argument from the `BGPRouter` and `BGPVRFInstance`. It only writes a
row that is missing, and never one under the ingress sidecar's Block. The
sidecar return path's tables are excluded by their reserved table ID range,
since the sidecar's own `vrf_table` rows that also mark them may be among the
rows lost. A VPC's shared `vrf_table` and `vpc_attribution_table` rows are
rebuilt from its lowest-ifindex attachment, while ADD leaves whichever
attachment registered last, so those two may differ from before the loss;
delivery reads the per-interface `ifindex_egress_kind_table` row, which is
exact. Once no egress shard resolves, a pass skips the remaining VRFs' shard
routes rather than wait out each one's neighbor solicitation.

`galactic-router`'s own reconciler independently derives the *same* SID value
from the same inputs (`srv6.ComputeSID`, `internal/plumbing/srv6/usid.go`,
called from `internal/reconcile/reconcile.go`) for the BGP control-plane
Expand Down
2 changes: 1 addition & 1 deletion docs/cni/environment-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,7 @@ Each entry names a shard's Block, Node-ID and Function. Whatever **Argument**
it carries is a placeholder: it identifies no tenant and never could, one
configured value being shared by every VRF on every node. Each attachment's
CNI ADD rewrites that Argument to its own VRFID before installing the route
(`internal/cnibgp`'s `shardSIDsForTenant`), which is what lets a shard tell
(`internal/plumbing/ebpf/attachreg`'s `ShardSIDsForTenant`), which is what lets a shard tell
two tenants on one node apart — see
[#538](https://github.com/datum-cloud/galactic/issues/538). An entry that is
not a well-formed uFMT 48+16 address fails the ADD there rather than being
Expand Down
6 changes: 6 additions & 0 deletions docs/cni/gc-cmd-sequence.md
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,12 @@ sequenceDiagram

### Notes

- **Missing rows are rebuilt alongside.** The same tick starts
`RepairAttachmentDatapath` (`internal/gc/datapath_repair.go`) off the main
loop, which writes back every row CNI ADD wrote that a recreated map has
lost, from kernel state and the CRDs. It also runs when the datapath loads.
The two need no ordering: this sweep reaps only keys no live
`BGPVRFInstance` names, and the repair writes keys for live ones only.
- **This is a reconcile against a generation cutoff, not a two-list diff.**
`Generation()` is captured *before* `BGPVRFInstance`s are listed. A CNI ADD
that registers a brand-new entry (`registerEBPFDatapath`, see
Expand Down
Loading
Loading