diff --git a/AGENTS.md b/AGENTS.md index 1f793771..52d68265 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,11 +9,11 @@ constraints, and a "For Claude" quick-start table). Start from whichever one matches the binary or subsystem you're touching; each cross-links to the other two where the boundary matters: -| Document | Covers | Start here when you're touching... | -| -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| [docs/agents/ARCHITECTURE-CNI.md](docs/agents/ARCHITECTURE-CNI.md) | The CNI attach chain: `galactic-cni` (installer), `galactic-veth`, `galactic-tap`, `galactic-ipam`, `galactic-bgp`, `galactic-route` | Pod/VM attach or detach behavior, CNI config fields/conflists, IPAM, the SRv6 uSID TC-BPF datapath's CNI-side registration, or anything under `cmd/galactic-{cni,veth,tap,ipam,bgp,route}`, `internal/cni*`, `internal/installer` | -| [docs/agents/ARCHITECTURE-ROUTER.md](docs/agents/ARCHITECTURE-ROUTER.md) | The BGP/EVPN control plane: `galactic-router` (BGPRouter/BGPPeer/BGPAdvertisement/BGPPolicy/BGPVRFInstance reconcilers, embedded GoBGP, GC) | BGP CRD reconciliation, GoBGP runtime behavior, EVPN path construction, orphaned-CRD/VRF garbage collection, or anything under `cmd/galactic-router`, `internal/reconcile`, `internal/runtime`, `internal/gc`, `internal/model`, `internal/hash` | -| [docs/agents/ARCHITECTURE-GATEWAY.md](docs/agents/ARCHITECTURE-GATEWAY.md) | The edge XDP NAT+LB gateway: `galactic-gateway`, `NetworkGateway`/`NetworkRule` reconcilers, the edge XDP datapath | Ingress load-balancing/NAT, `NetworkGateway`/`NetworkRule` CRDs, Active-Active BGP placement, or anything under `cmd/galactic-gateway`, `internal/gateway`, `internal/plumbing/ebpf/edge{prog,map,attach}`, the `NetworkGateway`/`NetworkRule`-related code in `internal/controller` | +| Document | Covers | Start here when you're touching... | +| -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [docs/agents/ARCHITECTURE-CNI.md](docs/agents/ARCHITECTURE-CNI.md) | The CNI attach chain: `galactic-cni` (installer), `galactic-veth`, `galactic-tap`, `galactic-ipam`, `galactic-bgp`, `galactic-route` | Pod/VM attach or detach behavior, CNI config fields/conflists, IPAM, the SRv6 uSID TC-BPF datapath's CNI-side registration, or anything under `cmd/galactic-{cni,veth,tap,ipam,bgp,route}`, `internal/cni*`, `internal/installer` | +| [docs/agents/ARCHITECTURE-ROUTER.md](docs/agents/ARCHITECTURE-ROUTER.md) | The BGP/EVPN control plane: `galactic-router` (BGPRouter/BGPPeer/BGPAdvertisement/BGPPolicy/BGPVRFInstance reconcilers, embedded GoBGP, GC) | BGP CRD reconciliation, GoBGP runtime behavior, EVPN path construction, orphaned-CRD/VRF garbage collection, or anything under `cmd/galactic-router`, `internal/reconcile`, `internal/runtime`, `internal/gc`, `internal/model`, `internal/hash` | +| [docs/agents/ARCHITECTURE-GATEWAY.md](docs/agents/ARCHITECTURE-GATEWAY.md) | The edge XDP DSR load balancer: `galactic-gateway`, `NetworkGateway`/`NetworkRule` reconcilers, the edge XDP datapath | Ingress L4 load balancing, `NetworkGateway`/`NetworkRule` CRDs, anycast VIP advertisement, or anything under `cmd/galactic-gateway`, `internal/gateway`, `internal/plumbing/ebpf/edge{prog,map,attach}`, the `NetworkGateway`/`NetworkRule`-related code in `internal/controller` | These three documents supersede the former monolithic `ARCHITECTURE.md`, which is now a short pointer to them. @@ -24,7 +24,7 @@ Galactic is the SRv6 data plane for multi-cloud VPC networking. It consists of t - **`galactic-veth`** — CNI plugin that wires containers into VPC networks (VRF, veth, SRv6 ingress route) and writes `BGPAdvertisement` CRDs. - **`galactic-router`** — controller-runtime reconciler that watches BGP CRDs and drives an embedded GoBGP server per node to distribute EVPN paths. -- **`galactic-gateway`** — controller-runtime process that loads an edge XDP NAT+LB datapath on gateway-role nodes and drives it from `NetworkGateway`/`NetworkRule` CRDs, publishing an Active-Active BGP local-preference model over the same EVPN mesh. +- **`galactic-gateway`** — controller-runtime process that loads an edge XDP DSR (Direct Server Return) L4 load balancer on gateway-role nodes and drives it from `NetworkGateway`/`NetworkRule` CRDs. Every gateway node advertises every VIP identically (anycast) over the same EVPN mesh. VPC and VPCAttachment CRD management lives in a separate companion operator; Galactic receives pre-populated identifiers through the CNI config and acts on them. @@ -37,7 +37,7 @@ VPC and VPCAttachment CRD management lives in a separate companion operator; Gal - **BGP API** (`go.datum.net/network`) — BGPRouter, BGPPeer, BGPAdvertisement, BGPPolicy, BGPVRFInstance, NetworkGateway, NetworkRule CRDs - **GoBGP v4** — embedded BGP server (default role) - **SRv6 + netlink** — kernel-level routing; `github.com/vishvananda/netlink` -- **eBPF** (`github.com/cilium/ebpf`) — TC-BPF SRv6 uSID decap datapath (CNI side) and XDP edge NAT+LB datapath (`galactic-gateway`) +- **eBPF** (`github.com/cilium/ebpf`) — TC-BPF SRv6 uSID decap datapath (CNI side) and XDP edge DSR load-balancer datapath (`galactic-gateway`) - **Multus CNI** — multi-network for pods; NAD generation handled by the external operator ## Development Workflow @@ -91,7 +91,7 @@ See [docs/node-labels.md](docs/node-labels.md) for the full strategy — why `ga - **`config/galactic-router/overlays/router/`** — the plain (non-reflector) per-node role (`galactic-router`); runs on every `compute` *and* every `edge` node except Kubernetes control-plane nodes, opt-in via `galactic.datumapis.com/galactic: router` (the same label `galactic-cni` keys off of — see `config/galactic-cni/daemonset.yaml`). This is the default role, so it carries no role-specific name/label of its own — only `control` below needs one, to coexist as a second DaemonSet. - **`config/galactic-router/overlays/control/`** — the BGP route-reflector role (`galactic-router-rr`, `GALACTIC_ROUTER_REFLECTOR=true`); opt-in only, requires `galactic.datumapis.com/galactic: control` (stays at zero replicas otherwise). Mutually exclusive with `router`'s `galactic: router` value on the same node — one label key, single-valued, so a route-reflector node structurally can't also run plain-mode `galactic-router`. `GALACTIC_ROUTER_BGP_LOCAL_ADDRESS` is auto-detected from the host's `lo` interface by default; see the comments in `daemonset-patch.yaml` for when to override it. - Apply both roles with `kubectl apply -k config/galactic-router/`, or a single role with e.g. `kubectl apply -k config/galactic-router/overlays/router/`. `galactic-router` no longer runs bundled inside `galactic-gateway`'s pod — see `config/galactic-gateway/` below. -- **`config/galactic-gateway/`** — the edge XDP NAT+LB gateway control plane, a single-container `galactic-gateway` DaemonSet — `galactic-router` used to run as a second container in this same pod; it's now a fully independent DaemonSet (via `galactic.datumapis.com/galactic: router`, which a gateway node may carry like any other), so a crash on either side no longer takes the other's *pod* down with it, not just the other's binary. `config/galactic-gateway/{serviceaccount.yaml,rbac.yaml}` (safe/idempotent to apply cluster-wide) are what `kubectl apply -k config/galactic-gateway/` applies; `config/galactic-gateway/base/` (requiring nodes labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/gateway: enabled`, so it runs on edge nodes alongside `galactic-vrf`) is **not** included in that kustomization and is **not** applied as-is — the same exemption as `config/fabric-router/` below, for the same reason: `GALACTIC_GATEWAY_SRV6_ADDRESS` must be unique per gateway node and has no generic default (no in-cluster mechanism yet derives it automatically — see `internal/config/gateway.go`'s `EnvGatewaySRv6Address` doc comment). It's designed to be instantiated once per gateway node by a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's own public-interface/SRv6-address values; see `deploy/containerlab/resources/galactic-gateway/` as of commit `abdd665b` for a worked example (the containerlab lab no longer deploys the gateway). Also **not** part of the root `config/kustomization.yaml`'s default resource list, matching `config/fabric-router/`'s exemption. +- **`config/galactic-gateway/`** — the edge XDP DSR load-balancer control plane, a single-container `galactic-gateway` DaemonSet — `galactic-router` used to run as a second container in this same pod; it's now a fully independent DaemonSet (via `galactic.datumapis.com/galactic: router`, which a gateway node may carry like any other), so a crash on either side no longer takes the other's *pod* down with it, not just the other's binary. `config/galactic-gateway/{serviceaccount.yaml,rbac.yaml}` (safe/idempotent to apply cluster-wide) are what `kubectl apply -k config/galactic-gateway/` applies; `config/galactic-gateway/base/` (requiring nodes labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/gateway: enabled`, so it runs on edge nodes alongside `galactic-vrf`) is **not** included in that kustomization and is **not** applied as-is — the same exemption as `config/fabric-router/` below, for the same reason: `GALACTIC_GATEWAY_SRV6_ADDRESS` must be unique per gateway node and has no generic default (no in-cluster mechanism yet derives it automatically — see `internal/config/gateway.go`'s `EnvGatewaySRv6Address` doc comment). It's designed to be instantiated once per gateway node by a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's own public-interface/SRv6-address values; see `deploy/containerlab/resources/galactic-gateway/` as of commit `abdd665b` for a worked example (the containerlab lab no longer deploys the gateway). Also **not** part of the root `config/kustomization.yaml`'s default resource list, matching `config/fabric-router/`'s exemption. - **`config/galactic-nat/`** — the sharded egress translation (NAT66/NAT64) control plane, one `galactic-nat` DaemonSet per cluster on every node labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/nat: enabled`, alongside `galactic-vrf`. It is independent of `galactic-gateway`. A shard's identity (SID, masquerade addresses, NAT64 prefix) comes from the `EgressShard` targeting its node, not from the DaemonSet. The base leaves `GALACTIC_NAT_XDP_ATTACH` at its default, `direct`. On a node that also runs the gateway, set it to `dispatch`: an interface takes one native XDP program, so the shard and the gateway (which defaults to `GALACTIC_GATEWAY_XDP_ATTACH=dispatch`) each run from their own slot of the node's shared, pinned XDP dispatcher (`internal/plumbing/ebpf/xdpdispatch`), and either can restart or be turned off (`GALACTIC_*_DATAPATH_ENABLED=false`) without touching the other's traffic. `hack/xdp-dispatch-release.sh` detaches the dispatcher from a node. Compute nodes reach their own site's shards over SRv6 via `GALACTIC_CNI_EGRESS_SHARD_SIDS`. `kubectl apply -k config/galactic-nat/` applies only the ServiceAccount/RBAC; `base/` is excluded because uplink auto-detection is deployment-specific (see `config/galactic-nat/kustomization.yaml`). Not part of the root `config/kustomization.yaml`. - **`config/fabric-router/`** — the FRR underlay eBGP DaemonSet (`fabric-router`; `galactic-router` needs a working underlay before it can start). Unlike `config/galactic-router/`, this is a single flat DaemonSet with no `default`/`rr`-style role split — its affinity matches any node labeled `galactic.datumapis.com/fabric: router` (a dedicated mode-enum label, independent of whatever `galactic.datumapis.com/node` role value or `galactic.datumapis.com/galactic` mode that node also carries) directly, since there's no env/config difference between running on a regular node vs. those roles — fabric-router itself is identical everywhere; only the per-node BGP underlay config it reads from the ConfigMap differs. That affinity can legitimately match more than one node per cluster, and BGP underlay config (hostname, router-id, interface addresses, remote-AS) inherently differs per physical node — so each node's config lives in its own `fabric-router.` ConfigMap (a `frr.conf` key, plus optional `daemons`/`vtysh.conf` overrides of the image defaults) rather than one shared ConfigMap for the whole DaemonSet. `fabric-config-agent` (`cmd/fabric-config-agent`, `internal/fabricconfig`, built into the `fabric-router` image) reads it from the API: as the `frr-init` init container it validates (`vtysh -C`) and installs it, holding the pod in `Init` until a valid one exists; as the `config-agent` sidecar it applies later changes live with `frr-reload.py`, so an edit to one node's ConfigMap neither restarts the pod nor touches other nodes, and reports `Applied`/`Rejected`/`ReloadFailed`/`RestartRequired` events on the ConfigMap. The sidecar also hard-resets any IPv6 session announcing an IPv4-mapped next hop (`internal/fabricconfig/nexthop.go`) and serves that check's metrics on the node's port `9343`. A third container, `frr-exporter` (upstream `tynany/frr_exporter`, pinned by digest), reads the daemons' vty sockets in the same pod-scoped `/run/frr` and serves BGP peer state, message counters and prefix counts for both underlay families on the node's port `9342` (`hostNetwork`); `task verify:fabric-metrics` in `deploy/containerlab/` checks it on every lab node. The image's default `daemons` loads bgpd's BMP module (`-M bmp`), so a node's `frr.conf` can stream to a BMP collector (`config/bmp-collector/`) with a `bmp targets` block; loading it alone streams nothing, and a change to the modules needs a pod restart rather than a reload. Every lab node's `frr.conf` streams its pre-policy Adj-RIB-In and Loc-RIB for both families, sourced from `lo` (`source-interface lo`), since the underlay carries loopbacks but not link subnets. Production `frr.conf`s come from infra's renderer, which has to emit the same block for production nodes to stream. A node with no ConfigMap of its own falls back, with a `DeprecatedConfigSource` warning, to key `frr.conf.` of the legacy shared `fabric-config` ConfigMap. **Not** part of the root `config/kustomization.yaml` and not covered by `kubectl apply -k config/` — unlike every other component here, it has no generic default: the deployer must hand-author one `fabric-router.` ConfigMap per matching node before applying `kubectl apply -k config/fabric-router/` (which also applies the agent's ServiceAccount and namespaced Role). - **`config/bmp-collector/`** — a single-replica GoBMP Deployment and its `bmp-collector` Service (TCP `5000`), the BMP (RFC 7854) collector `galactic-router` and `fabric-router` stream to; `galactic-router` does so when `GALACTIC_ROUTER_BMP_STATIONS` names it (`bmp-collector.galactic-system.svc:5000`; see [docs/router/configuration.md](docs/router/configuration.md)). It parses every Peer Up/Down, route-monitoring and statistics message to JSON on stdout, so the cluster's log pipeline is its store; an overlay swapping `--dump=console` for `--kafka-server` publishes to Kafka instead. Opt-in, like `config/fabric-router/`: not part of the root `config/kustomization.yaml`, since nothing streams to it until a router is configured to. The containerlab lab runs one on `iad-worker3`'s host network (`deploy/containerlab/resources/bmp-collector/iad/`), because its three clusters cannot reach each other's Services, and points every node's `fabric-router` and the EVPN route reflector at it. Of the `galactic-router`s, only the reflector streams: its pre-policy Adj-RIB-In already holds every node's advertisements. @@ -106,7 +106,7 @@ See [docs/node-labels.md](docs/node-labels.md) for the full strategy — why `ga 4. Read `internal/runtime/gobgp/runtime.go` to understand how `DesiredRouter` is applied to GoBGP. 5. Read `internal/plumbing/intf/intf.go` to understand SRv6 endpoint encoding, interface naming, and base62↔hex conversion. 6. Explore `internal/plumbing/` for shared kernel and network primitives (VRF, sysctl, interface naming, SRv6). -7. Read `internal/gateway/engine.go` and `internal/controller/networkgateway_controller.go` to understand the edge XDP NAT+LB gateway's convergence loop and Active-Active BGP placement model. See [ARCHITECTURE-GATEWAY.md](docs/agents/ARCHITECTURE-GATEWAY.md). +7. Read `internal/gateway/engine.go` and `internal/controller/networkgateway_controller.go` to understand the edge XDP DSR load balancer's convergence loop and anycast VIP advertisement. See [ARCHITECTURE-GATEWAY.md](docs/agents/ARCHITECTURE-GATEWAY.md). 8. See `docs/cni/README.md` for the CNI docs entry point — configuration reference (env vars, conflist fields, worked examples) and Mermaid sequence diagrams for the attach path and garbage collection. `docs/agent-startup.md` covers router startup, and `docs/router/configuration.md` documents router environment variables. 9. Read [docs/node-labels.md](docs/node-labels.md) to understand which `galactic.datumapis.com/*` labels put which DaemonSets on which nodes, and how gateway and NAT opt in on edge nodes — this cuts across all five binaries and isn't owned by any single component's architecture doc. 10. See [docs/reference.md](docs/reference.md) for a quick-reference table of every deployable asset in `config/` and every TCP port its containers configure — useful when checking for a port collision or confirming what `kubectl apply -k config//` actually applies before diving into any single component's doc. diff --git a/README.md b/README.md index 3e056e05..b96cfb5a 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Modern organizations run workloads everywhere: AWS, Azure, GCP, on-premises, and ## Our Approach -Galactic provides the SRv6 data plane that makes multi-cloud VPC connectivity work at the kernel level. It runs as a DaemonSet agent on every node, managing SRv6 routes and VRF isolation, and as a CNI plugin that attaches pods to the correct virtual network. On nodes dedicated to the gateway role, a third component loads an edge XDP NAT+LB datapath to handle ingress load balancing at the VPC boundary. VPC and VPCAttachment definitions are managed by a companion operator; Galactic acts on the identifiers that operator assigns. +Galactic provides the SRv6 data plane that makes multi-cloud VPC connectivity work at the kernel level. It runs as a DaemonSet agent on every node, managing SRv6 routes and VRF isolation, and as a CNI plugin that attaches pods to the correct virtual network. On nodes dedicated to the gateway role, a third component loads an edge XDP DSR (Direct Server Return) load balancer to handle L4 ingress at the VPC boundary. VPC and VPCAttachment definitions are managed by a companion operator; Galactic acts on the identifiers that operator assigns. Under the hood, Galactic uses Segment Routing over IPv6 (SRv6) for efficient, deterministic routing and Virtual Routing and Forwarding (VRF) for true network isolation at the kernel level. BGP is used to distribute SRv6 routes between agents across nodes and clusters. @@ -43,7 +43,7 @@ Each component can also be applied on its own, e.g. `kubectl apply -k config/gal Two components are **not** part of `kubectl apply -k config/` and must be applied separately, each with its own per-node prerequisite: - **`config/fabric-router/`** — the FRR underlay eBGP DaemonSet `galactic-router` depends on. -- **`config/galactic-gateway/`** — the edge XDP NAT+LB gateway control plane, a single-container `galactic-gateway` DaemonSet on dedicated `galactic.datumapis.com/node: edge` nodes (`galactic-router` runs there too, but as its own independent DaemonSet via `galactic.datumapis.com/galactic: router` — see `config/galactic-router/` above, not bundled into this pod). `kubectl apply -k config/galactic-gateway/` only installs the shared, cluster-safe ServiceAccount/RBAC; `config/galactic-gateway/base/` itself is a template meant to be instantiated once per gateway node by a further overlay (see `deploy/containerlab/resources/galactic-gateway/` as of commit `abdd665b`, the last before the containerlab lab stopped deploying the gateway, for a worked example) — apply that overlay per node instead of `base/` directly. +- **`config/galactic-gateway/`** — the edge XDP DSR load-balancer control plane, a single-container `galactic-gateway` DaemonSet on nodes labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/gateway: enabled` (`galactic-router` runs there too, but as its own independent DaemonSet via `galactic.datumapis.com/galactic: router` — see `config/galactic-router/` above, not bundled into this pod). `kubectl apply -k config/galactic-gateway/` only installs the shared, cluster-safe ServiceAccount/RBAC; `config/galactic-gateway/base/` itself is a template meant to be instantiated once per gateway node by a further overlay (see `deploy/containerlab/resources/galactic-gateway/` as of commit `abdd665b`, the last before the containerlab lab stopped deploying the gateway, for a worked example) — apply that overlay per node instead of `base/` directly. ```bash kubectl apply -k config/fabric-router/ diff --git a/cmd/galactic-gateway/root.go b/cmd/galactic-gateway/root.go index 6369fec5..b6531c62 100644 --- a/cmd/galactic-gateway/root.go +++ b/cmd/galactic-gateway/root.go @@ -34,7 +34,7 @@ import ( const ( appName = "galactic-gateway" - appDesc = `Galactic edge XDP NAT+LB gateway + appDesc = `Galactic edge XDP DSR load balancer Find more information at: https://www.datum.net/docs` ) @@ -248,7 +248,7 @@ func newRootCommand() *cobra.Command { config.DefaultGatewayGRPCHealthPort, "gRPC health check port") cmd.Flags().StringP("gateway-public-interface", "", "", - "Public/underlay-facing uplink interface for the edge NAT+LB gateway datapath (required)") + "Public/underlay-facing uplink interface for the edge DSR load-balancer datapath (required)") cmd.Flags().StringP("gateway-internal-interfaces", "", "", "Comma-separated compute-facing interfaces whose VIP-sourced return traffic this node "+ "forwards before netfilter (optional; empty means this node carries no return traffic)") diff --git a/config/galactic-gateway/base/daemonset.yaml b/config/galactic-gateway/base/daemonset.yaml index 638ee483..773427b9 100644 --- a/config/galactic-gateway/base/daemonset.yaml +++ b/config/galactic-gateway/base/daemonset.yaml @@ -150,14 +150,15 @@ spec: periodSeconds: 10 # This container needs CAP_BPF for the bpf() syscalls themselves # (program/map creation) on top of NET_ADMIN for the netlink XDP - # attach — the same pair config/galactic-cni/daemonset.yaml's eBPF-loading - # container already uses for its own (TC-BPF) program. PERFMON is - # additionally required: the verifier only allows pointer+scalar - # arithmetic on packet data/data_end (edgenat.c's DNAT/SNAT - # header rewrites) when the loading process is perfmon_capable() - # — without it the verifier falls back to its unprivileged-BPF - # ruleset and rejects the program with "pointer arithmetic ... - # prohibited for !root" even though the container runs as root. + # attach — the same pair config/galactic-cni/daemonset.yaml's + # eBPF-loading container already uses for its own (TC-BPF) program. + # PERFMON is additionally required: the verifier only allows + # pointer+scalar arithmetic on packet data/data_end (edgedsr.c's + # header parsing and its data_end - data byte count) when the + # loading process is perfmon_capable(). Without it the verifier + # falls back to its unprivileged-BPF ruleset and rejects the program + # with "pointer arithmetic ... prohibited for !root" even though the + # container runs as root. securityContext: runAsUser: 0 capabilities: @@ -175,7 +176,7 @@ spec: limits: memory: 256Mi volumeMounts: - # edgeattach.PinDir pins every edgenat map under + # edgeattach.PinDir pins every edge map under # /sys/fs/bpf/galactic-edge, same convention config/galactic-cni/ # daemonset.yaml's bpf-fs mount documents: bpffs must already be # mounted at this host path by the node itself (type: Directory, diff --git a/config/galactic-gateway/kustomization.yaml b/config/galactic-gateway/kustomization.yaml index 8d32aea4..e14f5ef4 100644 --- a/config/galactic-gateway/kustomization.yaml +++ b/config/galactic-gateway/kustomization.yaml @@ -1,4 +1,4 @@ -# The edge XDP NAT+LB gateway control plane. Unlike config/galactic-router/'s +# The edge XDP DSR load-balancer control plane. Unlike config/galactic-router/'s # equivalent kustomization.yaml, this deliberately does NOT include base/: # base/daemonset.yaml's galactic-gateway container has no generic default # for GALACTIC_GATEWAY_PUBLIC_INTERFACE/_SRV6_ADDRESS (both required — diff --git a/config/galactic-gateway/rbac.yaml b/config/galactic-gateway/rbac.yaml index 124da3ec..f5f5f6c9 100644 --- a/config/galactic-gateway/rbac.yaml +++ b/config/galactic-gateway/rbac.yaml @@ -1,4 +1,4 @@ -# The edge XDP NAT+LB gateway control plane's own RBAC. This ClusterRole +# The edge XDP DSR load-balancer control plane's own RBAC. This ClusterRole # covers only what NetworkGatewayReconciler/NetworkRuleReconciler # (internal/controller/networkgateway_controller.go, # internal/controller/networkrule_controller.go) actually touch. diff --git a/containers/galactic-gateway/Dockerfile b/containers/galactic-gateway/Dockerfile index d0658f67..0d0ca8bf 100644 --- a/containers/galactic-gateway/Dockerfile +++ b/containers/galactic-gateway/Dockerfile @@ -39,7 +39,7 @@ COPY cmd/ cmd/ COPY internal/ internal/ # galactic-gateway transitively imports both internal/plumbing/ebpf/edgeprog -# (the edge NAT+LB XDP program it loads directly) and, via +# (the edge DSR load-balancer XDP program it loads directly) and, via # internal/plumbing/ebpf/edgepreflight, internal/plumbing/ebpf/prog (the # SRv6 uSID TC-BPF program) -- neither package's bpf2go output # (*_bpfel.go/.o, *_bpfeb.go/.o) is committed to git (see each package's diff --git a/containers/galactic-nat/Dockerfile b/containers/galactic-nat/Dockerfile index 75ac9dfc..62dc3694 100644 --- a/containers/galactic-nat/Dockerfile +++ b/containers/galactic-nat/Dockerfile @@ -48,7 +48,7 @@ COPY internal/ internal/ # via internal/plumbing/ebpf/natattach. # - prog -- the SRv6 uSID TC-BPF program, via # internal/plumbing/ebpf/attach and .../usidmap. -# - edgeprog -- the edge NAT+LB DSR XDP program, via internal/gateway +# - edgeprog -- the edge DSR load-balancer XDP program, via internal/gateway # and internal/controller -> .../edgemap. galactic-nat # never loads it, but shares those packages with # galactic-gateway, so it links against the bindings. diff --git a/docs/README.md b/docs/README.md index 24e78d8b..699c41cc 100644 --- a/docs/README.md +++ b/docs/README.md @@ -16,7 +16,7 @@ data flow, module reference, known constraints): - [agents/ARCHITECTURE-ROUTER.md](agents/ARCHITECTURE-ROUTER.md) — the BGP/EVPN control plane (`galactic-router`, embedded GoBGP, GC). - [agents/ARCHITECTURE-GATEWAY.md](agents/ARCHITECTURE-GATEWAY.md) — the edge - XDP DSR NAT+LB gateway (`galactic-gateway`, `NetworkGateway`/`NetworkRule`). + XDP DSR load balancer (`galactic-gateway`, `NetworkGateway`/`NetworkRule`). - [agents/ARCHITECTURE.md](agents/ARCHITECTURE.md) — superseded by the three documents above; kept only as a redirect for old links. - [agents/CONVENTIONS.md](agents/CONVENTIONS.md) — Go naming, error diff --git a/docs/agents/ARCHITECTURE-CNI.md b/docs/agents/ARCHITECTURE-CNI.md index eb8c4c94..d4936956 100644 --- a/docs/agents/ARCHITECTURE-CNI.md +++ b/docs/agents/ARCHITECTURE-CNI.md @@ -10,7 +10,7 @@ _Last updated: 2026-09-09_ This document covers the CNI side of Galactic only. See [ARCHITECTURE-ROUTER.md](ARCHITECTURE-ROUTER.md) for the BGP/EVPN control plane (`galactic-router`) and [ARCHITECTURE-GATEWAY.md](ARCHITECTURE-GATEWAY.md) -for the edge XDP NAT+LB gateway (`galactic-gateway`). This file, together +for the edge XDP DSR load balancer (`galactic-gateway`). This file, together with those two, supersedes the former monolithic `ARCHITECTURE.md` — see [AGENTS.md](../../AGENTS.md) for which document to start from for a given task. diff --git a/docs/agents/ARCHITECTURE-GATEWAY.md b/docs/agents/ARCHITECTURE-GATEWAY.md index 170bf7bf..4ff95ea5 100644 --- a/docs/agents/ARCHITECTURE-GATEWAY.md +++ b/docs/agents/ARCHITECTURE-GATEWAY.md @@ -8,7 +8,7 @@ > existing tenant EVPN mesh — there is no primary/secondary node, no BGP > local-preference split, and no address rewriting anywhere in the datapath. -_Last updated: 2026-09-09_ +_Last updated: 2026-10-06_ This document covers `galactic-gateway` and the `NetworkGateway`/ `NetworkRule` reconcilers only. See @@ -54,9 +54,7 @@ reconcilers) still runs unmodified in that separate `galactic-router` pod — see [ARCHITECTURE-ROUTER.md](ARCHITECTURE-ROUTER.md). The load-balancing design itself is **DSR (Direct Server Return) over a -Maglev consistent-hash ring**, replacing an earlier Full-NAT (DNAT+SNAT) -design entirely — a breaking change with no migration path, not a second -mode alongside the old one. The defining simplification: this datapath does +Maglev consistent-hash ring**. The defining simplification: this datapath does **no address or port rewriting at all**. A client's packet travels inside the SRv6 encapsulation byte-for-byte unmodified all the way to the backend; the backend replies to the client *directly*, never re-entering the @@ -77,16 +75,14 @@ Consequences that follow directly from dropping rewriting: - No `conn_table`/flow state of any kind, and nothing to garbage-collect beyond the eBPF verifier's own bookkeeping — DSR is fully stateless. - Full-NAT needed a flow table to remember which backend/SNAT port a flow - was assigned; Maglev/DSR re-derives the same answer from the same input - on every packet. -- No un-DNAT/un-SNAT branch — a reply carries the client's own addressing + Maglev re-derives the same backend from the same input on every packet. +- No reply-translation branch — a reply carries the client's own addressing already, so the return program forwards it rather than translating it, and needs no flow state to do so. - No L3/L4 checksum touch anywhere in the datapath — the packet's own checksum is already correct for its own, completely unmodified content. -- No Active-Active BGP local-preference model, no primary/secondary node - election, and no per-node self-address to publish — every gateway node's +- No BGP local preference, no primary/secondary node election, and no + per-node self-address to publish — every gateway node's route is equally preferred by construction (a distinct Route Distinguisher per originating node keeps every node's identical-prefix advertisement alive as an independent, non-competing route rather than @@ -107,16 +103,15 @@ per-VPC gateway design to colocate with the workload's own VRF. ### The two CRDs -| CRD | Scope | Written by | Purpose | -| ---------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `NetworkGateway` | Namespaced, one per gateway node (`spec.targetRef.name` = node name) | Operator, once per gateway node (see [worked example](#worked-containerlab-example) below) | Node-scoped root object, mirroring `BGPRouter`'s pattern. Identifies which nodes participate in the anycast mesh and surfaces each node's engine health via `status.conditions`. Carries **no** self-address field — DSR rewrites nothing, so there is no SNAT source to publish. | -| `NetworkRule` | Namespaced, tenant-writable | Tenant, via an admission webhook that verifies VPC/VPCAttachment ownership (**webhook not yet deployed in this repo** — see Known Constraints) | Ingress load-balancing spec: `vpcRef`/`vpcAttachmentRef` (opaque tenant identifiers), `vipAddresses` (1–8, IPv4/IPv6), `protocol` (`tcp`/`udp`), `port`, `backends` (1–64 `address:port` pairs). Served by every `NetworkGateway` in the namespace identically — there is **no** `status.primaryNode` field. | +| CRD | Scope | Written by | Purpose | +| ---------------- | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `NetworkGateway` | Namespaced, one per gateway node (`spec.targetRef.name` = node name) | Operator, once per gateway node (see [worked example](#worked-containerlab-example) below) | Node-scoped root object, mirroring `BGPRouter`'s pattern. Identifies which nodes participate in the anycast mesh and surfaces each node's engine health via `status.conditions`. Carries **no** self-address field — DSR rewrites nothing, so there is no translation source to publish. | +| `NetworkRule` | Namespaced, tenant-writable | Tenant, via an admission webhook that verifies VPC/VPCAttachment ownership (**webhook not yet deployed in this repo** — see Known Constraints) | Ingress load-balancing spec: `vpcRef`/`vpcAttachmentRef` (opaque tenant identifiers), `vipAddresses` (1–8, IPv4/IPv6), `protocol` (`tcp`/`udp`), `port`, `backends` (1–64 `address:port` pairs). Served by every `NetworkGateway` in the namespace identically. | Both are defined in `go.datum.net/network`'s `api/v1alpha1` package (`gateway_types.go`, `rule_types.go`) — the same external CRD module the BGP-family types live in. Both types' own doc comments describe this -DSR/anycast model directly (they were rewritten alongside this redesign, -not left describing the old Full-NAT behavior). +DSR/anycast model directly. --- @@ -138,8 +133,7 @@ galactic/ │ │ # encap-source address │ ├── gateway/ # Engine, Datapath/QuotaEnforcer/TelemetryEmitter │ │ # interfaces + real implementations, crash -│ │ # recovery — no VRF/Geneve state, no -│ │ # primary/secondary placement +│ │ # recovery — no VRF/Geneve state │ ├── maglev/ # Pure-Go Maglev consistent-hash lookup table │ │ # (internal/maglev/table.go) — this binary's │ │ # only importer; galactic-nat has no @@ -185,12 +179,12 @@ public-interface/SRv6-address values. ### Worked ContainerLab example -The containerlab lab no longer deploys `galactic-gateway`; its edge nodes -run only the egress shards, attached directly to their uplinks. Until commit -`abdd665b`, `deploy/containerlab/resources/galactic-gateway/` ran this role -on four edge nodes across all three lab clusters — `dfw-worker2`/`dfw-worker3` (an -active-active pair), `sjc-worker2`, and `iad-worker2`. Each node's overlay -directory, named for the node itself, carries: +The containerlab lab no longer deploys `galactic-gateway`; its edge nodes run +only the egress shards, attached directly to their uplinks. Until commit +`abdd665b`, `deploy/containerlab/resources/galactic-gateway/` ran this role on +four edge nodes across all three lab clusters — `dfw-worker2`/`dfw-worker3`, +`sjc-worker2`, and `iad-worker2`. Each node's overlay directory, named for the +node itself, carries: - `node-patch.yaml` — pins the DaemonSet to one node via `kubernetes.io/hostname` and sets `GALACTIC_GATEWAY_PUBLIC_INTERFACE` @@ -231,16 +225,14 @@ that converges this node's whole gateway engine: different Route Distinguisher per originating node, not a decap Function, is what keeps every node's route alive as an independent, non-competing path). **No `LocalPreference` is set** — every gateway - node's route is equally preferred by construction, unlike the removed - Full-NAT design's primary/secondary local-pref split. + node's route is equally preferred by construction. 3. **Crash recovery.** Runs `Engine.ReconcileOrphans` (see [Crash recovery](#crash-recovery) below). `NetworkRuleReconciler.Reconcile` separately owns two **per-object** lifecycle pieces the aggregate pass above is the wrong place for: `status.conditions`'s `Accepted` condition (`updateAcceptedCondition` — set -once gateway nodes exist for the namespace; no `primary_node`-style -assignment exists anymore, since DSR has no primary node to assign) and the +once gateway nodes exist for the namespace) and the finalizer-guarded teardown ordering on deletion (withdraw the rule's `BGPAdvertisement`s *before* releasing quota/reservation state, so an in-flight flow is never blackholed through a route that has already @@ -259,11 +251,10 @@ to every object in the namespace on change (`ruleToGatewayRequests`, ### Packet path (`internal/plumbing/ebpf/edgeprog/edgedsr.c`, program `edge_lb`) IPv6-only, phase 1 scope (plain TCP/UDP, no extension headers). Attached to -the node's public/underlay-facing uplink. Unlike the removed Full-NAT -`edgenat.c`, there is no "is this a reply to me" direction check: a reply -never reaches this program, which sees only the forward half, so it has -exactly one branch, not two. Replies are `edge_return`'s, on a different -attach point — see below. +the node's public/underlay-facing uplink. There is no "is this a reply to me" +direction check: a reply never reaches this program, which sees only the +forward half, so it has exactly one branch, not two. Replies are +`edge_return`'s, on a different attach point — see below. 1. **Parse** the outer Ethernet + IPv6 header, then the L4 header (TCP or UDP only). Not IPv6, unparseable, or not TCP/UDP — unclaimed: handed to @@ -271,12 +262,10 @@ attach point — see below. interface carries their role (see [below](#sharing-the-xdp-hook-xdpdispatch)), otherwise `XDP_PASS` to the kernel stack, e.g. BGP/SSH to the node itself. Only the - source/destination port are ever read; nothing is rewritten, so there is - no pointer-to-field resolution the way a rewrite would need. -2. **Match** `(proto, dst port, dst addr)` against `vip_table` (keyed - identically to the removed `rule_table` — a VIP is globally unique by - construction, no tenant dimension). No match — unclaimed, as above (not - one of this gateway's VIPs). + source/destination port are ever read; nothing is rewritten. +2. **Match** `(proto, dst port, dst addr)` against `vip_table` (a VIP is + globally unique by construction, so the key has no tenant dimension). No + match — unclaimed, as above (not one of this gateway's VIPs). 3. **Claimed past this point** — every subsequent failure is a drop, not a pass-through (this gateway owns this VIP+port+protocol). Bump `vip_stats_table`'s hit counters (packets/bytes/last-seen), lazily @@ -293,28 +282,24 @@ attach point — see below. same backend. 5. **Push** a fresh 40-byte outer IPv6 header addressed to the chosen backend's own worker-node SRv6 uSID (`vip_table`'s per-backend field, - resolved by the Go control plane the same way any other cross-node - SRv6 destination is — see - [uSID resolution](#usid-resolution-for-backends) below), sourced from - this node's own `encap_config_table` entry (this node's plain - SRv6-reachable address — never a NAT/SNAT source and never compared - against anything on a receive path, since no reply ever re-enters this - program). Resolve the L2 next-hop via - `bpf_fib_lookup`, then leave over the interface that lookup selected: - `XDP_TX` where the route egresses the interface the client's packet - arrived on, `bpf_redirect` + `XDP_REDIRECT` where it does not. Those - differ on every gateway that reaches its clients and its compute nodes - over separate links, which is this role's normal shape — see - [Known Constraints](#known-constraints) for the bug that came from - returning `XDP_TX` unconditionally. The inner packet travels completely - unmodified — this is DSR's entire premise: no DNAT, no SNAT, no - checksum touch anywhere. + resolved by the Go control plane the same way any other cross-node SRv6 + destination is — see [uSID resolution](#usid-resolution-for-backends) + below), sourced from this node's own `encap_config_table` entry (this + node's plain SRv6-reachable address — never compared against anything on a + receive path, since no reply ever re-enters this program). Resolve the L2 + next-hop via `bpf_fib_lookup`, then leave over the interface that lookup + selected: `XDP_TX` where the route egresses the interface the client's + packet arrived on, `bpf_redirect` + `XDP_REDIRECT` where it does not. Those + differ on every gateway that reaches its clients and its compute nodes over + separate links, which is this role's normal shape — see [Known + Constraints](#known-constraints) for the bug that came from returning + `XDP_TX` unconditionally. The inner packet travels completely unmodified — + this is DSR's entire premise: no address or port rewriting and no checksum + touch anywhere. See `edgedsr.c`'s own header comment for the full byte-level walkthrough, -including the `EDGE_BARRIER_VAR` eBPF-verifier bounds-narrowing gotcha -carried over unchanged from the removed `edgenat.c`, and two real -wire-format bugs found and fixed in `push_outer_header` (inherited -unchanged from `edgenat.c`, not introduced by this rewrite): the pushed +including the `EDGE_BARRIER_VAR` eBPF-verifier bounds-narrowing workaround +and two wire-format bugs found and fixed in `push_outer_header`: the pushed outer IPv6 header's version nibble was left zeroed instead of set to `6`, and the outer header's `payload_len` undercounted the inner IPv6 header's own 40 bytes (a call site passed `ip6->payload_len` directly instead of @@ -486,13 +471,13 @@ normal while ingress traffic quietly stopped being intercepted at all. ### GatewayConfig environment variables (`internal/config/gateway.go`) -| Variable | Required | Default | Description | -| ----------------------------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -| `GALACTIC_GATEWAY_NODE_NAME` | Yes | — | Kubernetes node name | -| `GALACTIC_GATEWAY_PUBLIC_INTERFACE` | Yes | — | Public/underlay-facing uplink interface the XDP program attaches to. May name a Linux bonding master (`edgeattach.ResolveTargets` expands it to that bond's slaves — see [Known Constraints](#known-constraints)) | -| `GALACTIC_GATEWAY_SRV6_ADDRESS` | Yes | — | This node's own plain SRv6-reachable IPv6 address, used as the DSR outer-header encap source (`encap_config_table`) — never a NAT/SNAT source and never compared against anything on a receive path; must be a native IPv6 address (rejected if IPv4 or 4-in-6) | -| `GALACTIC_GATEWAY_METRICS_PORT` | No | `8081` | Prometheus metrics port | -| `GALACTIC_GATEWAY_GRPC_HEALTH_PORT` | No | `5181` | gRPC health check port | +| Variable | Required | Default | Description | +| ----------------------------------- | -------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `GALACTIC_GATEWAY_NODE_NAME` | Yes | — | Kubernetes node name | +| `GALACTIC_GATEWAY_PUBLIC_INTERFACE` | Yes | — | Public/underlay-facing uplink interface the XDP program attaches to. May name a Linux bonding master (`edgeattach.ResolveTargets` expands it to that bond's slaves — see [Known Constraints](#known-constraints)) | +| `GALACTIC_GATEWAY_SRV6_ADDRESS` | Yes | — | This node's own plain SRv6-reachable IPv6 address, used as the DSR outer-header encap source (`encap_config_table`) — never compared against anything on a receive path; must be a native IPv6 address (rejected if IPv4 or 4-in-6) | +| `GALACTIC_GATEWAY_METRICS_PORT` | No | `8081` | Prometheus metrics port | +| `GALACTIC_GATEWAY_GRPC_HEALTH_PORT` | No | `5181` | gRPC health check port | All three required fields are enforced by `GatewayConfig.Validate` at startup, not deferred to a later, less obvious kernel-datapath error — a @@ -519,9 +504,8 @@ not collide with any of the others': `GALACTIC_GATEWAY_SRV6_ADDRESS` is this gateway node's own plain SRv6-reachable address, used purely as the source of every outer header this node's `edge_lb` program pushes (`edge_return` pushes no header at -all, so it never reads this) — unlike the removed Full-NAT design's -identically-named field, it is never a NAT/SNAT source, never has -return-path significance (DSR has no return path through this node at +all, so it never reads this). It is never an address-translation source, +never has return-path significance (DSR has no return path through this node at all), and is never published to any CRD status (`NetworkGatewayStatus` carries no self-address field — see [The two CRDs](#the-two-crds) above). It is operator-supplied per gateway node today, with no in-cluster @@ -582,25 +566,25 @@ pod on that node is not a supported configuration. ## Module / Package Reference -| Package | Binary | Responsibility | Owns state | -| ------------------------------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | -| `internal/config` (`gateway.go`) | galactic-gateway | `GatewayConfig`: node name, ports, public interface, SRv6 encap-source address; three-tier CLI/env/default precedence via viper | No | -| `internal/controller` (`networkgateway_controller.go`) | galactic-gateway | `NetworkGatewayReconciler`: desired-state assembly, BGP wiring, orphan-crash recovery | No | -| `internal/controller` (`networkrule_controller.go`) | galactic-gateway | `NetworkRuleReconciler`: finalizer-guarded teardown ordering, `Accepted`-condition maintenance (`updateAcceptedCondition`) | No | -| `internal/controller` (`usidresolver.go`) | galactic-gateway | `backendSIDIndex`: resolves a `NetworkRule` backend address to the worker node's SRv6 uSID by matching against `BGPAdvertisement`/`BGPRouter`/`BGPVRFInstance` CRDs, verifying tenant ownership | No | -| `internal/gateway` (`engine.go`) | galactic-gateway | `Engine`: mutex-guarded convergence loop ("apply everything in desired, remove everything not in desired"), mirroring `GoBGPRuntime`'s shape | Yes (active-rule map) | -| `internal/gateway` (`types.go`) | galactic-gateway | `DesiredRule`/`DesiredBackend`/`EngineState`/`EngineStatus`/`RuleStatus` — the engine's own representation, assembled by the controllers above; `DesiredBackend` implements `internal/maglev.Backend` | No | -| `internal/gateway` (`datapath.go`, `kerneldatapath.go`) | galactic-gateway | `Datapath`/`QuotaEnforcer`/`TelemetryEmitter` interfaces; `KernelDatapath`, the real `Datapath` backed by `edgemap.VIPTable` over a loaded `edgeprog.EdgedsrObjects`, building a `internal/maglev.Table` per rule; `NoopDatapath` for tests | Yes (`vipKeysByName` bookkeeping) | -| `internal/gateway` (`quota.go`) | galactic-gateway | `NodeQuotaEnforcer` — real, coarse node-level admission caps (max rules/tenant, max total `vip_table` entries); `NoopQuotaEnforcer` for tests | Yes (in-memory reservation counters) | -| `internal/gateway` (`telemetry.go`) | galactic-gateway | `PrometheusTelemetryEmitter` — control-plane-drop counter only (no primary/secondary placement gauge — see Key Design Decisions); `NoopTelemetryEmitter` for tests | Yes (Prometheus metric state) | -| `internal/gateway` (`recovery.go`, `diff.go`) | galactic-gateway | `Engine.ReconcileOrphans` (crash recovery) and `diffRuleKeys` (the pure key-set diff both `Reconcile`/`ReconcileOrphans` build on) | No | -| `internal/maglev` (`table.go`) | galactic-gateway | Pure-Go Maglev consistent-hash lookup table (`New`/`Lookup`/`Backends`) — one `*Table` built per ring; this binary's only importer today (`galactic-nat` has no analogous shard-placement ring — see Known Constraints) | No | -| `internal/plumbing/ebpf/edgeprog` | galactic-gateway | Compiled XDP program (`edgedsr.c`, program `edge_lb`) + bpf2go-generated Go bindings (`EdgedsrObjects`) | No | -| `internal/plumbing/ebpf/edgemap` | galactic-gateway | `VIPTable`: `vip_table`/`vip_stats_table`/`vip_addr_table` read/write API, `Generation`/`Reconcile` crash-safety mechanism | Yes (via `KernelTable`) | -| `internal/plumbing/ebpf/edgeattach` | galactic-gateway | Load + native-XDP-only attach of the compiled program to one interface | Yes (pinned maps, held link) | -| `internal/plumbing/ebpf/edgemetrics` | galactic-gateway | Pull-based `prometheus.Collector` reading `vip_table`/`vip_stats_table`/`drop_reasons` live at every scrape | No | -| `internal/plumbing/ebpf/edgepreflight` | galactic-gateway | Startup kernel-capability check (`BPF_PROG_TYPE_XDP`, `BPF_MAP_TYPE_HASH`, kernel BTF, `bpf_xdp_adjust_head`) — no partial pass, no degraded fallback | No | -| `internal/plumbing/ebpf/xdpattach` | gateway, nat | Native-XDP attach across a target list, shared with `natattach`: per-NIC support check before any attach, each bond slave waited back into its aggregate before the next | No | +| Package | Binary | Responsibility | Owns state | +| ------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | +| `internal/config` (`gateway.go`) | galactic-gateway | `GatewayConfig`: node name, ports, public interface, SRv6 encap-source address; three-tier CLI/env/default precedence via viper | No | +| `internal/controller` (`networkgateway_controller.go`) | galactic-gateway | `NetworkGatewayReconciler`: desired-state assembly, BGP wiring, orphan-crash recovery | No | +| `internal/controller` (`networkrule_controller.go`) | galactic-gateway | `NetworkRuleReconciler`: finalizer-guarded teardown ordering, `Accepted`-condition maintenance (`updateAcceptedCondition`) | No | +| `internal/controller` (`usidresolver.go`) | galactic-gateway | `backendSIDIndex`: resolves a `NetworkRule` backend address to the worker node's SRv6 uSID by matching against `BGPAdvertisement`/`BGPRouter`/`BGPVRFInstance` CRDs, verifying tenant ownership | No | +| `internal/gateway` (`engine.go`) | galactic-gateway | `Engine`: mutex-guarded convergence loop ("apply everything in desired, remove everything not in desired"), mirroring `GoBGPRuntime`'s shape | Yes (active-rule map) | +| `internal/gateway` (`types.go`) | galactic-gateway | `DesiredRule`/`DesiredBackend`/`EngineState`/`EngineStatus`/`RuleStatus` — the engine's own representation, assembled by the controllers above; `DesiredBackend` implements `internal/maglev.Backend` | No | +| `internal/gateway` (`datapath.go`, `kerneldatapath.go`) | galactic-gateway | `Datapath`/`QuotaEnforcer`/`TelemetryEmitter` interfaces; `KernelDatapath`, the real `Datapath` backed by `edgemap.VIPTable` over a loaded `edgeprog.EdgedsrObjects`, building a `internal/maglev.Table` per rule; `NoopDatapath` for tests | Yes (`vipKeysByName` bookkeeping) | +| `internal/gateway` (`quota.go`) | galactic-gateway | `NodeQuotaEnforcer` — real, coarse node-level admission caps (max rules/tenant, max total `vip_table` entries); `NoopQuotaEnforcer` for tests | Yes (in-memory reservation counters) | +| `internal/gateway` (`telemetry.go`) | galactic-gateway | `PrometheusTelemetryEmitter` — control-plane-drop counter only; `NoopTelemetryEmitter` for tests | Yes (Prometheus metric state) | +| `internal/gateway` (`recovery.go`, `diff.go`) | galactic-gateway | `Engine.ReconcileOrphans` (crash recovery) and `diffRuleKeys` (the pure key-set diff both `Reconcile`/`ReconcileOrphans` build on) | No | +| `internal/maglev` (`table.go`) | galactic-gateway | Pure-Go Maglev consistent-hash lookup table (`New`/`Lookup`/`Backends`) — one `*Table` built per ring; this binary's only importer today (`galactic-nat` has no analogous shard-placement ring — see Known Constraints) | No | +| `internal/plumbing/ebpf/edgeprog` | galactic-gateway | Compiled XDP program (`edgedsr.c`, program `edge_lb`) + bpf2go-generated Go bindings (`EdgedsrObjects`) | No | +| `internal/plumbing/ebpf/edgemap` | galactic-gateway | `VIPTable`: `vip_table`/`vip_stats_table`/`vip_addr_table` read/write API, `Generation`/`Reconcile` crash-safety mechanism | Yes (via `KernelTable`) | +| `internal/plumbing/ebpf/edgeattach` | galactic-gateway | Load + native-XDP-only attach of the compiled program to one interface | Yes (pinned maps, held link) | +| `internal/plumbing/ebpf/edgemetrics` | galactic-gateway | Pull-based `prometheus.Collector` reading `vip_table`/`vip_stats_table`/`drop_reasons` live at every scrape | No | +| `internal/plumbing/ebpf/edgepreflight` | galactic-gateway | Startup kernel-capability check (`BPF_PROG_TYPE_XDP`, `BPF_MAP_TYPE_HASH`, kernel BTF, `bpf_xdp_adjust_head`) — no partial pass, no degraded fallback | No | +| `internal/plumbing/ebpf/xdpattach` | gateway, nat | Native-XDP attach across a target list, shared with `natattach`: per-NIC support check before any attach, each bond slave waited back into its aggregate before the next | No | --- @@ -620,21 +604,16 @@ pod on that node is not a supported configuration. ## Key Design Decisions -- **DSR (Direct Server Return), not Full-NAT.** The datapath does no - address/port rewriting at all: a client's packet is pushed inside an SRv6 - outer header toward the chosen backend's worker node completely - unmodified, and the backend replies to the client directly. This is a - breaking replacement of the earlier Full-NAT (DNAT+SNAT) design, not a - second mode alongside it — no migration path, per the redesign's - explicit decision to drop Full-NAT rather than grow a second personality. - Everything else in this section follows from that one choice. -- **Anycast, not Active-Active BGP local-preference.** Every gateway node - in a PoP advertises every accepted rule's VIPs at equal BGP preference — - there is no `AssignPrimaryNode`/`LocalPreference` primitive anywhere in - this codebase anymore (`internal/gateway/placement.go` and - `localpref.go`, and their tests, were deleted outright as part of this - rewrite). A distinct Route Distinguisher per originating node (RFC 4364 - §4.3.2, `internal/runtime/gobgp/paths.go`'s `deriveRD`) is what keeps +- **DSR (Direct Server Return).** The datapath does no address/port + rewriting at all: a client's packet is pushed inside an SRv6 outer header + toward the chosen backend's worker node completely unmodified, and the + backend replies to the client directly. Everything else in this section + follows from that one choice. +- **Anycast VIP advertisement.** Every gateway node in a PoP advertises + every accepted rule's VIPs at equal BGP preference; nothing in this + codebase sets a local preference or picks a primary node. A distinct + Route Distinguisher per originating node (RFC 4364 §4.3.2, + `internal/runtime/gobgp/paths.go`'s `deriveRD`) is what keeps every node's identical-prefix advertisement alive as an independent, non-competing route instead of BGP collapsing them to one best path — see the go/no-go anycast spike, `internal/runtime/gobgp/anycast_spike_test.go`. @@ -676,11 +655,9 @@ pod on that node is not a supported configuration. - **SRv6 encap-source address has no in-cluster derivation mechanism.** `GALACTIC_GATEWAY_SRV6_ADDRESS` is operator-supplied per gateway node today; nothing in this repo yet computes it automatically from a node's - own `BGPRouter` locator/node-ID. Unlike the removed Full-NAT design, this - value is never published to any CRD status — there is no reconcile step - analogous to the old `publishSelfAddress` at all, since DSR rewrites - nothing and so has no SNAT source that needs advertising as a - node-reachability route. + own `BGPRouter` locator/node-ID. The value is never published to any CRD + status: DSR rewrites nothing, so there is no translation source that + needs advertising as a node-reachability route. - **Quota/telemetry are real but deliberately coarse.** `NodeQuotaEnforcer` enforces two node-level admission caps entirely from control-plane state already held by `Engine` (no eBPF map read required): max `NetworkRule`s @@ -695,11 +672,9 @@ pod on that node is not a supported configuration. own per-packet counters are exposed separately by `edgemetrics`'s pull-based collector. - **Per-VIP hit counters live in their own map (`vip_stats_table`), not - `vip_table`.** The removed Full-NAT predecessor's `rule_table`/ - `rule_stats_table` split already established this convention (issue - #361: a control-plane `Register` read-modify-write racing the datapath's - own per-packet increments silently discarded whichever landed second). - `vip_table`/`vip_stats_table` keep the same split: `Register` is a blind + `vip_table`.** A control-plane `Register` read-modify-write racing the + datapath's own per-packet increments would silently discard whichever + landed second (issue #361). So `Register` is a blind overwrite of `vip_table` alone and never touches `vip_stats_table`, which `edgedsr.c` alone populates, lazily, on a VIP's first matching packet — so re-registering a VIP (e.g. every controller reconcile pass) can never @@ -724,6 +699,16 @@ pod on that node is not a supported configuration. [ARCHITECTURE-ROUTER.md](ARCHITECTURE-ROUTER.md) for what's left in that binary. +### History + +The gateway first shipped as a Full-NAT datapath (`edgenat.c`) with +primary/secondary BGP local-preference placement (`AssignPrimaryNode`, +`internal/gateway/placement.go`, `localpref.go`) and a per-node +self-address route (`publishSelfAddress`). +[#427](https://github.com/datum-cloud/galactic/pull/427) replaced it with +the DSR/anycast design described here, with no migration path, and deleted +that code. + --- ## Testing @@ -774,8 +759,8 @@ that tag into `config/galactic-gateway/base`. - **`XDP_REDIRECT` into a veth needs NAPI enabled on the *peer*, which is a lab-only concern.** `edgedsr.c` returns `XDP_REDIRECT` whenever the route to the backend egresses an interface other than the ingress one. In native mode that calls the egress device's `ndo_xdp_xmit`, and veth's implementation silently discards the frame unless the peer end has NAPI enabled — which for a veth means the peer runs its own XDP program or has GRO turned on. A containerlab peer inside an FRR/transit container has neither by default, so a redirect there can fail exactly the way this datapath's original bug did: no drop counter moves, because `bpf_redirect()` itself succeeded and the discard happens later in `xdp_do_redirect`. Both programs are affected, and `edge_return` more so: it attaches to compute-facing links, which are veth pairs in the lab and may be veth or a plain NIC in production. `bpftool prog tracelog` and the `xdp:xdp_redirect_err` tracepoint are the diagnostics; `ethtool -K gro on` is the lab fix. A production uplink (physical NIC, or a bond slave per the bullet above) implements `ndo_xdp_xmit` natively and has no peer to ask about. This is the redirect-side analogue of the `XDP_TX`-on-veth observability quirk noted above. - **The return path is opt-in per node and fails closed at startup, not silently.** `edge_return` attaches only where `GALACTIC_GATEWAY_INTERNAL_INTERFACES` names an interface; a node that needs it and does not set it drops every reply in `KUBE-FORWARD` with nothing to say why, exactly as before this program existed. Where it *is* set, an attach failure is fatal to `setupGatewayDatapath` and the pod crash-loops rather than running with the forward half working and replies dying — the same all-or-nothing choice `edgeattach.Attach` already makes across a bond's slaves. - **A backend in another site replies through that site's edge node.** The gateway can pick a backend anywhere its `NetworkRule` reaches, and the reply then leaves through the edge node in front of *that* backend, whose own `vip_addr_table` must hold the same anycast VIP or the reply meets the unmodified kernel path and dies. Within one cluster the reconciler guarantees this; across clusters it depends on the same `NetworkRule` existing on both sides, which nothing in this repo enforces. -- **`vip_table` has no active GC beyond crash-recovery reconcile.** By design (see Key Design Decisions above) — DSR keeps no flow state to leak in the first place, unlike the removed Full-NAT predecessor's `conn_table`, which relied on `BPF_MAP_TYPE_LRU_HASH` self-eviction for the same purpose. -- **Egress is out of this binary's scope, not unimplemented.** An earlier plan (`docs/plans/865-edge-gateway-nat66-egress.md`) proposed adding a second, egress-masquerading XDP personality to this same program and process; that approach was superseded by a separate, sharded stateful egress translation tier (`galactic-nat`, its own binary — see `cmd/galactic-nat` and `internal/controller/egressshard_controller.go`) rather than built here. `NetworkRule`/this datapath remain ingress-only: external client → VIP → tenant backend. +- **`vip_table` has no active GC beyond crash-recovery reconcile.** By design (see Key Design Decisions above) — DSR keeps no flow state to leak in the first place. +- **Egress is out of this binary's scope, not unimplemented.** Egress translation is a separate, sharded tier (`galactic-nat`, its own binary — see `cmd/galactic-nat` and `internal/controller/egressshard_controller.go`). `NetworkRule`/this datapath remain ingress-only: external client → VIP → tenant backend. --- @@ -806,7 +791,7 @@ that tag into `config/galactic-gateway/base`. **Non-obvious patterns:** - `gatewayDatapathKeepAlive` (`cmd/galactic-gateway/gateway.go`) intentionally never calls `Close` on the loaded eBPF objects or the XDP `link.Link` — see that var's doc comment for the live incident this guards against (silent GC-triggered detach with every control-plane signal still looking healthy). - Every gateway node reconciles every `NetworkGateway`/`NetworkRule` in the namespace — there is no leader election and no per-node filtering predicate on the watch itself; filtering happens inside `Reconcile` (`gw.Spec.TargetRef.Name != r.NodeName` early-return) and via `isGatewayNode`/broadcast watch mappers, not via `SetupWithManager` predicates. -- `BGPAdvertisement` names for gateway-originated routes are node-qualified (`--v4`/`-v6`) specifically because the anycast model means every gateway node computes the same rule independently — omitting the node qualifier caused two nodes to race to create/update one shared object, confirmed live (`AlreadyExists` forever on the second node, only the first node's route ever advertised) under the earlier Full-NAT design and never reintroduced here. +- `BGPAdvertisement` names for gateway-originated routes are node-qualified (`--v4`/`-v6`) specifically because the anycast model means every gateway node computes the same rule independently — omitting the node qualifier caused two nodes to race to create/update one shared object, which would leave the second node failing with `AlreadyExists` forever and only the first node's route ever advertised. - A deleted `NetworkGateway` reconcile can't just call `Engine.Stop()` unconditionally — every gateway node's process reconciles every `NetworkGateway` in the namespace, so a *sibling* node's deletion reaches this reconciler too. `isGatewayNode` re-checks whether *this* node still has its own `NetworkGateway` before stopping the engine (issue #364). - Advertisement failures during `Reconcile` are collected, not returned immediately — one bad rule's BGP-wiring failure must not stop the rest of the pass, but a node that converged its engine while failing to publish any route must still report `AdvertisementFailed`, not `EngineHealthy` (issue #365) — see `readyConditionFor`. -- `withdrawNodeAdvertisements` matches only the `-v4`/`-v6` rule-advertisement name suffixes now — an earlier, Full-NAT-era version of this function also withdrew a `-selfaddr` self-address route; DSR's anycast model has no self-address to publish at all, so that name pattern no longer applies (see this file's package doc comment in `networkgateway_controller.go`). +- `withdrawNodeAdvertisements` matches only the `-v4`/`-v6` rule-advertisement name suffixes; the anycast model publishes no other per-node route (see the package doc comment in `networkgateway_controller.go`). diff --git a/docs/agents/ARCHITECTURE-ROUTER.md b/docs/agents/ARCHITECTURE-ROUTER.md index 07421595..bf3fb285 100644 --- a/docs/agents/ARCHITECTURE-ROUTER.md +++ b/docs/agents/ARCHITECTURE-ROUTER.md @@ -10,8 +10,8 @@ _Last updated: 2026-09-09_ This document covers `galactic-router`'s tenant-BGP core only. See [ARCHITECTURE-CNI.md](ARCHITECTURE-CNI.md) for the CNI attach chain that writes the CRDs this binary reconciles, and -[ARCHITECTURE-GATEWAY.md](ARCHITECTURE-GATEWAY.md) for the edge XDP NAT+LB -gateway (`galactic-gateway`, its own separate DaemonSet co-located on +[ARCHITECTURE-GATEWAY.md](ARCHITECTURE-GATEWAY.md) for the edge XDP DSR load +balancer (`galactic-gateway`, its own separate DaemonSet co-located on gateway-role nodes — not a `galactic-router` container in the same pod; that arrangement was retired when the gateway split into its own binary — see that doc for the `NetworkGateway`/`NetworkRule` reconcilers, which run @@ -268,7 +268,7 @@ co-located `galactic-gateway` container's own health port on the same - **RuntimeFactory pattern.** `galactic-router` always builds a `gobgp.NewRuntimeFactory(...)` at startup — GoBGP is the only backend. A `RuntimeFactory` is a plain function value (`func(types.NamespacedName) (RouterRuntime, error)`), so a future backend still only needs a new factory function and a call-site change here, not a controller change. - **DEL is intentionally minimal everywhere in the CNI chain; GC reclaims shared state asynchronously.** See [ARCHITECTURE-CNI.md](ARCHITECTURE-CNI.md#key-design-decisions) for the CNI-side half of this decision. `galactic-router`'s GC controller (ticker-driven, default every 5m) reclaims orphaned CRDs, stale kernel VRFs, and stale eBPF entries once no live container still references them. - **gRPC health, configurable port.** Liveness and readiness probes use the gRPC health protocol (`google.golang.org/grpc/health`) on a configurable port (default `5179`). No HTTP health endpoint. `galactic-gateway` follows the identical convention on its own port — see [ARCHITECTURE-GATEWAY.md](ARCHITECTURE-GATEWAY.md). -- **`galactic-router` carries no gateway-role code.** The edge XDP NAT+LB gateway used to be a `galactic-router` mode; it was split into its own `galactic-gateway` binary specifically so a crash on either side no longer takes the other down with it — see [ARCHITECTURE-GATEWAY.md](ARCHITECTURE-GATEWAY.md) for the full rationale and design. +- **`galactic-router` carries no gateway-role code.** The edge XDP load-balancing gateway used to be a `galactic-router` mode; it was split into its own `galactic-gateway` binary specifically so a crash on either side no longer takes the other down with it — see [ARCHITECTURE-GATEWAY.md](ARCHITECTURE-GATEWAY.md) for the full rationale and design. - **Peer FSM transitions are logged via a dedicated GoBGP event watcher, not GoBGP's own logger — and also drive real-time BGPPeer Kubernetes Events.** `GALACTIC_ROUTER`'s `LogLevel` config only gates GoBGP's *internal* `slog` logger, whose handler is wired to `io.Discard` regardless of level (`internal/runtime/gobgp/server.go`'s `buildServerOptions`) — so none of GoBGP's own peer state log lines ever reach this process's output. Instead, `internal/runtime/gobgp/peer_monitor.go` runs a shared `BgpServer.WatchEvent(..., WatchPeer())` goroutine (the same one-goroutine-per-runtime pattern `monitor.go`'s `watchEVPNRIB` uses for EVPN best-path) that diffs each peer's FSM state against `lastPeerState` and logs a `slog.Info` line on every real transition, including `disconnectReason`/`disconnectMessage` when a peer drops out of Established. The same transition is also handed to an optional `model.PeerStateObserver` (set via `gobgp.NewRuntimeFactory`'s `observer` parameter): in production this is `internal/controller/peer_events.go`'s `PeerStateEventEmitter`, a `manager.Runnable` that resolves the peer's address to its `BGPPeer` object and raises a `SessionEstablished`/`SessionDown` Kubernetes Event on it — but only for transitions into/out of Established, not every intermediate FSM hop (see the emitter's own doc comments for why). This is independent of, and does not replace, the periodic (`peerStatusRequeue`, 30s) `BGPPeer.Status` update in `internal/controller/bgprouter_controller.go`'s `updatePeerStatuses`: that keeps the CRD's status current on a poll and can miss a transition that reverts between two polls; the `slog` line and the Event are both real-time and don't have that gap. See `docs/plans/router-bgp-peer-session-events.md` for the full design. - **Routes for route-target-less EVPN paths defer to the underlay.** A path with no route target (the gateway's anycast VIPs, and each egress shard's SID `/64` and masquerade address) becomes a plain kernel route toward its BGP next hop, with no encapsulation. These routes live in their own table, `srv6.PlainRouteTable` (`0xFFFE0000`), marked `proto 250`, and are reached through two policy rules ahead of the main table's own: `32000: lookup main suppress_prefixlength 0`, then `32001: lookup 0xFFFE0000`. An underlay route for the same destination, or for anything more specific than the default route, therefore always wins. Keeping them out of the main table is required, not cosmetic: zebra imports every main-table route it did not install as a `kernel` route with distance 0, so a plain route there stopped FRR installing its own BGP route for the same prefix (#670). The kernel cannot resolve an IPv6 next hop recursively, so each route is written with the link and next hop the gateway resolves to at the time. `internal/runtime/gobgp/plainroutes.go` re-resolves every route whenever the main table changes and every 30s, and retries one that cannot resolve yet. It removes owned routes no path asks for once a two-minute startup grace has passed, and deletes the unmarked main-table route an earlier version left for each prefix. `ip route show table all proto 250` lists every route it owns. XDP datapaths that call `bpf_fib_lookup` with `BPF_FIB_LOOKUP_DIRECT` (`natprog`, `edgeprog`) read the main table only and do not see these routes. - **A tenant VRF route that fails to install is retried, not dropped.** `processEVPNPath` writes a VRF-scoped path's route into `egress_route_table` through `internal/runtime/gobgp/vrfroutes.go`. `Register` resolves the SID's uplink and neighbor MAC at write time, which fails while an uplink is still coming up after a restart, and BGP never redelivers an unchanged best path, so before #676 such a route stayed missing until the path changed upstream. `vrfRoutes` keeps each failed route in a pending set and retries it whenever the main table changes (1s debounce) and every 30s, until it installs or its path is withdrawn; removing a VRF drops its pending routes. One mutex serializes every write to the map from this runtime, taken per route during a retry so an unresolvable neighbor (up to 2s) does not hold up the RIB watcher for the whole pass. `EgressRouteTable.Refresh` cannot cover this: it walks existing entries only. The pending count, and the plain-route reconciler's failing count, are exported as `galactic_router_route_install_failing{kind="vrf"|"plain"}`, and `GalacticRouterRoutesNotInstalled` fires when either stays above 0 for 10 minutes. diff --git a/docs/agents/ARCHITECTURE.md b/docs/agents/ARCHITECTURE.md index 43b4db34..5a94ca82 100644 --- a/docs/agents/ARCHITECTURE.md +++ b/docs/agents/ARCHITECTURE.md @@ -10,11 +10,11 @@ Galactic now ships four binaries per node (the third and fourth only on dedicated gateway-role `edge` nodes); three of them have their own architecture document: -| Document | Covers | -| -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | +| Document | Covers | +| -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | [ARCHITECTURE-CNI.md](ARCHITECTURE-CNI.md) | The CNI attach chain — `galactic-cni` (installer), `galactic-veth`, `galactic-tap`, `galactic-ipam`, `galactic-bgp`, `galactic-route` | -| [ARCHITECTURE-ROUTER.md](ARCHITECTURE-ROUTER.md) | The BGP/EVPN control plane — `galactic-router` | -| [ARCHITECTURE-GATEWAY.md](ARCHITECTURE-GATEWAY.md) | The edge XDP NAT+LB gateway — `galactic-gateway`, `NetworkGateway`/`NetworkRule` | +| [ARCHITECTURE-ROUTER.md](ARCHITECTURE-ROUTER.md) | The BGP/EVPN control plane — `galactic-router` | +| [ARCHITECTURE-GATEWAY.md](ARCHITECTURE-GATEWAY.md) | The edge XDP DSR load balancer — `galactic-gateway`, `NetworkGateway`/`NetworkRule` | The fourth, `galactic-nat` (sharded stateful egress translation, `EgressShard`), has no architecture document of its own yet — see diff --git a/docs/architecture/containers.png b/docs/architecture/containers.png index 207536b0..a183a3a2 100644 Binary files a/docs/architecture/containers.png and b/docs/architecture/containers.png differ diff --git a/docs/architecture/containers.puml b/docs/architecture/containers.puml index ec1c5ba1..cb23292e 100644 --- a/docs/architecture/containers.puml +++ b/docs/architecture/containers.puml @@ -15,8 +15,8 @@ System_Boundary(galactic, "Galactic") { Container(router, "galactic-router", "Go, controller-runtime", "Reconciles BGPRouter/BGPPeer/BGPAdvertisement/BGPPolicy/BGPVRFInstance CRDs; runs orphaned-CRD/VRF GC; own DaemonSet on every compute *and* gateway-role node") Container(gobgp, "Embedded GoBGP", "osrg/gobgp v4 (in-process library)", "Tenant-role BGP speaker; EVPN path origination, per-node RIB (ephemeral)") - Container(gateway, "galactic-gateway", "Go, controller-runtime", "Reconciles NetworkGateway/NetworkRule CRDs; Active-Active BGP placement; own single-container DaemonSet on gateway-role nodes") - Container(edgebpf, "Edge XDP NAT+LB datapath", "XDP (cilium/ebpf)", "DSR (Direct Server Return) load-balancing over a Maglev consistent-hash ring on the gateway node's public interface") + Container(gateway, "galactic-gateway", "Go, controller-runtime", "Reconciles NetworkGateway/NetworkRule CRDs; anycast VIP advertisement; own single-container DaemonSet on gateway-role nodes") + Container(edgebpf, "Edge XDP DSR datapath", "XDP (cilium/ebpf)", "DSR (Direct Server Return) load-balancing over a Maglev consistent-hash ring on the gateway node's public interface") Container(nat66, "galactic-nat", "Go, controller-runtime", "Reconciles EgressShard CRD; publishes a BGPAdvertisement for this shard's SID (its covering /64, one destination per tenant Argument) + public address (/128); own DaemonSet on every compute node") Container(nat66bpf, "NAT66 egress XDP datapath", "XDP (cilium/ebpf)", "Sharded, stateful SNAT/un-SNAT for tenant egress traffic on the shard's uplink interface") diff --git a/docs/architecture/context.png b/docs/architecture/context.png index d23d8065..4f510b61 100644 Binary files a/docs/architecture/context.png and b/docs/architecture/context.png differ diff --git a/docs/architecture/context.puml b/docs/architecture/context.puml index 0e41733c..e2526e71 100644 --- a/docs/architecture/context.puml +++ b/docs/architecture/context.puml @@ -26,11 +26,11 @@ Rel(client, galactic, "Connects to a NetworkGateway VIP:port", "IPv6") Rel(compute, galactic, "Relies on for pod/VM VPC-attached networking", "SRv6, EVPN") Rel(connectors, galactic, "Relies on for EVPN route distribution between sites", "SRv6, EVPN") -Rel(gateways, galactic, "Relies on for edge NAT+LB and Active-Active BGP placement", "XDP, EVPN") +Rel(gateways, galactic, "Relies on for edge DSR load balancing and anycast VIP advertisement", "XDP, EVPN") Rel(galactic, k8s, "Watches / writes BGP and Gateway CRDs", "controller-runtime, bare client") Rel(vpcop, galactic, "Pre-populates VPC/VPCAttachment identifiers", "CNI config / NAD annotation") Rel(galactic, fabric, "Peers over eBGP underlay", "eBGP, port 179") -Rel(galactic, rr, "Distributes tenant EVPN + Active-Active gateway routes", "iBGP, port 1790") +Rel(galactic, rr, "Distributes tenant EVPN + anycast gateway VIP routes", "iBGP, port 1790") @enduml diff --git a/docs/gateway/configuration.md b/docs/gateway/configuration.md index 057d5132..44eace8e 100644 --- a/docs/gateway/configuration.md +++ b/docs/gateway/configuration.md @@ -1,7 +1,7 @@ # Gateway Deployment & Configuration This is a how-to/reference guide for deploying and configuring -`galactic-gateway`, the edge XDP Maglev/DSR NAT+LB gateway control plane. For +`galactic-gateway`, the edge XDP Maglev/DSR L4 load-balancer control plane. For the design rationale (why DSR, why anycast, why no VRF dependency) see [docs/agents/ARCHITECTURE-GATEWAY.md](../agents/ARCHITECTURE-GATEWAY.md) — this document only covers the "how", not the "why", and cross-links back to @@ -36,11 +36,7 @@ each from its own slot (`GALACTIC_GATEWAY_XDP_ATTACH=dispatch` here, `GALACTIC_NAT_XDP_ATTACH=dispatch` there). See [docs/nat/configuration.md](../nat/configuration.md). See [ARCHITECTURE-GATEWAY.md](../agents/ARCHITECTURE-GATEWAY.md) for the full -design (DSR vs. the removed Full-NAT design, the anycast BGP model, the -XDP packet path) and -[gateway-ingress-packet-trace.md](gateway-ingress-packet-trace.md) for a -packet-level sequence diagram (currently describing the earlier Full-NAT -datapath — see that file's own header for the caveat). +design (why DSR, the anycast BGP model, the XDP packet path). ## Prerequisite: node labeling @@ -116,29 +112,20 @@ node is not a supported configuration. `config/galactic-gateway/base/` is intentionally excluded from `config/galactic-gateway/`'s own kustomization and must never be applied -directly — doing so produces a crash-looping `galactic-gateway` container. -The reason is `GALACTIC_GATEWAY_SRV6_ADDRESS`: it is this node's own plain -SRv6-reachable IPv6 address, used purely as the source address of every -outer header the node's `edge_lb` XDP program pushes -(`edgedsr.c`'s `encap_config_table`) — it is **never** a NAT/SNAT source and -has no return-path significance, unlike the identically-named field in the -now-removed Full-NAT design. Because DSR rewrites nothing, there is no -reconcile step that derives or publishes this value automatically (no -analogue of a "self-address" status field exists on `NetworkGateway` at -all — see that CRD's own doc comment). It must be unique per gateway node -and is operator-supplied today, with no in-cluster derivation mechanism. -`GALACTIC_GATEWAY_PUBLIC_INTERFACE` is deployment-specific for the same -reason — every node's public/underlay-facing uplink interface name can -differ. - -> A few older comments in this repo (`config/galactic-gateway/base/daemonset.yaml`, -> `AGENTS.md`, `README.md`) attribute this constraint to a `publishSelfAddress` -> doc comment on `internal/controller/networkgateway_controller.go`. That -> function was removed as part of the DSR/anycast rewrite — it belonged to -> the earlier Full-NAT design, which *did* publish a per-node self-address -> route. The underlying constraint (unique per-node SRv6 address, no -> in-cluster derivation) still holds; it's just no longer explained by a -> function of that name. The authoritative explanation today is +directly — doing so produces a crash-looping `galactic-gateway` container. The +reason is `GALACTIC_GATEWAY_SRV6_ADDRESS`: it is this node's own plain +SRv6-reachable IPv6 address, used purely as the source address of every outer +header the node's `edge_lb` XDP program pushes (`edgedsr.c`'s +`encap_config_table`). It is not an address-translation source and has no +return-path significance. Because DSR rewrites nothing, there is no reconcile +step that derives or publishes this value automatically (no analogue of a +"self-address" status field exists on `NetworkGateway` at all — see that CRD's +own doc comment). It must be unique per gateway node and is operator-supplied +today, with no in-cluster derivation mechanism. +`GALACTIC_GATEWAY_PUBLIC_INTERFACE` is deployment-specific for the same reason +— every node's public/underlay-facing uplink interface name can differ. + +> The authoritative explanation of this constraint is > `internal/config/gateway.go`'s `EnvGatewaySRv6Address` doc comment and > ARCHITECTURE-GATEWAY.md's ["SRv6 encap-source address"](../agents/ARCHITECTURE-GATEWAY.md#srv6-encap-source-address) > section. @@ -582,9 +569,6 @@ access. constraints. - [docs/node-labels.md](../node-labels.md) — the full node-labeling strategy shared across every Galactic component. -- [docs/gateway/gateway-ingress-packet-trace.md](gateway-ingress-packet-trace.md) — - packet-level sequence diagrams (currently describing the earlier - Full-NAT datapath). - [docs/router/configuration.md](../router/configuration.md) — the `GALACTIC_ROUTER_*` environment variables the co-located `galactic-router` pod on the same gateway node also reads. diff --git a/internal/controller/networkgateway_controller.go b/internal/controller/networkgateway_controller.go index d8292ed6..21945f0e 100644 --- a/internal/controller/networkgateway_controller.go +++ b/internal/controller/networkgateway_controller.go @@ -38,7 +38,7 @@ type GatewayEngine interface { // ReconcileOrphans. DatapathGeneration() uint64 - // ReconcileOrphans removes rule_table state left behind by a mid-reconcile + // ReconcileOrphans removes vip_table state left behind by a mid-reconcile // crash. cutoff must come from DatapathGeneration, read before desired's // NetworkRule CRDs were listed, so an entry written during the listing // survives. @@ -233,7 +233,7 @@ func (r *NetworkGatewayReconciler) Reconcile(ctx context.Context, req ctrl.Reque rule := &ruleList.Items[i] if !rule.DeletionTimestamp.IsZero() { // Being torn down: excluded from desired state immediately, so - // this node's rule_table converges toward gone without waiting on + // this node's vip_table converges toward gone without waiting on // NetworkRuleReconciler's finalizer to finish withdrawing BGP. continue } @@ -285,12 +285,12 @@ func (r *NetworkGatewayReconciler) Reconcile(ctx context.Context, req ctrl.Reque logger.Error(updateErr, "update NetworkGateway status") } - // Crash recovery. A failed sweep leaves orphaned rule_table state behind + // Crash recovery. A failed sweep leaves orphaned vip_table state behind // until a later pass succeeds, so it is returned for retry, after the // status write above so the failure stays visible on the object. if err := r.Engine.ReconcileOrphans(ctx, desired, cutoff); err != nil { - logger.Error(err, "reconcile orphaned rule_table state") - return ctrl.Result{}, errors.Join(advErr, fmt.Errorf("reconcile orphaned rule_table state: %w", err)) + logger.Error(err, "reconcile orphaned vip_table state") + return ctrl.Result{}, errors.Join(advErr, fmt.Errorf("reconcile orphaned vip_table state: %w", err)) } return ctrl.Result{}, advErr @@ -530,10 +530,10 @@ func (r *NetworkGatewayReconciler) SetupWithManager(mgr ctrl.Manager) error { } // ruleToGatewayRequests maps a NetworkRule change to every NetworkGateway in -// its namespace. A NetworkRule carries no gatewayRef, and under the -// Active-Active model every gateway node in the rule's PoP must re-evaluate its -// own engine state whenever any rule changes, so the broadcast is intentional -// rather than a missing index. +// its namespace. A NetworkRule carries no gatewayRef, and every gateway node +// advertises every VIP identically (anycast), so every gateway node in the +// rule's PoP must re-evaluate its own engine state whenever any rule changes. +// The broadcast is intentional rather than a missing index. func ruleToGatewayRequests(ctx context.Context, c client.Client, obj client.Object) []ctrlreconcile.Request { rule, ok := obj.(*bgpv1alpha1.NetworkRule) if !ok { diff --git a/internal/controller/networkgateway_controller_test.go b/internal/controller/networkgateway_controller_test.go index 76c4f94e..d8327f10 100644 --- a/internal/controller/networkgateway_controller_test.go +++ b/internal/controller/networkgateway_controller_test.go @@ -33,8 +33,7 @@ var ( ) // fakeGatewayEngine records the EngineState passed to Reconcile so tests can -// assert on exactly which rules were included/excluded and at what -// local-pref. +// assert on exactly which rules were included or excluded. type fakeGatewayEngine struct { mu sync.Mutex lastDesired gateway.EngineState @@ -214,11 +213,8 @@ func TestNetworkGatewayReconciler_IgnoresDeletionOfOtherNodesGateway(t *testing. // TestNetworkGatewayReconciler_BuildsDesiredStateForAcceptedRules covers // DSR's anycast model directly: every accepted, non-deleting NetworkRule in // the namespace goes into desired state, with no primary/secondary -// distinction to gate on (an earlier, Full-NAT-era version of this test, -// "...ForAcceptedAssignedRules", also required status.primaryNode to be -// set -- that field and the active-passive model it implemented no longer -// exist; every gateway node in a PoP now serves every accepted rule -// identically). +// distinction to gate on: every gateway node in a PoP serves every accepted +// rule identically. func TestNetworkGatewayReconciler_BuildsDesiredStateForAcceptedRules(t *testing.T) { scheme := newRuleTestScheme(t) gwA := newTestGateway(testNodeGWA) @@ -344,11 +340,8 @@ func TestNetworkGatewayReconciler_SkipsBGPAdvertisementWiringWithoutRouter(t *te } // TestNetworkGatewayReconciler_CreatesBGPAdvertisement covers the anycast -// BGPAdvertisement shape directly: no LocalPreference is set at all (an -// earlier, Full-NAT-era version of this test, -// "...WithComputedLocalPref", asserted a computed primary/secondary -// preference value here — that model no longer exists; every gateway -// node's route is equally preferred by construction, see +// BGPAdvertisement shape directly: no LocalPreference is set at all, since +// every gateway node's route is equally preferred by construction (see // networkgateway_controller.go's applyBGPAdvertisements doc comment). func TestNetworkGatewayReconciler_CreatesBGPAdvertisement(t *testing.T) { scheme := newRuleTestScheme(t) @@ -576,14 +569,11 @@ func newAdvertisement(name string) *bgpv1alpha1.BGPAdvertisement { // TestNetworkGatewayReconciler_WithdrawsAdvertisementsForDepartedGatewayNode // is the regression test for #406: gw-b's NetworkGateway is deleted while -// gw-b's own per-rule BGPAdvertisement routes are still around (an -// earlier, Full-NAT-era version of this test also seeded a self-address -// route -- DSR's anycast model has no self-address to advertise at all, so -// that fixture no longer applies here). gw-a's process -- the only one -// left to react, since gw-b's own process is presumably already gone -- -// must withdraw every one of them on the NotFound reconcile it receives -// for gw-b's deletion, without touching gw-a's own advertisements for the -// same rule. +// gw-b's own per-rule BGPAdvertisement routes are still around. gw-a's +// process -- the only one left to react, since gw-b's own process is +// presumably already gone -- must withdraw every one of them on the NotFound +// reconcile it receives for gw-b's deletion, without touching gw-a's own +// advertisements for the same rule. func TestNetworkGatewayReconciler_WithdrawsAdvertisementsForDepartedGatewayNode(t *testing.T) { scheme := newRuleTestScheme(t) gwA := newTestGateway(testNodeGWA) // gw-a's own gateway; still exists diff --git a/internal/controller/networkrule_controller.go b/internal/controller/networkrule_controller.go index cefa64db..20625eab 100644 --- a/internal/controller/networkrule_controller.go +++ b/internal/controller/networkrule_controller.go @@ -188,7 +188,7 @@ func (r *NetworkRuleReconciler) reconcileDelete( rule.Namespace, rule.Name, err) } - logger.Info("NetworkRule BGP route withdrawn; rule_table teardown proceeds independently on each gateway node", + logger.Info("NetworkRule BGP route withdrawn; vip_table teardown proceeds independently on each gateway node", "networkRule", rule.Name) patchBase := rule.DeepCopy() diff --git a/internal/controller/usidresolver_test.go b/internal/controller/usidresolver_test.go index 01994ff7..c071745e 100644 --- a/internal/controller/usidresolver_test.go +++ b/internal/controller/usidresolver_test.go @@ -63,8 +63,7 @@ func TestBackendSIDIndex_NoMatchingPrefixIsError(t *testing.T) { // TestBackendSIDIndex_ExcludesAdvertisementsWithoutVRFIDOrFunction verifies // that an advertisement with no SRv6 VRFID/Function (e.g. a NetworkRule VIP -// preference advertisement, or a rule's own self-address advertisement -- -// see NetworkGatewayReconciler.applyBGPAdvertisements/publishSelfAddress) +// advertisement -- see NetworkGatewayReconciler.applyBGPAdvertisements) // is excluded from matching even if its prefix would otherwise contain the // address, since it carries no SRv6 decap behavior of its own. func TestBackendSIDIndex_ExcludesAdvertisementsWithoutVRFIDOrFunction(t *testing.T) { diff --git a/internal/plumbing/ebpf/edgeprog/dropreason.go b/internal/plumbing/ebpf/edgeprog/dropreason.go index dd314312..6a525c1e 100644 --- a/internal/plumbing/ebpf/edgeprog/dropreason.go +++ b/internal/plumbing/ebpf/edgeprog/dropreason.go @@ -11,9 +11,9 @@ package edgeprog // as a literal constant. // // A small set: direct server return has no connection table and no port -// allocation, so there is nothing analogous to a full-NAT datapath's -// state-related reasons. Both programs share these buckets -- a FIB or redirect -// failure means the same thing whichever direction hit it. +// allocation, so there are no state-related reasons. Both programs share +// these buckets -- a FIB or redirect failure means the same thing whichever +// direction hit it. const ( DropReasonEmptyBackendList uint32 = 0 DropReasonNoEncapConfig uint32 = 1 diff --git a/internal/plumbing/ebpf/edgeprog/edgedsr.c b/internal/plumbing/ebpf/edgeprog/edgedsr.c index f97f30b3..5ba9b0ea 100644 --- a/internal/plumbing/ebpf/edgeprog/edgedsr.c +++ b/internal/plumbing/ebpf/edgeprog/edgedsr.c @@ -18,10 +18,8 @@ // inside unmodified. The backend answers the client directly, so reply traffic // never re-enters this program. That has three consequences: // -// - No connection table. A full-NAT datapath needs one to remember which -// backend and translated port a flow was assigned, because that state must -// persist for the life of the connection. Here the same hash table produces -// the same backend for the same flow on every packet, with no state. +// - No connection table. The same hash table produces the same backend for +// the same flow on every packet, with no state. // - No return or decap branch, since this node never sees replies. // - No port allocation and no checksum touch anywhere in this file, the // packet's own checksum already being correct for its unmodified content. @@ -152,9 +150,8 @@ struct backend { __u8 usid[16]; }; -// struct vip_key is vip_table's key -- identical composition to edgenat.c's -// former rule_key (proto, dst port, VIP address). No tenant dimension: a -// VIP is globally unique by construction. +// struct vip_key is vip_table's key: proto, dst port, VIP address. No tenant +// dimension: a VIP is globally unique by construction. struct vip_key { __u8 proto; __u8 pad[1]; @@ -211,9 +208,9 @@ struct encap_config { __u8 encap_src[16]; }; -// Drop reason indices into the drop_reasons map -- a much smaller set than -// edgenat.c's former enum edge_drop_reason, since DSR has no conn_table, -// no PAT allocation, and no return/decap branch to fail in. +// Drop reason indices into the drop_reasons map -- a small set, since DSR has +// no connection table, no port allocation, and no return/decap branch to fail +// in. enum edge_drop_reason { DROP_REASON_EMPTY_BACKEND_LIST = 0, DROP_REASON_NO_ENCAP_CONFIG = 1, diff --git a/internal/runtime/gobgp/anycast_spike_test.go b/internal/runtime/gobgp/anycast_spike_test.go index 490984c8..1c78830c 100644 --- a/internal/runtime/gobgp/anycast_spike_test.go +++ b/internal/runtime/gobgp/anycast_spike_test.go @@ -24,9 +24,9 @@ import ( // before RT match, and rtIndex being 1:1 not fan-out. Reading those two // functions directly (see monitor.go) shows both are scoped to advertisements // that carry a VRFID/Function (tenant-VRF SEG6 kernel-route installation) — -// a gateway VIP/self-address advertisement carries neither (see -// publishSelfAddress's doc comment), so it never reaches that code path at -// all. Those two issues are real but irrelevant to this design. +// a gateway VIP advertisement carries neither (see +// NetworkGatewayReconciler.applyBGPAdvertisements), so it never reaches that +// code path at all. Those two issues are real but irrelevant to this design. // // What actually determines whether two gateway nodes' identical-VIP-prefix // advertisements coexist is buildEVPNPaths' RD derivation (paths.go's @@ -46,18 +46,17 @@ func TestAnycastSpike_TwoGatewayNodesSamePrefixSurviveIndependently(t *testing.T const vipPrefix = "203.0.113.5/32" gw1 := model.DesiredAdvertisement{ - Name: "gw1-selfaddr", + Name: "gw1-vip", AddressFamily: model.AddressFamily{AFI: afiL2VPN, SAFI: safiEVPN}, Prefixes: []string{vipPrefix}, NextHop: "2001:db8::1", // VRFID intentionally nil: this is the exact shape - // NetworkGatewayReconciler.publishSelfAddress produces for a VIP/ - // self-address advertisement — no VRFID/Function, per its own doc - // comment ("carries no VRFID/Function... not reached via a per-tenant - // VRF decap at all"). + // NetworkGatewayReconciler.applyBGPAdvertisements produces for a VIP + // advertisement, which carries no VRFID/Function and is not reached + // via a per-tenant VRF decap at all. } gw2 := model.DesiredAdvertisement{ - Name: "gw2-selfaddr", + Name: "gw2-vip", AddressFamily: model.AddressFamily{AFI: afiL2VPN, SAFI: safiEVPN}, Prefixes: []string{vipPrefix}, NextHop: "2001:db8::2",