privateip · Oct 7, 2026 · Oct 7, 2026 · Oct 7, 2026
@@ -91,12 +91,12 @@ See [docs/node-labels.md](docs/node-labels.md) for the full strategy — why `ga
   - **`config/galactic-router/overlays/router/`** — the plain (non-reflector) per-node role (`galactic-router`); runs on every `compute` *and* every `edge` node except Kubernetes control-plane nodes, opt-in via `galactic.datumapis.com/galactic: router` (the same label `galactic-cni` keys off of — see `config/galactic-cni/daemonset.yaml`). This is the default role, so it carries no role-specific name/label of its own — only `control` below needs one, to coexist as a second DaemonSet.
   - **`config/galactic-router/overlays/control/`** — the BGP route-reflector role (`galactic-router-rr`, `GALACTIC_ROUTER_REFLECTOR=true`); opt-in only, requires `galactic.datumapis.com/galactic: control` (stays at zero replicas otherwise). Mutually exclusive with `router`'s `galactic: router` value on the same node — one label key, single-valued, so a route-reflector node structurally can't also run plain-mode `galactic-router`. `GALACTIC_ROUTER_BGP_LOCAL_ADDRESS` is auto-detected from the host's `lo` interface by default; see the comments in `daemonset-patch.yaml` for when to override it.
   - Apply both roles with `kubectl apply -k config/galactic-router/`, or a single role with e.g. `kubectl apply -k config/galactic-router/overlays/router/`. `galactic-router` no longer runs bundled inside `galactic-gateway`'s pod — see `config/galactic-gateway/` below.
-- **`config/galactic-gateway/`** — the edge XDP DSR load-balancer control plane, a single-container `galactic-gateway` DaemonSet — `galactic-router` used to run as a second container in this same pod; it's now a fully independent DaemonSet (via `galactic.datumapis.com/galactic: router`, which a gateway node may carry like any other), so a crash on either side no longer takes the other's *pod* down with it, not just the other's binary. `config/galactic-gateway/{serviceaccount.yaml,rbac.yaml}` (safe/idempotent to apply cluster-wide) are what `kubectl apply -k config/galactic-gateway/` applies; `config/galactic-gateway/base/` (requiring nodes labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/gateway: enabled`, so it runs on edge nodes alongside `galactic-vrf`) is **not** included in that kustomization and is **not** applied as-is — the same exemption as `config/fabric-router/` below, for the same reason: `GALACTIC_GATEWAY_SRV6_ADDRESS` must be unique per gateway node and has no generic default (no in-cluster mechanism yet derives it automatically — see `internal/config/gateway.go`'s `EnvGatewaySRv6Address` doc comment). It's designed to be instantiated once per gateway node by a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's own public-interface/SRv6-address values; see `deploy/containerlab/resources/galactic-gateway/` as of commit `abdd665b` for a worked example (the containerlab lab no longer deploys the gateway). Also **not** part of the root `config/kustomization.yaml`'s default resource list, matching `config/fabric-router/`'s exemption.
+- **`config/galactic-gateway/`** — the edge XDP DSR load-balancer control plane, a single-container `galactic-gateway` DaemonSet — `galactic-router` used to run as a second container in this same pod; it's now a fully independent DaemonSet (via `galactic.datumapis.com/galactic: router`, which a gateway node may carry like any other), so a crash on either side no longer takes the other's *pod* down with it, not just the other's binary. `config/galactic-gateway/{serviceaccount.yaml,rbac.yaml}` (safe/idempotent to apply cluster-wide) are what `kubectl apply -k config/galactic-gateway/` applies; `config/galactic-gateway/base/` (requiring nodes labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/gateway: enabled`, so it runs on edge nodes alongside `galactic-vrf`) is **not** included in that kustomization and is **not** applied as-is — the same exemption as `config/fabric-router/` below, for the same reason: `GALACTIC_GATEWAY_SRV6_ADDRESS` must be unique per gateway node and has no generic default (no in-cluster mechanism yet derives it automatically — see `internal/config/gateway.go`'s `EnvGatewaySRv6Address` doc comment). It's designed to be instantiated once per gateway node by a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's own public-interface/SRv6-address values; see `deploy/containerlab/resources/galactic-gateway/` for a worked example (one overlay per lab edge node). Also **not** part of the root `config/kustomization.yaml`'s default resource list, matching `config/fabric-router/`'s exemption.
 - **`config/galactic-nat/`** — the sharded egress translation (NAT66/NAT64) control plane, one `galactic-nat` DaemonSet per cluster on every node labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/nat: enabled`, alongside `galactic-vrf`. It is independent of `galactic-gateway`. A shard's identity (SID, masquerade addresses, NAT64 prefix) comes from the `EgressShard` targeting its node, not from the DaemonSet. The base leaves `GALACTIC_NAT_XDP_ATTACH` at its default, `direct`. On a node that also runs the gateway, set it to `dispatch`: an interface takes one native XDP program, so the shard and the gateway (which defaults to `GALACTIC_GATEWAY_XDP_ATTACH=dispatch`) each run from their own slot of the node's shared, pinned XDP dispatcher (`internal/plumbing/ebpf/xdpdispatch`), and either can restart or be turned off (`GALACTIC_*_DATAPATH_ENABLED=false`) without touching the other's traffic. `hack/xdp-dispatch-release.sh` detaches the dispatcher from a node. Compute nodes reach their own site's shards over SRv6 via `GALACTIC_CNI_EGRESS_SHARD_SIDS`. `kubectl apply -k config/galactic-nat/` applies only the ServiceAccount/RBAC; `base/` is excluded because uplink auto-detection is deployment-specific (see `config/galactic-nat/kustomization.yaml`). Not part of the root `config/kustomization.yaml`.
 - **`config/fabric-router/`** — the FRR underlay eBGP DaemonSet (`fabric-router`; `galactic-router` needs a working underlay before it can start). Unlike `config/galactic-router/`, this is a single flat DaemonSet with no `default`/`rr`-style role split — its affinity matches any node labeled `galactic.datumapis.com/fabric: router` (a dedicated mode-enum label, independent of whatever `galactic.datumapis.com/node` role value or `galactic.datumapis.com/galactic` mode that node also carries) directly, since there's no env/config difference between running on a regular node vs. those roles — fabric-router itself is identical everywhere; only the per-node BGP underlay config it reads from the ConfigMap differs. That affinity can legitimately match more than one node per cluster, and BGP underlay config (hostname, router-id, interface addresses, remote-AS) inherently differs per physical node — so each node's config lives in its own `fabric-router.<nodename>` ConfigMap (a `frr.conf` key, plus optional `daemons`/`vtysh.conf` overrides of the image defaults) rather than one shared ConfigMap for the whole DaemonSet. `fabric-config-agent` (`cmd/fabric-config-agent`, `internal/fabricconfig`, built into the `fabric-router` image) reads it from the API: as the `frr-init` init container it validates (`vtysh -C`) and installs it, holding the pod in `Init` until a valid one exists; as the `config-agent` sidecar it applies later changes live with `frr-reload.py`, so an edit to one node's ConfigMap neither restarts the pod nor touches other nodes, and reports `Applied`/`Rejected`/`ReloadFailed`/`RestartRequired` events on the ConfigMap. The sidecar also hard-resets any IPv6 session announcing an IPv4-mapped next hop (`internal/fabricconfig/nexthop.go`) and serves that check's metrics on the node's port `9343`. A third container, `frr-exporter` (upstream `tynany/frr_exporter`, pinned by digest), reads the daemons' vty sockets in the same pod-scoped `/run/frr` and serves BGP peer state, message counters and prefix counts for both underlay families on the node's port `9342` (`hostNetwork`); `task verify:fabric-metrics` in `deploy/containerlab/` checks it on every lab node. The image's default `daemons` loads bgpd's BMP module (`-M bmp`), so a node's `frr.conf` can stream to a BMP collector (`config/bmp-collector/`) with a `bmp targets` block; loading it alone streams nothing, and a change to the modules needs a pod restart rather than a reload. Every lab node's `frr.conf` streams its pre-policy Adj-RIB-In and Loc-RIB for both families, sourced from `lo` (`source-interface lo`), since the underlay carries loopbacks but not link subnets. Production `frr.conf`s come from infra's renderer, which has to emit the same block for production nodes to stream. A node with no ConfigMap of its own falls back, with a `DeprecatedConfigSource` warning, to key `frr.conf.<nodename>` of the legacy shared `fabric-config` ConfigMap. **Not** part of the root `config/kustomization.yaml` and not covered by `kubectl apply -k config/` — unlike every other component here, it has no generic default: the deployer must hand-author one `fabric-router.<nodename>` ConfigMap per matching node before applying `kubectl apply -k config/fabric-router/` (which also applies the agent's ServiceAccount and namespaced Role).
 - **`config/bmp-collector/`** — a single-replica GoBMP Deployment and its `bmp-collector` Service (TCP `5000`), the BMP (RFC 7854) collector `galactic-router` and `fabric-router` stream to; `galactic-router` does so when `GALACTIC_ROUTER_BMP_STATIONS` names it (`bmp-collector.galactic-system.svc:5000`; see [docs/router/configuration.md](docs/router/configuration.md)). It parses every Peer Up/Down, route-monitoring and statistics message to JSON on stdout, so the cluster's log pipeline is its store; an overlay swapping `--dump=console` for `--kafka-server` publishes to Kafka instead. Opt-in, like `config/fabric-router/`: not part of the root `config/kustomization.yaml`, since nothing streams to it until a router is configured to. The containerlab lab runs one on `iad-worker3`'s host network (`deploy/containerlab/resources/bmp-collector/iad/`), because its three clusters cannot reach each other's Services, and points every node's `fabric-router` and the EVPN route reflector at it. Of the `galactic-router`s, only the reflector streams: its pre-policy Adj-RIB-In already holds every node's advertisements.
 - **`config/monitoring/`** — Prometheus Operator resources for BGP and NAT: a `fabric-router` PodMonitor (frr-exporter, port `frr-metrics`/`9342`, as `job=fabric-router`; fabric-config-agent, port `agent-metrics`/`9343`, as `job=fabric-config-agent`), a `galactic-router` PodMonitor covering both roles (port `metrics`/`9179`), a `galactic-nat` PodMonitor for the egress shards (port `metrics`/`9182`), each adding a `node` label, and the `galactic-bgp` PrometheusRule: underlay session down, administratively down for an hour, flapping, received prefixes halved, exporter collector or scrape failing, a session announcing an IPv4-mapped next hop, BGP routes not installed, the agent's metrics scrape down, a `galactic-router` BMP station or metrics scrape down, `galactic-router` routes failing to install, an egress shard's session table evicting live sessions, a single stalled NIC receive queue on a `galactic-nat` uplink (#673), and a `galactic-nat` metrics scrape down. Requires the `monitoring.coreos.com` CRDs, which this repo does not install, so it is not part of the root `config/kustomization.yaml`. `task test:alerts` unit-tests the rules with `promtool` (`config/monitoring/tests/`). Every alert carries what infra's alerting standard checks for (`service: galactic`, `team: connect`, `runbook_url` into infra's `docs/runbooks/galactic/README.md`). The PrometheusRule here is the tested source, not where the rules run on Datum's infra: there the PodMonitors are converted and scraped on each edge cluster, but edge vmalert never notifies, so infra evaluates a copy of the rules centrally. Change the rules here first, with their tests, then update infra's copy.
-- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an SRv6 transit mesh whose underlay is dual-stack — every transit link and loopback carries an IPv4 and an IPv6 address, with one BGP session per address family (the SRv6 data plane itself stays IPv6-only; tenant IPv4 rides inside the encapsulation). Each site has one `compute` worker and at least one `edge` worker (dfw has two); every worker has its own eBGP uplink to its site's transit router, with FRR running as a hostNetwork DaemonSet. `galactic-router` (plain mode) distributes EVPN paths over iBGP to one route reflector for the whole lab (`iad-worker3`, `config/galactic-router/overlays/control/`) — compute and edge nodes never peer directly. Every `edge` node runs an egress shard (`galactic-nat`, run from the node's XDP dispatcher, `GALACTIC_NAT_XDP_ATTACH=dispatch`); the lab deploys no `galactic-gateway`. Each site's compute node egresses only through its own site's edge shards. A standalone off-cluster nginx host on the fourth transit router gives tenant egress a real destination. See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands.
+- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an SRv6 transit mesh whose underlay is dual-stack — every transit link and loopback carries an IPv4 and an IPv6 address, with one BGP session per address family (the SRv6 data plane itself stays IPv6-only; tenant IPv4 rides inside the encapsulation). Each site has one `compute` worker and at least one `edge` worker (dfw has two); every worker has its own eBGP uplink to its site's transit router, with FRR running as a hostNetwork DaemonSet. `galactic-router` (plain mode) distributes EVPN paths over iBGP to one route reflector for the whole lab (`iad-worker3`, `config/galactic-router/overlays/control/`) — compute and edge nodes never peer directly. Every `edge` node runs an egress shard (`galactic-nat`) and a `galactic-gateway`, which share the node's uplinks through its XDP dispatcher (`GALACTIC_NAT_XDP_ATTACH=dispatch`). Each site's compute node egresses only through its own site's edge shards, and each site has its own VIP `/64`, served only through its own edge nodes to one `ns60` backend in that site. A standalone off-cluster host on the fourth transit router gives tenant egress a real destination and is the client for the VIP checks (`verify:gateway-ingress`). See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands.
 
 ## New Developer Entry Points
 

@@ -43,7 +43,7 @@ Each component can also be applied on its own, e.g. `kubectl apply -k config/gal
 Two components are **not** part of `kubectl apply -k config/` and must be applied separately, each with its own per-node prerequisite:
 
 - **`config/fabric-router/`** — the FRR underlay eBGP DaemonSet `galactic-router` depends on.
-- **`config/galactic-gateway/`** — the edge XDP DSR load-balancer control plane, a single-container `galactic-gateway` DaemonSet on nodes labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/gateway: enabled` (`galactic-router` runs there too, but as its own independent DaemonSet via `galactic.datumapis.com/galactic: router` — see `config/galactic-router/` above, not bundled into this pod). `kubectl apply -k config/galactic-gateway/` only installs the shared, cluster-safe ServiceAccount/RBAC; `config/galactic-gateway/base/` itself is a template meant to be instantiated once per gateway node by a further overlay (see `deploy/containerlab/resources/galactic-gateway/` as of commit `abdd665b`, the last before the containerlab lab stopped deploying the gateway, for a worked example) — apply that overlay per node instead of `base/` directly.
+- **`config/galactic-gateway/`** — the edge XDP DSR load-balancer control plane, a single-container `galactic-gateway` DaemonSet on nodes labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/gateway: enabled` (`galactic-router` runs there too, but as its own independent DaemonSet via `galactic.datumapis.com/galactic: router` — see `config/galactic-router/` above, not bundled into this pod). `kubectl apply -k config/galactic-gateway/` only installs the shared, cluster-safe ServiceAccount/RBAC; `config/galactic-gateway/base/` itself is a template meant to be instantiated once per gateway node by a further overlay (see `deploy/containerlab/resources/galactic-gateway/` for a worked example) — apply that overlay per node instead of `base/` directly.
 
 ```bash
 kubectl apply -k config/fabric-router/
@@ -73,7 +73,7 @@ kubectl apply -k config/galactic-gateway/
 
   A ConfigMap edit doesn't trigger a rollout, and doesn't need one: the `config-agent` sidecar applies a change to a node's `fabric-router.<nodename>` ConfigMap to that node's running FRR in place. Check the result with `kubectl -n galactic-system get events --field-selector involvedObject.name=fabric-router.<nodename>`: `Applied` carries the hash of the `frr.conf` the node now runs, `Rejected` means the change failed `vtysh -C` and the node kept its previous config, and `ReloadFailed` means `frr-reload.py` failed and the agent is retrying. A change to `daemons` can't be applied live; it raises `RestartRequired`, and takes effect when that node's pod is deleted. The sidecar also resets any IPv6 session announcing an IPv4-mapped next hop and raises `MappedNextHopReset` on the pod.
 
-- **`config/galactic-gateway/`: per-node public interface and SRv6 address.** `config/galactic-gateway/base/daemonset.yaml`'s `galactic-gateway` container requires `GALACTIC_GATEWAY_PUBLIC_INTERFACE` and `GALACTIC_GATEWAY_SRV6_ADDRESS` — the latter must be unique per gateway node and has no generic default (there's no in-cluster mechanism yet that derives it automatically; see `internal/config/gateway.go`'s `EnvGatewaySRv6Address` doc comment). Applying `base/` as shipped, without pinning both per node, produces a crash-looping container. Instantiate `base/` via a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's values — see `deploy/containerlab/resources/galactic-gateway/` as of commit `abdd665b` for a worked example (the lab no longer deploys the gateway).
+- **`config/galactic-gateway/`: per-node public interface and SRv6 address.** `config/galactic-gateway/base/daemonset.yaml`'s `galactic-gateway` container requires `GALACTIC_GATEWAY_PUBLIC_INTERFACE` and `GALACTIC_GATEWAY_SRV6_ADDRESS` — the latter must be unique per gateway node and has no generic default (there's no in-cluster mechanism yet that derives it automatically; see `internal/config/gateway.go`'s `EnvGatewaySRv6Address` doc comment). Applying `base/` as shipped, without pinning both per node, produces a crash-looping container. Instantiate `base/` via a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's values — see `deploy/containerlab/resources/galactic-gateway/` for a worked example.
 
 - **Talos: gRPC health port.** `galactic-router` runs `hostNetwork: true` and defaults to gRPC health checks on port `5000`, which collides with Talos's built-in dashboard (`/sbin/dashboard` permanently binds `127.0.0.1:5000` on every Talos node). `config/galactic-router/base/daemonset.yaml` already ships with `GALACTIC_ROUTER_GRPC_HEALTH_PORT=5179` (and matching probe/containerPort) to avoid this; if you run `galactic-router` outside these manifests on Talos, set `GALACTIC_ROUTER_GRPC_HEALTH_PORT` to something other than `5000` yourself.