fix: Place the gateway and NAT on edge nodes

config/galactic-gateway/base/daemonset.yaml

@@ -18,21 +18,19 @@ spec:
       labels:
         app.kubernetes.io/name: galactic-gateway
     spec:
-      # Opt-in only: runs on a node an operator has explicitly labeled
+      # Opt-in only: runs on a galactic.datumapis.com/node=edge node an
+      # operator has explicitly labeled
       # galactic.datumapis.com/gateway=enabled, and nowhere else. Nothing
       # deploys it automatically -- see ../kustomization.yaml for why this
       # base is meant to be instantiated once per gateway node rather than
       # applied as-is. It is independent of galactic-nat: a node can carry
       # either label or both; on a node with both, each runs from its own
       # slot of the node's shared XDP dispatcher.
       #
-      # It never shares a node with galactic-vrf. That sidecar's pods are
-      # placed on galactic.datumapis.com/node=edge nodes, and the affinity
-      # below requires this node to carry no galactic.datumapis.com/node
-      # value at all, so the two exclude each other by construction (a
-      # node's `node` label is single-valued). Excludes control-plane
-      # outright for the same reason the other roles do. See
-      # docs/node-labels.md.
+      # It shares the edge node with galactic-vrf, whose sidecar pods are
+      # placed on galactic.datumapis.com/node=edge nodes too. Compute nodes
+      # (node=compute), unlabeled nodes and control-plane nodes never run
+      # it. See docs/node-labels.md.
       #
       # Single-container pod: galactic-router used to run here as a
       # second container (see git history prior to this file), so a crash
@@ -50,12 +48,14 @@ spec:
               - matchExpressions:
                   - key: node-role.kubernetes.io/control-plane
                     operator: DoesNotExist
+                  - key: galactic.datumapis.com/node
+                    operator: In
+                    values:
+                      - edge
                   - key: galactic.datumapis.com/gateway
                     operator: In
                     values:
                       - enabled
-                  - key: galactic.datumapis.com/node
-                    operator: DoesNotExist
       serviceAccountName: galactic-gateway
       hostNetwork: true
       dnsPolicy: ClusterFirstWithHostNet

config/galactic-nat/base/daemonset.yaml

@@ -18,20 +18,19 @@ spec:
       labels:
         app.kubernetes.io/name: galactic-nat
     spec:
-      # Opt-in only: runs on a node an operator has explicitly labeled
-      # galactic.datumapis.com/nat=enabled, and nowhere else. Nothing
+      # Opt-in only: runs on a galactic.datumapis.com/node=edge node an
+      # operator has explicitly labeled galactic.datumapis.com/nat=enabled,
+      # and nowhere else. Nothing
       # deploys it automatically. It is independent of galactic-gateway: a
       # node can carry either label or both. The shard attaches its own XDP
       # program by default (GALACTIC_NAT_XDP_ATTACH=direct); alongside a
       # gateway, set dispatch, so both run from the node's shared XDP
       # dispatcher, since an interface takes only one program.
       #
-      # It never shares a node with galactic-vrf. That sidecar's pods are
-      # placed on galactic.datumapis.com/node=edge nodes, and the affinity
-      # below requires this node to carry no galactic.datumapis.com/node
-      # value at all, so the two exclude each other by construction (a
-      # node's `node` label is single-valued). Compute nodes reach their
-      # site's shards over SRv6 (GALACTIC_CNI_EGRESS_SHARD_SIDS).
+      # It shares the edge node with galactic-vrf, whose sidecar pods are
+      # placed on galactic.datumapis.com/node=edge nodes too. Compute nodes
+      # (node=compute) never run a shard; they reach their site's shards
+      # over SRv6 (GALACTIC_CNI_EGRESS_SHARD_SIDS).
       # galactic-cni and galactic-router may run here too, via the separate
       # galactic.datumapis.com/galactic=router label
       # (config/galactic-cni/daemonset.yaml,
@@ -53,12 +52,14 @@ spec:
               - matchExpressions:
                   - key: node-role.kubernetes.io/control-plane
                     operator: DoesNotExist
+                  - key: galactic.datumapis.com/node
+                    operator: In
+                    values:
+                      - edge
                   - key: galactic.datumapis.com/nat
                     operator: In
                     values:
                       - enabled
-                  - key: galactic.datumapis.com/node
-                    operator: DoesNotExist
       serviceAccountName: galactic-nat
       hostNetwork: true
       dnsPolicy: ClusterFirstWithHostNet

deploy/containerlab/AGENTS.md

@@ -17,7 +17,7 @@
 - **Worker SRv6 blackhole route**: A single standalone blackhole route covering the node's `/64` uSID space — its `/48` Block plus its own Node-ID, which is the width `internal/plumbing/ebpf/uformat.go`'s `LocatorBits = BlockBits + NodeIDBits` (64 bits) gives a node, so it covers exactly the SIDs that node can issue (`metric 2048`, no interface — `ip -6 route add blackhole` takes no `dev`) prevents the default route from matching any USID this node could compute, set via `exec` commands in the ContainerLab topology. One aggregate route rather than one per VPC: IPv6 FIB lookup is longest-prefix-match first, so a real seg6local `/128` decap route always wins over the covering blackhole regardless of metric — the aggregate protects every current and future VPC's SID with no per-VPC config needed. The USID is **not** configured as an address on any interface. The seg6local route (`metric 1024`) in the main table handles SRv6 decapsulation. If the USID were an interface address, the kernel's local table route (metric 0) would shadow the seg6local route and break decapsulation. (A dedicated `lo-galactic` dummy interface previously hosted this — removed since it wasn't referenced by any Go code or by the blackhole route itself; `install.sh`'s `all`/`default` sysctl entries already cover the SRv6 kernel knobs it existed for.)
 - **Node labels**: this lab follows the same label strategy as production (see the root `AGENTS.md`'s "Node label strategy" table) — a primary-role enum (`node`) plus a mode enum per binary family (`galactic`, `fabric`), not one giant enum or a pile of independent booleans:
   - `galactic.datumapis.com/node: compute` — one per site (`dfw-worker`, `sjc-worker`, `iad-worker`); deploys nothing keyed on this label itself. Each compute node egresses only through its own site's NAT shards (`resources/galactic-cni/<site>/egress-shards-patch.yaml`).
-  - `galactic.datumapis.com/nat: enabled` (and no `galactic.datumapis.com/node` label, so it never shares a node with `galactic-vrf`) — `dfw-worker2`, `dfw-worker3`, `sjc-worker2`, `iad-worker2`, this lab's edge nodes; deploys `galactic-nat` (one DaemonSet per site, identity from each node's `EgressShard`), so all four are egress shards — dfw has two. No node carries `galactic.datumapis.com/gateway`, so the lab runs no `galactic-gateway` and every shard attaches its own XDP program (`GALACTIC_NAT_XDP_ATTACH=direct`, the default). Each of these nodes carries a `galactic.datumapis.com/nat=enabled:NoSchedule` taint.
+  - `galactic.datumapis.com/node: edge` plus `galactic.datumapis.com/nat: enabled` — `dfw-worker2`, `dfw-worker3`, `sjc-worker2`, `iad-worker2`, this lab's edge nodes; the NAT base requires both, so these deploy `galactic-nat` (one DaemonSet per site, identity from each node's `EgressShard`), so all four are egress shards — dfw has two. No node carries `galactic.datumapis.com/gateway`, so the lab runs no `galactic-gateway` and every shard attaches its own XDP program (`GALACTIC_NAT_XDP_ATTACH=direct`, the default). Each of these nodes carries a `galactic.datumapis.com/nat=enabled:NoSchedule` taint. Kind applies labels only when a cluster is created, so a lab brought up before `node=edge` was added needs `kubectl label node <edge worker> galactic.datumapis.com/node=edge` on all four before `deploy:galactic-nat` re-applies the base, or the shards are removed. Adding the label alone, with the old manifests still applied, removes them too.
   - `galactic.datumapis.com/galactic: router` — every `compute` node and every NAT node (seven in all); deploys `galactic-cni` and plain-mode `galactic-router` together. Mutually exclusive with `galactic: control` below (one label key, one value).
   - `galactic.datumapis.com/galactic: control` — `iad-worker3` only; deploys `galactic-router-rr`. Not a `galactic.datumapis.com/node` value — a matching `NoSchedule` taint uses the same key/value.
   - `galactic.datumapis.com/fabric: router` — every worker in the lab (all eight); deploys `fabric-router`, independently of whatever other labels a node carries. Note this is *not* the same as being transit-facing: every worker runs FRR, but only the edge nodes' FRR holds an eBGP session to the transit; the rest run iBGP to their site's edge node.

deploy/containerlab/Taskfile.yaml

@@ -452,8 +452,8 @@ tasks:
       # One shard per edge node: two in dfw, one each in sjc and iad. Each
       # must report Programmed=True -- attached is not enough, since a shard
       # with no programmed identity claims no packet -- and must target a
-      # node labeled galactic.datumapis.com/nat=enabled, the only nodes the
-      # shard DaemonSet runs on.
+      # node labeled galactic.datumapis.com/nat=enabled. The shard
+      # DaemonSet runs only on node=edge nodes carrying that label.
       - |
         set -e
         rc=0

deploy/containerlab/node_files/dfw/config.yaml

@@ -22,6 +22,7 @@ nodes:
   - role: worker
     labels:
       topology.kubernetes.io/region: dfw
+      galactic.datumapis.com/node: edge
       galactic.datumapis.com/nat: enabled
       galactic.datumapis.com/galactic: router
       galactic.datumapis.com/fabric: router
@@ -36,6 +37,7 @@ nodes:
   - role: worker
     labels:
       topology.kubernetes.io/region: dfw
+      galactic.datumapis.com/node: edge
       galactic.datumapis.com/nat: enabled
       galactic.datumapis.com/galactic: router
       galactic.datumapis.com/fabric: router

deploy/containerlab/node_files/iad/config.yaml

@@ -13,6 +13,7 @@ nodes:
   - role: worker
     labels:
       topology.kubernetes.io/region: iad
+      galactic.datumapis.com/node: edge
       galactic.datumapis.com/nat: enabled
       galactic.datumapis.com/galactic: router
       galactic.datumapis.com/fabric: router

deploy/containerlab/node_files/sjc/config.yaml

@@ -13,6 +13,7 @@ nodes:
   - role: worker
     labels:
       topology.kubernetes.io/region: sjc
+      galactic.datumapis.com/node: edge
       galactic.datumapis.com/nat: enabled
       galactic.datumapis.com/galactic: router
       galactic.datumapis.com/fabric: router

deploy/containerlab/resources/galactic-nat/README.md

@@ -14,8 +14,9 @@ uplinks. Compute nodes run none.
   `eth0`, which carries every node's IPv6 default route.
 - `dfw/`, `sjc/`, `iad/` — one per-site overlay each, applying one
   `galactic-nat` DaemonSet to that site's edge nodes. There is no per-node
-  pin: the base's `galactic.datumapis.com/nat: enabled` affinity already
-  selects them (the lab labels every edge node so), and nothing about a
+  pin: the base's affinity (`galactic.datumapis.com/node: edge` plus
+  `galactic.datumapis.com/nat: enabled`) already selects them (the lab
+  labels every edge node with both), and nothing about a
   shard's identity lives in the
   DaemonSet. Each site's `node-patch.yaml` sets only
   `GALACTIC_NAT_UPLINK_INTERFACES=bond0,bond1` — an edge node's transit

docs/agents/ARCHITECTURE-GATEWAY.md

@@ -40,8 +40,9 @@ document to start from for a given task.
 load-balances into a tenant VPC's backend Pods, without a VRF or tunnel
 dependency and without a per-tenant Geneve device. It is a **separate
 binary from `galactic-router`**, deployed as its own single-container,
-`hostNetwork: true` DaemonSet on nodes an operator has labeled
-`galactic.datumapis.com/gateway: enabled` (opt-in, never automatic), so a crash on either side — tenant
+`hostNetwork: true` DaemonSet on `galactic.datumapis.com/node: edge` nodes
+an operator has labeled `galactic.datumapis.com/gateway: enabled` (opt-in,
+never automatic; it shares those nodes with `galactic-vrf`), so a crash on either side — tenant
 BGP vs. the XDP-holding gateway engine — no longer takes the other down
 with it. `galactic-router` used to run as a second container in this same
 pod; it's now a fully independent DaemonSet instead, opted in via

docs/gateway/configuration.md

@@ -44,16 +44,19 @@ datapath — see that file's own header for the caveat).
 
 ## Prerequisite: node labeling
 
-`galactic-gateway`'s DaemonSet only schedules onto nodes labeled
-`galactic.datumapis.com/node: edge` (see
-`config/galactic-gateway/base/daemonset.yaml`'s node affinity, which also
-excludes `node-role.kubernetes.io/control-plane` nodes outright). Read
+`galactic-gateway`'s DaemonSet only schedules onto nodes labeled both
+`galactic.datumapis.com/node: edge` and `galactic.datumapis.com/gateway:
+enabled` (see `config/galactic-gateway/base/daemonset.yaml`'s node affinity,
+which also excludes `node-role.kubernetes.io/control-plane` nodes outright).
+It runs there alongside `galactic-vrf`, which `node=edge` also places, so
+label an edge node `gateway=enabled` to opt it in. Read
 [docs/node-labels.md](../node-labels.md) for the full node-labeling
 strategy before doing anything else here.
 
 **Do not confuse this label with the gateway's own "edge XDP" terminology.**
-`galactic.datumapis.com/node=edge` is the *node role* label — it identifies a
-dedicated gateway node (which also runs `galactic-nat`), distinct from the
+`galactic.datumapis.com/node=edge` is the *node role* label — it identifies an
+edge node (which hosts `galactic-vrf`, and `galactic-gateway` or
+`galactic-nat` where it carries their opt-in label), distinct from the
 ordinary tenant-serving `galactic.datumapis.com/node=compute` role that runs
 `galactic-router` (default role)/`galactic-cni`. "Edge XDP" is a separate,
 pre-existing description of *what the datapath is* (an XDP program attached

docs/metrics.md

@@ -9,8 +9,9 @@ themselves; when this page and the code disagree, the code wins.
 
 Every DaemonSet under `config/` runs `hostNetwork: true`, so its port is
 bound on the node itself. `galactic-vrf` runs as a sidecar in the ingress
-proxy's pod instead, and shares the 9182 default with `galactic-nat`
-without clashing, since the two never share a node.
+proxy's pod instead, which has its own network namespace, so it shares
+the 9182 default with `galactic-nat` without clashing, even on the edge
+nodes that run both.
 
 | Area                   | Component                                                 | Metric prefix                           | Port           | `job` from `config/monitoring/`          |
 |------------------------|-----------------------------------------------------------|-----------------------------------------|----------------|------------------------------------------|

docs/nat/configuration.md

@@ -18,9 +18,12 @@ covers the "what", not the "why" or the step-by-step.
 
 ## Placement
 
-`galactic-nat` runs on every `galactic.datumapis.com/node: edge` node, one
-DaemonSet per cluster, alongside `galactic-gateway` (see
-[docs/node-labels.md](../node-labels.md)). A shard there is one hop from the
+`galactic-nat` runs on every `galactic.datumapis.com/node: edge` node that
+also carries `galactic.datumapis.com/nat: enabled`, one DaemonSet per
+cluster, alongside `galactic-vrf` and, on nodes that opt into it,
+`galactic-gateway` (see [docs/node-labels.md](../node-labels.md)). An edge
+node without `nat=enabled`, a compute node, an unlabeled node and a
+control-plane node never run a shard. A shard there is one hop from the
 transit its masquerade addresses are originated into. Compute nodes run no
 shard: each compute node's tenant egress is SRv6-encapsulated toward its own
 site's edge shards (`GALACTIC_CNI_EGRESS_SHARD_SIDS`, [below](#shard-membership-galactic-cni-side)).