diff --git a/AGENTS.md b/AGENTS.md index 6ca92662..1f793771 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -69,18 +69,18 @@ Summary: `config/` is Kustomize-composed: `kubectl apply -k config/` deploys everything (namespace, both DaemonSets' RBAC/ServiceAccounts, and all three DaemonSets) in one command — `kubectl` sorts by kind before applying, so the namespace and RBAC/ServiceAccounts always land before anything namespace-scoped needs them. Each component also has its own `kustomization.yaml` and can be applied independently: -**Node label strategy.** Every DaemonSet's node affinity keys off one or more of these labels — a primary-role enum (`node`) plus a mode enum per binary family (`galactic`, `fabric`), rather than one giant enum or a pile of independent booleans. `node`'s two values are mutually exclusive by design (a node is never both `compute` and `edge`), and so are each family's two mode values (a node running plain `galactic-router` never also runs `galactic-router-rr`) — but `node` and `galactic`/`fabric` are independent keys, so their values freely combine (e.g. `node=edge` + `galactic=router` + `fabric=router` on one node). `galactic-gateway` and `galactic-nat` are deliberately outside `node`: each has its own opt-in label, is never deployed automatically, and requires the node to carry no `node` value, so neither can share a node with `galactic-vrf` (which runs on `node=edge`): - -| Label | Deploys | -| ----- | ------- | -| `galactic.datumapis.com/node=compute` | nothing exclusively — the ordinary tenant-serving role (`galactic-cni`, `galactic-router`, `fabric-router` via the labels below). Never hosts `galactic-vrf`, `galactic-gateway` or `galactic-nat` | -| `galactic.datumapis.com/node=edge` | `galactic-vrf` (the ingress sidecar's pods, placed by whatever defines them — no manifest in this repo yet); also runs `galactic-cni`, `galactic-router`, `fabric-router` via the labels below. Never hosts `galactic-gateway` or `galactic-nat` | -| `galactic.datumapis.com/gateway=enabled` | `galactic-gateway` only; opt-in, independent of `nat`, requires no `galactic.datumapis.com/node` label on the node | -| `galactic.datumapis.com/nat=enabled` | `galactic-nat` only; opt-in, independent of `gateway`, requires no `galactic.datumapis.com/node` label on the node | -| `galactic.datumapis.com/galactic=router` | `galactic-cni`, `galactic-router` (plain/tenant mode) — runs on both `compute` and `edge` nodes | -| `galactic.datumapis.com/galactic=control` | `galactic-router-rr`; mutually exclusive with `galactic=router` on the same node | -| `galactic.datumapis.com/fabric=router` | `fabric-router` | -| `galactic.datumapis.com/fabric=control` | `fabric-router`, same image as `fabric=router` — reflector-vs-plain is driven entirely by the per-node `frr.conf` infra's own renderer produces, not a separate binary or overlay | +**Node label strategy.** Every DaemonSet's node affinity keys off one or more of these labels — a primary-role enum (`node`) plus a mode enum per binary family (`galactic`, `fabric`), rather than one giant enum or a pile of independent booleans. `node`'s two values are mutually exclusive by design (a node is never both `compute` and `edge`), and so are each family's two mode values (a node running plain `galactic-router` never also runs `galactic-router-rr`) — but `node` and `galactic`/`fabric` are independent keys, so their values freely combine (e.g. `node=edge` + `galactic=router` + `fabric=router` on one node). `galactic-gateway` and `galactic-nat` run only on `node=edge` nodes, alongside `galactic-vrf`, and each also needs its own opt-in label, so neither is ever deployed automatically: + +| Label | Deploys | +| ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `galactic.datumapis.com/node=compute` | nothing exclusively — the ordinary tenant-serving role (`galactic-cni`, `galactic-router`, `fabric-router` via the labels below). Never hosts `galactic-vrf`, `galactic-gateway` or `galactic-nat` | +| `galactic.datumapis.com/node=edge` | `galactic-vrf` (the ingress sidecar's pods, placed by whatever defines them — no manifest in this repo yet); also runs `galactic-cni`, `galactic-router`, `fabric-router` via the labels below, and `galactic-gateway`/`galactic-nat` when the node also carries their opt-in label | +| `galactic.datumapis.com/gateway=enabled` | `galactic-gateway` only; opt-in, independent of `nat`, requires `galactic.datumapis.com/node=edge` on the node | +| `galactic.datumapis.com/nat=enabled` | `galactic-nat` only; opt-in, independent of `gateway`, requires `galactic.datumapis.com/node=edge` on the node | +| `galactic.datumapis.com/galactic=router` | `galactic-cni`, `galactic-router` (plain/tenant mode) — runs on both `compute` and `edge` nodes | +| `galactic.datumapis.com/galactic=control` | `galactic-router-rr`; mutually exclusive with `galactic=router` on the same node | +| `galactic.datumapis.com/fabric=router` | `fabric-router` | +| `galactic.datumapis.com/fabric=control` | `fabric-router`, same image as `fabric=router` — reflector-vs-plain is driven entirely by the per-node `frr.conf` infra's own renderer produces, not a separate binary or overlay | See [docs/node-labels.md](docs/node-labels.md) for the full strategy — why `galactic`/`fabric` are each a mode enum rather than independent booleans, the naming collision between `node=edge` and `galactic-gateway`'s pre-existing "edge XDP" terminology, and the specific bugs this scheme replaced. @@ -91,8 +91,8 @@ See [docs/node-labels.md](docs/node-labels.md) for the full strategy — why `ga - **`config/galactic-router/overlays/router/`** — the plain (non-reflector) per-node role (`galactic-router`); runs on every `compute` *and* every `edge` node except Kubernetes control-plane nodes, opt-in via `galactic.datumapis.com/galactic: router` (the same label `galactic-cni` keys off of — see `config/galactic-cni/daemonset.yaml`). This is the default role, so it carries no role-specific name/label of its own — only `control` below needs one, to coexist as a second DaemonSet. - **`config/galactic-router/overlays/control/`** — the BGP route-reflector role (`galactic-router-rr`, `GALACTIC_ROUTER_REFLECTOR=true`); opt-in only, requires `galactic.datumapis.com/galactic: control` (stays at zero replicas otherwise). Mutually exclusive with `router`'s `galactic: router` value on the same node — one label key, single-valued, so a route-reflector node structurally can't also run plain-mode `galactic-router`. `GALACTIC_ROUTER_BGP_LOCAL_ADDRESS` is auto-detected from the host's `lo` interface by default; see the comments in `daemonset-patch.yaml` for when to override it. - Apply both roles with `kubectl apply -k config/galactic-router/`, or a single role with e.g. `kubectl apply -k config/galactic-router/overlays/router/`. `galactic-router` no longer runs bundled inside `galactic-gateway`'s pod — see `config/galactic-gateway/` below. -- **`config/galactic-gateway/`** — the edge XDP NAT+LB gateway control plane, a single-container `galactic-gateway` DaemonSet — `galactic-router` used to run as a second container in this same pod; it's now a fully independent DaemonSet (via `galactic.datumapis.com/galactic: router`, which a gateway node may carry like any other), so a crash on either side no longer takes the other's *pod* down with it, not just the other's binary. `config/galactic-gateway/{serviceaccount.yaml,rbac.yaml}` (safe/idempotent to apply cluster-wide) are what `kubectl apply -k config/galactic-gateway/` applies; `config/galactic-gateway/base/` (requiring nodes labeled `galactic.datumapis.com/gateway: enabled` and carrying no `galactic.datumapis.com/node` label, so it never shares a node with `galactic-vrf`) is **not** included in that kustomization and is **not** applied as-is — the same exemption as `config/fabric-router/` below, for the same reason: `GALACTIC_GATEWAY_SRV6_ADDRESS` must be unique per gateway node and has no generic default (no in-cluster mechanism yet derives it automatically — see `internal/config/gateway.go`'s `EnvGatewaySRv6Address` doc comment). It's designed to be instantiated once per gateway node by a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's own public-interface/SRv6-address values; see `deploy/containerlab/resources/galactic-gateway/` as of commit `abdd665b` for a worked example (the containerlab lab no longer deploys the gateway). Also **not** part of the root `config/kustomization.yaml`'s default resource list, matching `config/fabric-router/`'s exemption. -- **`config/galactic-nat/`** — the sharded egress translation (NAT66/NAT64) control plane, one `galactic-nat` DaemonSet per cluster on every node labeled `galactic.datumapis.com/nat: enabled` (and carrying no `galactic.datumapis.com/node` label). It is independent of `galactic-gateway`. A shard's identity (SID, masquerade addresses, NAT64 prefix) comes from the `EgressShard` targeting its node, not from the DaemonSet. The base leaves `GALACTIC_NAT_XDP_ATTACH` at its default, `direct`. On a node that also runs the gateway, set it to `dispatch`: an interface takes one native XDP program, so the shard and the gateway (which defaults to `GALACTIC_GATEWAY_XDP_ATTACH=dispatch`) each run from their own slot of the node's shared, pinned XDP dispatcher (`internal/plumbing/ebpf/xdpdispatch`), and either can restart or be turned off (`GALACTIC_*_DATAPATH_ENABLED=false`) without touching the other's traffic. `hack/xdp-dispatch-release.sh` detaches the dispatcher from a node. Compute nodes reach their own site's shards over SRv6 via `GALACTIC_CNI_EGRESS_SHARD_SIDS`. `kubectl apply -k config/galactic-nat/` applies only the ServiceAccount/RBAC; `base/` is excluded because uplink auto-detection is deployment-specific (see `config/galactic-nat/kustomization.yaml`). Not part of the root `config/kustomization.yaml`. +- **`config/galactic-gateway/`** — the edge XDP NAT+LB gateway control plane, a single-container `galactic-gateway` DaemonSet — `galactic-router` used to run as a second container in this same pod; it's now a fully independent DaemonSet (via `galactic.datumapis.com/galactic: router`, which a gateway node may carry like any other), so a crash on either side no longer takes the other's *pod* down with it, not just the other's binary. `config/galactic-gateway/{serviceaccount.yaml,rbac.yaml}` (safe/idempotent to apply cluster-wide) are what `kubectl apply -k config/galactic-gateway/` applies; `config/galactic-gateway/base/` (requiring nodes labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/gateway: enabled`, so it runs on edge nodes alongside `galactic-vrf`) is **not** included in that kustomization and is **not** applied as-is — the same exemption as `config/fabric-router/` below, for the same reason: `GALACTIC_GATEWAY_SRV6_ADDRESS` must be unique per gateway node and has no generic default (no in-cluster mechanism yet derives it automatically — see `internal/config/gateway.go`'s `EnvGatewaySRv6Address` doc comment). It's designed to be instantiated once per gateway node by a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's own public-interface/SRv6-address values; see `deploy/containerlab/resources/galactic-gateway/` as of commit `abdd665b` for a worked example (the containerlab lab no longer deploys the gateway). Also **not** part of the root `config/kustomization.yaml`'s default resource list, matching `config/fabric-router/`'s exemption. +- **`config/galactic-nat/`** — the sharded egress translation (NAT66/NAT64) control plane, one `galactic-nat` DaemonSet per cluster on every node labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/nat: enabled`, alongside `galactic-vrf`. It is independent of `galactic-gateway`. A shard's identity (SID, masquerade addresses, NAT64 prefix) comes from the `EgressShard` targeting its node, not from the DaemonSet. The base leaves `GALACTIC_NAT_XDP_ATTACH` at its default, `direct`. On a node that also runs the gateway, set it to `dispatch`: an interface takes one native XDP program, so the shard and the gateway (which defaults to `GALACTIC_GATEWAY_XDP_ATTACH=dispatch`) each run from their own slot of the node's shared, pinned XDP dispatcher (`internal/plumbing/ebpf/xdpdispatch`), and either can restart or be turned off (`GALACTIC_*_DATAPATH_ENABLED=false`) without touching the other's traffic. `hack/xdp-dispatch-release.sh` detaches the dispatcher from a node. Compute nodes reach their own site's shards over SRv6 via `GALACTIC_CNI_EGRESS_SHARD_SIDS`. `kubectl apply -k config/galactic-nat/` applies only the ServiceAccount/RBAC; `base/` is excluded because uplink auto-detection is deployment-specific (see `config/galactic-nat/kustomization.yaml`). Not part of the root `config/kustomization.yaml`. - **`config/fabric-router/`** — the FRR underlay eBGP DaemonSet (`fabric-router`; `galactic-router` needs a working underlay before it can start). Unlike `config/galactic-router/`, this is a single flat DaemonSet with no `default`/`rr`-style role split — its affinity matches any node labeled `galactic.datumapis.com/fabric: router` (a dedicated mode-enum label, independent of whatever `galactic.datumapis.com/node` role value or `galactic.datumapis.com/galactic` mode that node also carries) directly, since there's no env/config difference between running on a regular node vs. those roles — fabric-router itself is identical everywhere; only the per-node BGP underlay config it reads from the ConfigMap differs. That affinity can legitimately match more than one node per cluster, and BGP underlay config (hostname, router-id, interface addresses, remote-AS) inherently differs per physical node — so each node's config lives in its own `fabric-router.` ConfigMap (a `frr.conf` key, plus optional `daemons`/`vtysh.conf` overrides of the image defaults) rather than one shared ConfigMap for the whole DaemonSet. `fabric-config-agent` (`cmd/fabric-config-agent`, `internal/fabricconfig`, built into the `fabric-router` image) reads it from the API: as the `frr-init` init container it validates (`vtysh -C`) and installs it, holding the pod in `Init` until a valid one exists; as the `config-agent` sidecar it applies later changes live with `frr-reload.py`, so an edit to one node's ConfigMap neither restarts the pod nor touches other nodes, and reports `Applied`/`Rejected`/`ReloadFailed`/`RestartRequired` events on the ConfigMap. The sidecar also hard-resets any IPv6 session announcing an IPv4-mapped next hop (`internal/fabricconfig/nexthop.go`) and serves that check's metrics on the node's port `9343`. A third container, `frr-exporter` (upstream `tynany/frr_exporter`, pinned by digest), reads the daemons' vty sockets in the same pod-scoped `/run/frr` and serves BGP peer state, message counters and prefix counts for both underlay families on the node's port `9342` (`hostNetwork`); `task verify:fabric-metrics` in `deploy/containerlab/` checks it on every lab node. The image's default `daemons` loads bgpd's BMP module (`-M bmp`), so a node's `frr.conf` can stream to a BMP collector (`config/bmp-collector/`) with a `bmp targets` block; loading it alone streams nothing, and a change to the modules needs a pod restart rather than a reload. Every lab node's `frr.conf` streams its pre-policy Adj-RIB-In and Loc-RIB for both families, sourced from `lo` (`source-interface lo`), since the underlay carries loopbacks but not link subnets. Production `frr.conf`s come from infra's renderer, which has to emit the same block for production nodes to stream. A node with no ConfigMap of its own falls back, with a `DeprecatedConfigSource` warning, to key `frr.conf.` of the legacy shared `fabric-config` ConfigMap. **Not** part of the root `config/kustomization.yaml` and not covered by `kubectl apply -k config/` — unlike every other component here, it has no generic default: the deployer must hand-author one `fabric-router.` ConfigMap per matching node before applying `kubectl apply -k config/fabric-router/` (which also applies the agent's ServiceAccount and namespaced Role). - **`config/bmp-collector/`** — a single-replica GoBMP Deployment and its `bmp-collector` Service (TCP `5000`), the BMP (RFC 7854) collector `galactic-router` and `fabric-router` stream to; `galactic-router` does so when `GALACTIC_ROUTER_BMP_STATIONS` names it (`bmp-collector.galactic-system.svc:5000`; see [docs/router/configuration.md](docs/router/configuration.md)). It parses every Peer Up/Down, route-monitoring and statistics message to JSON on stdout, so the cluster's log pipeline is its store; an overlay swapping `--dump=console` for `--kafka-server` publishes to Kafka instead. Opt-in, like `config/fabric-router/`: not part of the root `config/kustomization.yaml`, since nothing streams to it until a router is configured to. The containerlab lab runs one on `iad-worker3`'s host network (`deploy/containerlab/resources/bmp-collector/iad/`), because its three clusters cannot reach each other's Services, and points every node's `fabric-router` and the EVPN route reflector at it. Of the `galactic-router`s, only the reflector streams: its pre-policy Adj-RIB-In already holds every node's advertisements. - **`config/monitoring/`** — Prometheus Operator resources for BGP and NAT: a `fabric-router` PodMonitor (frr-exporter, port `frr-metrics`/`9342`, as `job=fabric-router`; fabric-config-agent, port `agent-metrics`/`9343`, as `job=fabric-config-agent`), a `galactic-router` PodMonitor covering both roles (port `metrics`/`9179`), a `galactic-nat` PodMonitor for the egress shards (port `metrics`/`9182`), each adding a `node` label, and the `galactic-bgp` PrometheusRule: underlay session down, administratively down for an hour, flapping, received prefixes halved, exporter collector or scrape failing, a session announcing an IPv4-mapped next hop, BGP routes not installed, the agent's metrics scrape down, a `galactic-router` BMP station or metrics scrape down, `galactic-router` routes failing to install, an egress shard's session table evicting live sessions, a single stalled NIC receive queue on a `galactic-nat` uplink (#673), and a `galactic-nat` metrics scrape down. Requires the `monitoring.coreos.com` CRDs, which this repo does not install, so it is not part of the root `config/kustomization.yaml`. `task test:alerts` unit-tests the rules with `promtool` (`config/monitoring/tests/`). Every alert carries what infra's alerting standard checks for (`service: galactic`, `team: connect`, `runbook_url` into infra's `docs/runbooks/galactic/README.md`). The PrometheusRule here is the tested source, not where the rules run on Datum's infra: there the PodMonitors are converted and scraped on each edge cluster, but edge vmalert never notifies, so infra evaluates a copy of the rules centrally. Change the rules here first, with their tests, then update infra's copy. @@ -108,5 +108,5 @@ See [docs/node-labels.md](docs/node-labels.md) for the full strategy — why `ga 6. Explore `internal/plumbing/` for shared kernel and network primitives (VRF, sysctl, interface naming, SRv6). 7. Read `internal/gateway/engine.go` and `internal/controller/networkgateway_controller.go` to understand the edge XDP NAT+LB gateway's convergence loop and Active-Active BGP placement model. See [ARCHITECTURE-GATEWAY.md](docs/agents/ARCHITECTURE-GATEWAY.md). 8. See `docs/cni/README.md` for the CNI docs entry point — configuration reference (env vars, conflist fields, worked examples) and Mermaid sequence diagrams for the attach path and garbage collection. `docs/agent-startup.md` covers router startup, and `docs/router/configuration.md` documents router environment variables. -9. Read [docs/node-labels.md](docs/node-labels.md) to understand which `galactic.datumapis.com/*` labels put which DaemonSets on which nodes, and why gateway and NAT sit outside `node` — this cuts across all five binaries and isn't owned by any single component's architecture doc. +9. Read [docs/node-labels.md](docs/node-labels.md) to understand which `galactic.datumapis.com/*` labels put which DaemonSets on which nodes, and how gateway and NAT opt in on edge nodes — this cuts across all five binaries and isn't owned by any single component's architecture doc. 10. See [docs/reference.md](docs/reference.md) for a quick-reference table of every deployable asset in `config/` and every TCP port its containers configure — useful when checking for a port collision or confirming what `kubectl apply -k config//` actually applies before diving into any single component's doc. diff --git a/config/galactic-gateway/base/daemonset.yaml b/config/galactic-gateway/base/daemonset.yaml index 55c477c0..638ee483 100644 --- a/config/galactic-gateway/base/daemonset.yaml +++ b/config/galactic-gateway/base/daemonset.yaml @@ -18,7 +18,8 @@ spec: labels: app.kubernetes.io/name: galactic-gateway spec: - # Opt-in only: runs on a node an operator has explicitly labeled + # Opt-in only: runs on a galactic.datumapis.com/node=edge node an + # operator has explicitly labeled # galactic.datumapis.com/gateway=enabled, and nowhere else. Nothing # deploys it automatically -- see ../kustomization.yaml for why this # base is meant to be instantiated once per gateway node rather than @@ -26,13 +27,10 @@ spec: # either label or both; on a node with both, each runs from its own # slot of the node's shared XDP dispatcher. # - # It never shares a node with galactic-vrf. That sidecar's pods are - # placed on galactic.datumapis.com/node=edge nodes, and the affinity - # below requires this node to carry no galactic.datumapis.com/node - # value at all, so the two exclude each other by construction (a - # node's `node` label is single-valued). Excludes control-plane - # outright for the same reason the other roles do. See - # docs/node-labels.md. + # It shares the edge node with galactic-vrf, whose sidecar pods are + # placed on galactic.datumapis.com/node=edge nodes too. Compute nodes + # (node=compute), unlabeled nodes and control-plane nodes never run + # it. See docs/node-labels.md. # # Single-container pod: galactic-router used to run here as a # second container (see git history prior to this file), so a crash @@ -50,12 +48,14 @@ spec: - matchExpressions: - key: node-role.kubernetes.io/control-plane operator: DoesNotExist + - key: galactic.datumapis.com/node + operator: In + values: + - edge - key: galactic.datumapis.com/gateway operator: In values: - enabled - - key: galactic.datumapis.com/node - operator: DoesNotExist serviceAccountName: galactic-gateway hostNetwork: true dnsPolicy: ClusterFirstWithHostNet diff --git a/config/galactic-nat/base/daemonset.yaml b/config/galactic-nat/base/daemonset.yaml index 7569049c..889ab25f 100644 --- a/config/galactic-nat/base/daemonset.yaml +++ b/config/galactic-nat/base/daemonset.yaml @@ -18,20 +18,19 @@ spec: labels: app.kubernetes.io/name: galactic-nat spec: - # Opt-in only: runs on a node an operator has explicitly labeled - # galactic.datumapis.com/nat=enabled, and nowhere else. Nothing + # Opt-in only: runs on a galactic.datumapis.com/node=edge node an + # operator has explicitly labeled galactic.datumapis.com/nat=enabled, + # and nowhere else. Nothing # deploys it automatically. It is independent of galactic-gateway: a # node can carry either label or both. The shard attaches its own XDP # program by default (GALACTIC_NAT_XDP_ATTACH=direct); alongside a # gateway, set dispatch, so both run from the node's shared XDP # dispatcher, since an interface takes only one program. # - # It never shares a node with galactic-vrf. That sidecar's pods are - # placed on galactic.datumapis.com/node=edge nodes, and the affinity - # below requires this node to carry no galactic.datumapis.com/node - # value at all, so the two exclude each other by construction (a - # node's `node` label is single-valued). Compute nodes reach their - # site's shards over SRv6 (GALACTIC_CNI_EGRESS_SHARD_SIDS). + # It shares the edge node with galactic-vrf, whose sidecar pods are + # placed on galactic.datumapis.com/node=edge nodes too. Compute nodes + # (node=compute) never run a shard; they reach their site's shards + # over SRv6 (GALACTIC_CNI_EGRESS_SHARD_SIDS). # galactic-cni and galactic-router may run here too, via the separate # galactic.datumapis.com/galactic=router label # (config/galactic-cni/daemonset.yaml, @@ -53,12 +52,14 @@ spec: - matchExpressions: - key: node-role.kubernetes.io/control-plane operator: DoesNotExist + - key: galactic.datumapis.com/node + operator: In + values: + - edge - key: galactic.datumapis.com/nat operator: In values: - enabled - - key: galactic.datumapis.com/node - operator: DoesNotExist serviceAccountName: galactic-nat hostNetwork: true dnsPolicy: ClusterFirstWithHostNet diff --git a/deploy/containerlab/AGENTS.md b/deploy/containerlab/AGENTS.md index f02c3a2b..c79cc2a6 100644 --- a/deploy/containerlab/AGENTS.md +++ b/deploy/containerlab/AGENTS.md @@ -17,7 +17,7 @@ - **Worker SRv6 blackhole route**: A single standalone blackhole route covering the node's `/64` uSID space — its `/48` Block plus its own Node-ID, which is the width `internal/plumbing/ebpf/uformat.go`'s `LocatorBits = BlockBits + NodeIDBits` (64 bits) gives a node, so it covers exactly the SIDs that node can issue (`metric 2048`, no interface — `ip -6 route add blackhole` takes no `dev`) prevents the default route from matching any USID this node could compute, set via `exec` commands in the ContainerLab topology. One aggregate route rather than one per VPC: IPv6 FIB lookup is longest-prefix-match first, so a real seg6local `/128` decap route always wins over the covering blackhole regardless of metric — the aggregate protects every current and future VPC's SID with no per-VPC config needed. The USID is **not** configured as an address on any interface. The seg6local route (`metric 1024`) in the main table handles SRv6 decapsulation. If the USID were an interface address, the kernel's local table route (metric 0) would shadow the seg6local route and break decapsulation. (A dedicated `lo-galactic` dummy interface previously hosted this — removed since it wasn't referenced by any Go code or by the blackhole route itself; `install.sh`'s `all`/`default` sysctl entries already cover the SRv6 kernel knobs it existed for.) - **Node labels**: this lab follows the same label strategy as production (see the root `AGENTS.md`'s "Node label strategy" table) — a primary-role enum (`node`) plus a mode enum per binary family (`galactic`, `fabric`), not one giant enum or a pile of independent booleans: - `galactic.datumapis.com/node: compute` — one per site (`dfw-worker`, `sjc-worker`, `iad-worker`); deploys nothing keyed on this label itself. Each compute node egresses only through its own site's NAT shards (`resources/galactic-cni//egress-shards-patch.yaml`). - - `galactic.datumapis.com/nat: enabled` (and no `galactic.datumapis.com/node` label, so it never shares a node with `galactic-vrf`) — `dfw-worker2`, `dfw-worker3`, `sjc-worker2`, `iad-worker2`, this lab's edge nodes; deploys `galactic-nat` (one DaemonSet per site, identity from each node's `EgressShard`), so all four are egress shards — dfw has two. No node carries `galactic.datumapis.com/gateway`, so the lab runs no `galactic-gateway` and every shard attaches its own XDP program (`GALACTIC_NAT_XDP_ATTACH=direct`, the default). Each of these nodes carries a `galactic.datumapis.com/nat=enabled:NoSchedule` taint. + - `galactic.datumapis.com/node: edge` plus `galactic.datumapis.com/nat: enabled` — `dfw-worker2`, `dfw-worker3`, `sjc-worker2`, `iad-worker2`, this lab's edge nodes; the NAT base requires both, so these deploy `galactic-nat` (one DaemonSet per site, identity from each node's `EgressShard`), so all four are egress shards — dfw has two. No node carries `galactic.datumapis.com/gateway`, so the lab runs no `galactic-gateway` and every shard attaches its own XDP program (`GALACTIC_NAT_XDP_ATTACH=direct`, the default). Each of these nodes carries a `galactic.datumapis.com/nat=enabled:NoSchedule` taint. Kind applies labels only when a cluster is created, so a lab brought up before `node=edge` was added needs `kubectl label node galactic.datumapis.com/node=edge` on all four before `deploy:galactic-nat` re-applies the base, or the shards are removed. Adding the label alone, with the old manifests still applied, removes them too. - `galactic.datumapis.com/galactic: router` — every `compute` node and every NAT node (seven in all); deploys `galactic-cni` and plain-mode `galactic-router` together. Mutually exclusive with `galactic: control` below (one label key, one value). - `galactic.datumapis.com/galactic: control` — `iad-worker3` only; deploys `galactic-router-rr`. Not a `galactic.datumapis.com/node` value — a matching `NoSchedule` taint uses the same key/value. - `galactic.datumapis.com/fabric: router` — every worker in the lab (all eight); deploys `fabric-router`, independently of whatever other labels a node carries. Note this is *not* the same as being transit-facing: every worker runs FRR, but only the edge nodes' FRR holds an eBGP session to the transit; the rest run iBGP to their site's edge node. diff --git a/deploy/containerlab/Taskfile.yaml b/deploy/containerlab/Taskfile.yaml index 81f6154b..c1622488 100644 --- a/deploy/containerlab/Taskfile.yaml +++ b/deploy/containerlab/Taskfile.yaml @@ -452,8 +452,8 @@ tasks: # One shard per edge node: two in dfw, one each in sjc and iad. Each # must report Programmed=True -- attached is not enough, since a shard # with no programmed identity claims no packet -- and must target a - # node labeled galactic.datumapis.com/nat=enabled, the only nodes the - # shard DaemonSet runs on. + # node labeled galactic.datumapis.com/nat=enabled. The shard + # DaemonSet runs only on node=edge nodes carrying that label. - | set -e rc=0 diff --git a/deploy/containerlab/node_files/dfw/config.yaml b/deploy/containerlab/node_files/dfw/config.yaml index 7c622c88..581cee2f 100644 --- a/deploy/containerlab/node_files/dfw/config.yaml +++ b/deploy/containerlab/node_files/dfw/config.yaml @@ -22,6 +22,7 @@ nodes: - role: worker labels: topology.kubernetes.io/region: dfw + galactic.datumapis.com/node: edge galactic.datumapis.com/nat: enabled galactic.datumapis.com/galactic: router galactic.datumapis.com/fabric: router @@ -36,6 +37,7 @@ nodes: - role: worker labels: topology.kubernetes.io/region: dfw + galactic.datumapis.com/node: edge galactic.datumapis.com/nat: enabled galactic.datumapis.com/galactic: router galactic.datumapis.com/fabric: router diff --git a/deploy/containerlab/node_files/iad/config.yaml b/deploy/containerlab/node_files/iad/config.yaml index 84e7e0ed..3789f1b8 100644 --- a/deploy/containerlab/node_files/iad/config.yaml +++ b/deploy/containerlab/node_files/iad/config.yaml @@ -13,6 +13,7 @@ nodes: - role: worker labels: topology.kubernetes.io/region: iad + galactic.datumapis.com/node: edge galactic.datumapis.com/nat: enabled galactic.datumapis.com/galactic: router galactic.datumapis.com/fabric: router diff --git a/deploy/containerlab/node_files/sjc/config.yaml b/deploy/containerlab/node_files/sjc/config.yaml index e79d8bcb..30bd6b59 100644 --- a/deploy/containerlab/node_files/sjc/config.yaml +++ b/deploy/containerlab/node_files/sjc/config.yaml @@ -13,6 +13,7 @@ nodes: - role: worker labels: topology.kubernetes.io/region: sjc + galactic.datumapis.com/node: edge galactic.datumapis.com/nat: enabled galactic.datumapis.com/galactic: router galactic.datumapis.com/fabric: router diff --git a/deploy/containerlab/resources/galactic-nat/README.md b/deploy/containerlab/resources/galactic-nat/README.md index 27edf895..e883a606 100644 --- a/deploy/containerlab/resources/galactic-nat/README.md +++ b/deploy/containerlab/resources/galactic-nat/README.md @@ -14,8 +14,9 @@ uplinks. Compute nodes run none. `eth0`, which carries every node's IPv6 default route. - `dfw/`, `sjc/`, `iad/` — one per-site overlay each, applying one `galactic-nat` DaemonSet to that site's edge nodes. There is no per-node - pin: the base's `galactic.datumapis.com/nat: enabled` affinity already - selects them (the lab labels every edge node so), and nothing about a + pin: the base's affinity (`galactic.datumapis.com/node: edge` plus + `galactic.datumapis.com/nat: enabled`) already selects them (the lab + labels every edge node with both), and nothing about a shard's identity lives in the DaemonSet. Each site's `node-patch.yaml` sets only `GALACTIC_NAT_UPLINK_INTERFACES=bond0,bond1` — an edge node's transit diff --git a/docs/agents/ARCHITECTURE-GATEWAY.md b/docs/agents/ARCHITECTURE-GATEWAY.md index 35b5dfad..170bf7bf 100644 --- a/docs/agents/ARCHITECTURE-GATEWAY.md +++ b/docs/agents/ARCHITECTURE-GATEWAY.md @@ -40,8 +40,9 @@ document to start from for a given task. load-balances into a tenant VPC's backend Pods, without a VRF or tunnel dependency and without a per-tenant Geneve device. It is a **separate binary from `galactic-router`**, deployed as its own single-container, -`hostNetwork: true` DaemonSet on nodes an operator has labeled -`galactic.datumapis.com/gateway: enabled` (opt-in, never automatic), so a crash on either side — tenant +`hostNetwork: true` DaemonSet on `galactic.datumapis.com/node: edge` nodes +an operator has labeled `galactic.datumapis.com/gateway: enabled` (opt-in, +never automatic; it shares those nodes with `galactic-vrf`), so a crash on either side — tenant BGP vs. the XDP-holding gateway engine — no longer takes the other down with it. `galactic-router` used to run as a second container in this same pod; it's now a fully independent DaemonSet instead, opted in via diff --git a/docs/gateway/configuration.md b/docs/gateway/configuration.md index 9d7fdcea..057d5132 100644 --- a/docs/gateway/configuration.md +++ b/docs/gateway/configuration.md @@ -44,16 +44,19 @@ datapath — see that file's own header for the caveat). ## Prerequisite: node labeling -`galactic-gateway`'s DaemonSet only schedules onto nodes labeled -`galactic.datumapis.com/node: edge` (see -`config/galactic-gateway/base/daemonset.yaml`'s node affinity, which also -excludes `node-role.kubernetes.io/control-plane` nodes outright). Read +`galactic-gateway`'s DaemonSet only schedules onto nodes labeled both +`galactic.datumapis.com/node: edge` and `galactic.datumapis.com/gateway: +enabled` (see `config/galactic-gateway/base/daemonset.yaml`'s node affinity, +which also excludes `node-role.kubernetes.io/control-plane` nodes outright). +It runs there alongside `galactic-vrf`, which `node=edge` also places, so +label an edge node `gateway=enabled` to opt it in. Read [docs/node-labels.md](../node-labels.md) for the full node-labeling strategy before doing anything else here. **Do not confuse this label with the gateway's own "edge XDP" terminology.** -`galactic.datumapis.com/node=edge` is the *node role* label — it identifies a -dedicated gateway node (which also runs `galactic-nat`), distinct from the +`galactic.datumapis.com/node=edge` is the *node role* label — it identifies an +edge node (which hosts `galactic-vrf`, and `galactic-gateway` or +`galactic-nat` where it carries their opt-in label), distinct from the ordinary tenant-serving `galactic.datumapis.com/node=compute` role that runs `galactic-router` (default role)/`galactic-cni`. "Edge XDP" is a separate, pre-existing description of *what the datapath is* (an XDP program attached diff --git a/docs/metrics.md b/docs/metrics.md index 689597cf..5f30847c 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -9,8 +9,9 @@ themselves; when this page and the code disagree, the code wins. Every DaemonSet under `config/` runs `hostNetwork: true`, so its port is bound on the node itself. `galactic-vrf` runs as a sidecar in the ingress -proxy's pod instead, and shares the 9182 default with `galactic-nat` -without clashing, since the two never share a node. +proxy's pod instead, which has its own network namespace, so it shares +the 9182 default with `galactic-nat` without clashing, even on the edge +nodes that run both. | Area | Component | Metric prefix | Port | `job` from `config/monitoring/` | |------------------------|-----------------------------------------------------------|-----------------------------------------|----------------|------------------------------------------| diff --git a/docs/nat/configuration.md b/docs/nat/configuration.md index 3a591e38..c2030871 100644 --- a/docs/nat/configuration.md +++ b/docs/nat/configuration.md @@ -18,9 +18,12 @@ covers the "what", not the "why" or the step-by-step. ## Placement -`galactic-nat` runs on every `galactic.datumapis.com/node: edge` node, one -DaemonSet per cluster, alongside `galactic-gateway` (see -[docs/node-labels.md](../node-labels.md)). A shard there is one hop from the +`galactic-nat` runs on every `galactic.datumapis.com/node: edge` node that +also carries `galactic.datumapis.com/nat: enabled`, one DaemonSet per +cluster, alongside `galactic-vrf` and, on nodes that opt into it, +`galactic-gateway` (see [docs/node-labels.md](../node-labels.md)). An edge +node without `nat=enabled`, a compute node, an unlabeled node and a +control-plane node never run a shard. A shard there is one hop from the transit its masquerade addresses are originated into. Compute nodes run no shard: each compute node's tenant egress is SRv6-encapsulated toward its own site's edge shards (`GALACTIC_CNI_EGRESS_SHARD_SIDS`, [below](#shard-membership-galactic-cni-side)). diff --git a/docs/node-labels.md b/docs/node-labels.md index b80b3a3b..6474862b 100644 --- a/docs/node-labels.md +++ b/docs/node-labels.md @@ -2,7 +2,7 @@ > How Galactic decides which DaemonSets run on which nodes. -_Last updated: 2026-09-25_ +_Last updated: 2026-10-06_ This document is cross-cutting — it covers the node-selection contract shared by `galactic-cni`, `galactic-router`, `galactic-gateway`, `galactic-nat`, @@ -14,22 +14,24 @@ for everything else about a given binary. ## The labels -| Label | Deploys | Kind | -|--------------------------------------------------|-------------------------------------------------------|-------------------| -| `galactic.datumapis.com/node=compute` | nothing exclusively (see below) | primary role enum | -| `galactic.datumapis.com/node=edge` | `galactic-vrf` (ingress sidecar pods) | primary role enum | -| `galactic.datumapis.com/gateway=enabled` | `galactic-gateway` | independent flag | -| `galactic.datumapis.com/nat=enabled` | `galactic-nat` | independent flag | -| `galactic.datumapis.com/galactic=router` | `galactic-cni`, `galactic-router` (plain/tenant mode) | mode enum | -| `galactic.datumapis.com/galactic=control` | `galactic-router-rr` | mode enum | -| `galactic.datumapis.com/fabric=router` | `fabric-router` (plain mode) | mode enum | -| `galactic.datumapis.com/fabric=control` | `fabric-router` (reflector mode, same image) | mode enum | +| Label | Deploys | Kind | +|-------------------------------------------|--------------------------------------------------------------------------------|-------------------------| +| `galactic.datumapis.com/node=compute` | nothing exclusively (see below) | primary role enum | +| `galactic.datumapis.com/node=edge` | `galactic-vrf` (ingress sidecar pods); hosts `galactic-gateway`/`galactic-nat` | primary role enum | +| `galactic.datumapis.com/gateway=enabled` | `galactic-gateway`, on a `node=edge` node only | opt-in flag (edge only) | +| `galactic.datumapis.com/nat=enabled` | `galactic-nat`, on a `node=edge` node only | opt-in flag (edge only) | +| `galactic.datumapis.com/galactic=router` | `galactic-cni`, `galactic-router` (plain/tenant mode) | mode enum | +| `galactic.datumapis.com/galactic=control` | `galactic-router-rr` | mode enum | +| `galactic.datumapis.com/fabric=router` | `fabric-router` (plain mode) | mode enum | +| `galactic.datumapis.com/fabric=control` | `fabric-router` (reflector mode, same image) | mode enum | `galactic-gateway` and `galactic-nat` are **never deployed automatically**: a node runs one only after an operator labels it, and their manifests live in -`base/` directories no default kustomization applies. Each requires its own -label *and* `galactic.datumapis.com/node` to be absent (`DoesNotExist`). They -are independent of each other, so a node can carry either label or both. +`base/` directories no default kustomization applies. Each requires +`galactic.datumapis.com/node=edge` *and* its own opt-in label, so they run +on edge nodes alongside `galactic-vrf`, and only on the edge nodes an +operator picks. They are independent of each other, so an edge node can +carry either opt-in label or both. Every affinity also excludes Kubernetes control-plane nodes (`node-role.kubernetes.io/control-plane: DoesNotExist`), independently of @@ -99,9 +101,9 @@ bug, not a hypothetical: configuration that's ever actually used. Fixed by dropping `nat66` from the enum and folding shard duty into `node=compute` directly. It has since moved to `node=edge`, where a shard is one hop from the transit its - masquerade addresses are originated into — every edge node now runs - `galactic-nat` unconditionally, sharing each uplink's XDP hook with - `galactic-gateway` through the node's XDP dispatcher. + masquerade addresses are originated into. An edge node runs it once it + carries `galactic.datumapis.com/nat=enabled`, sharing each uplink's XDP + hook with `galactic-gateway` through the node's XDP dispatcher. - `galactic-router-rr` used to require a dedicated `galactic.datumapis.com/galactic-route-reflector=true` boolean flag, independent of everything else. Once `galactic-cni`/`galactic-router` @@ -114,21 +116,32 @@ bug, not a hypothetical: --- -## What `edge` means, and why gateway and NAT sit outside `node` +## What `edge` means, and how gateway and NAT opt in **`galactic.datumapis.com/node=edge` marks the nodes that host `galactic-vrf`** — the ingress sidecar that runs next to Envoy and wires a -tenant VRF into the pod. Like `compute`, an `edge` node also runs -`galactic-cni`, `galactic-router` and `fabric-router`, through their own -labels (`galactic=router`, `fabric=router`). It does **not** run -`galactic-gateway` or `galactic-nat`. - -Earlier versions of this scheme used `node=edge` to select both of those, so -they could never be placed independently (one label, one value) and both -landed wherever `galactic-vrf`'s nodes were. They now have their own labels -and refuse any node that carries a `node` value at all. Because a node has at -most one `node` value, that makes them exclude `galactic-vrf` by -construction rather than by convention. +tenant VRF into the pod — **and the only nodes that may host +`galactic-gateway` and `galactic-nat`.** Like `compute`, an `edge` node also +runs `galactic-cni`, `galactic-router` and `fabric-router`, through their own +labels (`galactic=router`, `fabric=router`). + +`node=edge` alone does not deploy the gateway or a shard. Each needs its own +opt-in label on top (`gateway=enabled`, `nat=enabled`), so an operator picks +which edge nodes run which, and a cluster can have edge nodes that run +neither. Their affinities require both labels, so neither ever lands on a +`compute` node, a node with no `node` value, or a control-plane node. + +Earlier versions of this scheme went the other way twice. First `node=edge` +alone selected both, so they could never be placed independently (one label, +one value). Then they got their own labels but refused any node that carried +a `node` value at all, which kept them off `galactic-vrf`'s nodes. No +datapath reason required that: `galactic-vrf` claims nothing on the uplinks +(its replies arrive encapsulated to the node's own uSID, which the gateway +and the shard pass untouched), and egress shards are meant to run on edge +nodes. So all three now share the edge node. One configuration hazard +remains: an `EgressShard` whose SID reuses its node's router Block and +Node-ID captures that node's tenant ingress +([#711](https://github.com/datum-cloud/galactic/issues/711)). Note the historical wording elsewhere in this repo: "edge XDP" describes `galactic-gateway`'s datapath and predates this scheme; it is unrelated to @@ -160,23 +173,25 @@ a fallback). The nodes that host `galactic-vrf`. Runs (via `galactic.datumapis.com/galactic=router` and `galactic.datumapis.com/fabric=router`) `galactic-cni`, `galactic-router` and -`fabric-router`, and nothing keyed on this label by a manifest in this repo. -Never runs `galactic-gateway` or `galactic-nat`. +`fabric-router`. A manifest in this repo deploys nothing on this label alone, +but it is required by `galactic-gateway` and `galactic-nat`, which run here +when the node also carries `gateway=enabled` or `nat=enabled` (below). ### `galactic.datumapis.com/gateway=enabled` Runs `galactic-gateway`'s standalone, single-container pod (`config/galactic-gateway/base/daemonset.yaml`). Opt-in, never automatic, and -independent of `nat`. The node must carry no `galactic.datumapis.com/node` -label. `galactic-router` and `galactic-cni` run here too if the node also has -`galactic=router`, as their own independent DaemonSets. +independent of `nat`. The node must also carry +`galactic.datumapis.com/node=edge`. `galactic-router` and `galactic-cni` run +here too if the node also has `galactic=router`, as their own independent +DaemonSets. ### `galactic.datumapis.com/nat=enabled` Runs `galactic-nat`, the egress shard (`config/galactic-nat/base/daemonset.yaml`). Opt-in, never automatic, and -independent of `gateway`. The node must carry no `galactic.datumapis.com/node` -label. +independent of `gateway`. The node must also carry +`galactic.datumapis.com/node=edge`. A node with no gateway can use the base, whose `GALACTIC_NAT_XDP_ATTACH` defaults to `direct`: the shard attaches its own XDP program to its uplinks. @@ -233,14 +248,13 @@ Mutually exclusive with `fabric=router` on the same node, the same way | Node | `node` | `galactic` | `fabric` | Runs | |------------------------------------------------------------|-----------|------------|----------|------------------------------------------------------------------------------------| | `dfw-worker`, `sjc-worker`, `iad-worker` | `compute` | `router` | `router` | `galactic-cni`, `galactic-router`, `fabric-router` | -| `dfw-worker2`, `dfw-worker3`, `sjc-worker2`, `iad-worker2` | — | `router` | `router` | `galactic-nat` (`nat=enabled`), `galactic-cni`, `galactic-router`, `fabric-router` | +| `dfw-worker2`, `dfw-worker3`, `sjc-worker2`, `iad-worker2` | `edge` | `router` | `router` | `galactic-nat` (`nat=enabled`), `galactic-cni`, `galactic-router`, `fabric-router` | | `iad-worker3` | — | `control` | `router` | `galactic-router-rr`, `fabric-router` | -The lab has no `galactic-vrf` nodes, so no worker carries `node=edge`; its -NAT workers carry `nat=enabled` and no `node` value, which is what keeps -them apart from any future `node=edge` host. No worker carries -`gateway=enabled`: the lab deploys no `galactic-gateway`, so each shard -attaches its own XDP program rather than chaining behind one. +The lab's edge workers carry `node=edge` plus `nat=enabled`, which together +place a `galactic-nat` shard on each; the lab deploys no `galactic-vrf` +yet. No worker carries `gateway=enabled`: the lab deploys no +`galactic-gateway`, so each shard attaches its own XDP program. The reflector row is the one that shows why `galactic` is a mode enum rather than a boolean: `iad-worker3` carries no `node` value at all. Both `node` @@ -258,23 +272,25 @@ these DaemonSets. --- -## Migrating gateway and NAT nodes off `node=edge` - -Existing pods keep running when a node's labels change -(`IgnoredDuringExecution`), but a restarted pod reschedules only onto a node -that matches its affinity. Order matters, or a restart leaves no pod at all: - -1. On each gateway or NAT node, add `galactic.datumapis.com/gateway=enabled` - and/or `galactic.datumapis.com/nat=enabled`. Nothing changes yet. -2. Apply the new manifests. Running pods stay put. -3. On each of those nodes, remove `galactic.datumapis.com/node=edge`, then - restart its `galactic-gateway` / `galactic-nat` pods. -4. On a node running both, set `GALACTIC_NAT_XDP_ATTACH=dispatch` for the - shard, or it will refuse to attach over the gateway's dispatcher. - -Skipping step 1 and removing `node=edge` first leaves the DaemonSets with no -matching node the next time the pod restarts, which blackholes ingress or -egress for that node. +## Migrating gateway and NAT nodes onto `node=edge` + +The DaemonSet controller removes a pod from any node its affinity stops +matching, whether the node's labels or the DaemonSet's affinity changed. The +old manifests refuse a node carrying `galactic.datumapis.com/node` and the new +ones require `node=edge`, so no order avoids one restart of each gateway and +shard. Keep the gap to that restart: + +1. On each gateway or NAT node, add `galactic.datumapis.com/node=edge`, + keeping `galactic.datumapis.com/gateway=enabled` and/or + `galactic.datumapis.com/nat=enabled`. The old manifests remove the gateway + and shard pods from the node at once, and anything placed on `node=edge`, + such as `galactic-vrf` and the Envoy it serves, can now land on it. +2. Apply the new manifests straight away. They place the pods back on the + node. + +Production is not affected: infra sets its own placement and already requires +`node=edge`. A gateway or NAT node that should not become an edge node has no +place in the new scheme: move its role to an edge node instead. --- diff --git a/internal/manifests/placement_test.go b/internal/manifests/placement_test.go index 76df12e7..bb76acb4 100644 --- a/internal/manifests/placement_test.go +++ b/internal/manifests/placement_test.go @@ -21,6 +21,7 @@ const ( labelNAT = "galactic.datumapis.com/nat" enabled = "enabled" roleEdge = "edge" + roleCompute = "compute" controlPlane = "node-role.kubernetes.io/control-plane" ) @@ -85,32 +86,41 @@ func TestGatewayAndNATPlacement(t *testing.T) { wantNAT bool }{ {name: "unlabeled node runs neither (never automatic)", labels: map[string]string{}}, - {name: "compute node runs neither", labels: map[string]string{labelNode: "compute"}}, - {name: "edge node (galactic-vrf host) runs neither", labels: map[string]string{labelNode: roleEdge}}, - {name: "gateway only", labels: map[string]string{labelGateway: enabled}, wantGateway: true}, - {name: "nat only", labels: map[string]string{labelNAT: enabled}, wantNAT: true}, + {name: "compute node runs neither", labels: map[string]string{labelNode: roleCompute}}, + {name: "edge node without an opt-in label runs neither", labels: map[string]string{labelNode: roleEdge}}, { - name: "both labels run both", - labels: map[string]string{labelGateway: enabled, labelNAT: enabled}, - wantGateway: true, wantNAT: true, + name: "gateway label on an edge node runs the gateway", + labels: map[string]string{labelNode: roleEdge, labelGateway: enabled}, + wantGateway: true, + }, + { + name: "nat label on an edge node runs a shard", + labels: map[string]string{labelNode: roleEdge, labelNAT: enabled}, + wantNAT: true, }, { - name: "gateway label on an edge node is refused", - labels: map[string]string{labelGateway: enabled, labelNode: roleEdge}, + name: "both labels on an edge node run both", + labels: map[string]string{labelNode: roleEdge, labelGateway: enabled, labelNAT: enabled}, + wantGateway: true, wantNAT: true, }, { - name: "nat label on an edge node is refused", - labels: map[string]string{labelNAT: enabled, labelNode: roleEdge}, + name: "opt-in labels without a node role are refused", + labels: map[string]string{labelGateway: enabled, labelNAT: enabled}, }, { name: "both labels on a compute node are refused", - labels: map[string]string{labelGateway: enabled, labelNAT: enabled, labelNode: "compute"}, + labels: map[string]string{labelNode: roleCompute, labelGateway: enabled, labelNAT: enabled}, + }, + { + name: "control-plane edge node is refused", + labels: map[string]string{ + labelNode: roleEdge, labelGateway: enabled, labelNAT: enabled, controlPlane: "", + }, }, { - name: "control-plane node is refused", - labels: map[string]string{labelGateway: enabled, labelNAT: enabled, controlPlane: ""}, + name: "wrong label value is refused", + labels: map[string]string{labelNode: roleEdge, labelGateway: "true", labelNAT: "true"}, }, - {name: "wrong label value is refused", labels: map[string]string{labelGateway: "true", labelNAT: "true"}}, } for _, tc := range tests { t.Run(tc.name, func(t *testing.T) {