diff --git a/AGENTS.md b/AGENTS.md index a8630b1d..2ef956fb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -91,12 +91,12 @@ See [docs/node-labels.md](docs/node-labels.md) for the full strategy — why `ga - **`config/galactic-router/overlays/router/`** — the plain (non-reflector) per-node role (`galactic-router`); runs on every `compute` *and* every `edge` node except Kubernetes control-plane nodes, opt-in via `galactic.datumapis.com/galactic: router` (the same label `galactic-cni` keys off of — see `config/galactic-cni/daemonset.yaml`). This is the default role, so it carries no role-specific name/label of its own — only `control` below needs one, to coexist as a second DaemonSet. - **`config/galactic-router/overlays/control/`** — the BGP route-reflector role (`galactic-router-rr`, `GALACTIC_ROUTER_REFLECTOR=true`); opt-in only, requires `galactic.datumapis.com/galactic: control` (stays at zero replicas otherwise). Mutually exclusive with `router`'s `galactic: router` value on the same node — one label key, single-valued, so a route-reflector node structurally can't also run plain-mode `galactic-router`. `GALACTIC_ROUTER_BGP_LOCAL_ADDRESS` is auto-detected from the host's `lo` interface by default; see the comments in `daemonset-patch.yaml` for when to override it. - Apply both roles with `kubectl apply -k config/galactic-router/`, or a single role with e.g. `kubectl apply -k config/galactic-router/overlays/router/`. `galactic-router` no longer runs bundled inside `galactic-gateway`'s pod — see `config/galactic-gateway/` below. -- **`config/galactic-gateway/`** — the edge XDP DSR load-balancer control plane, a single-container `galactic-gateway` DaemonSet — `galactic-router` used to run as a second container in this same pod; it's now a fully independent DaemonSet (via `galactic.datumapis.com/galactic: router`, which a gateway node may carry like any other), so a crash on either side no longer takes the other's *pod* down with it, not just the other's binary. `config/galactic-gateway/{serviceaccount.yaml,rbac.yaml}` (safe/idempotent to apply cluster-wide) are what `kubectl apply -k config/galactic-gateway/` applies; `config/galactic-gateway/base/` (requiring nodes labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/gateway: enabled`, so it runs on edge nodes alongside `galactic-vrf`) is **not** included in that kustomization and is **not** applied as-is — the same exemption as `config/fabric-router/` below, for the same reason: `GALACTIC_GATEWAY_SRV6_ADDRESS` must be unique per gateway node and has no generic default (no in-cluster mechanism yet derives it automatically — see `internal/config/gateway.go`'s `EnvGatewaySRv6Address` doc comment). It's designed to be instantiated once per gateway node by a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's own public-interface/SRv6-address values; see `deploy/containerlab/resources/galactic-gateway/` as of commit `abdd665b` for a worked example (the containerlab lab no longer deploys the gateway). Also **not** part of the root `config/kustomization.yaml`'s default resource list, matching `config/fabric-router/`'s exemption. +- **`config/galactic-gateway/`** — the edge XDP DSR load-balancer control plane, a single-container `galactic-gateway` DaemonSet — `galactic-router` used to run as a second container in this same pod; it's now a fully independent DaemonSet (via `galactic.datumapis.com/galactic: router`, which a gateway node may carry like any other), so a crash on either side no longer takes the other's *pod* down with it, not just the other's binary. `config/galactic-gateway/{serviceaccount.yaml,rbac.yaml}` (safe/idempotent to apply cluster-wide) are what `kubectl apply -k config/galactic-gateway/` applies; `config/galactic-gateway/base/` (requiring nodes labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/gateway: enabled`, so it runs on edge nodes alongside `galactic-vrf`) is **not** included in that kustomization and is **not** applied as-is — the same exemption as `config/fabric-router/` below, for the same reason: `GALACTIC_GATEWAY_SRV6_ADDRESS` must be unique per gateway node and has no generic default (no in-cluster mechanism yet derives it automatically — see `internal/config/gateway.go`'s `EnvGatewaySRv6Address` doc comment). It's designed to be instantiated once per gateway node by a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's own public-interface/SRv6-address values; see `deploy/containerlab/resources/galactic-gateway/` for a worked example (one overlay per lab edge node). Also **not** part of the root `config/kustomization.yaml`'s default resource list, matching `config/fabric-router/`'s exemption. - **`config/galactic-nat/`** — the sharded egress translation (NAT66/NAT64) control plane, one `galactic-nat` DaemonSet per cluster on every node labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/nat: enabled`, alongside `galactic-vrf`. It is independent of `galactic-gateway`. A shard's identity (SID, masquerade addresses, NAT64 prefix) comes from the `EgressShard` targeting its node, not from the DaemonSet. The base leaves `GALACTIC_NAT_XDP_ATTACH` at its default, `direct`. On a node that also runs the gateway, set it to `dispatch`: an interface takes one native XDP program, so the shard and the gateway (which defaults to `GALACTIC_GATEWAY_XDP_ATTACH=dispatch`) each run from their own slot of the node's shared, pinned XDP dispatcher (`internal/plumbing/ebpf/xdpdispatch`), and either can restart or be turned off (`GALACTIC_*_DATAPATH_ENABLED=false`) without touching the other's traffic. `hack/xdp-dispatch-release.sh` detaches the dispatcher from a node. Compute nodes reach their own site's shards over SRv6 via `GALACTIC_CNI_EGRESS_SHARD_SIDS`. `kubectl apply -k config/galactic-nat/` applies only the ServiceAccount/RBAC; `base/` is excluded because uplink auto-detection is deployment-specific (see `config/galactic-nat/kustomization.yaml`). Not part of the root `config/kustomization.yaml`. - **`config/fabric-router/`** — the FRR underlay eBGP DaemonSet (`fabric-router`; `galactic-router` needs a working underlay before it can start). Unlike `config/galactic-router/`, this is a single flat DaemonSet with no `default`/`rr`-style role split — its affinity matches any node labeled `galactic.datumapis.com/fabric: router` (a dedicated mode-enum label, independent of whatever `galactic.datumapis.com/node` role value or `galactic.datumapis.com/galactic` mode that node also carries) directly, since there's no env/config difference between running on a regular node vs. those roles — fabric-router itself is identical everywhere; only the per-node BGP underlay config it reads from the ConfigMap differs. That affinity can legitimately match more than one node per cluster, and BGP underlay config (hostname, router-id, interface addresses, remote-AS) inherently differs per physical node — so each node's config lives in its own `fabric-router.` ConfigMap (a `frr.conf` key, plus optional `daemons`/`vtysh.conf` overrides of the image defaults) rather than one shared ConfigMap for the whole DaemonSet. `fabric-config-agent` (`cmd/fabric-config-agent`, `internal/fabricconfig`, built into the `fabric-router` image) reads it from the API: as the `frr-init` init container it validates (`vtysh -C`) and installs it, holding the pod in `Init` until a valid one exists; as the `config-agent` sidecar it applies later changes live with `frr-reload.py`, so an edit to one node's ConfigMap neither restarts the pod nor touches other nodes, and reports `Applied`/`Rejected`/`ReloadFailed`/`RestartRequired` events on the ConfigMap. The sidecar also hard-resets any IPv6 session announcing an IPv4-mapped next hop (`internal/fabricconfig/nexthop.go`) and serves that check's metrics on the node's port `9343`. A third container, `frr-exporter` (upstream `tynany/frr_exporter`, pinned by digest), reads the daemons' vty sockets in the same pod-scoped `/run/frr` and serves BGP peer state, message counters and prefix counts for both underlay families on the node's port `9342` (`hostNetwork`); `task verify:fabric-metrics` in `deploy/containerlab/` checks it on every lab node. The image's default `daemons` loads bgpd's BMP module (`-M bmp`), so a node's `frr.conf` can stream to a BMP collector (`config/bmp-collector/`) with a `bmp targets` block; loading it alone streams nothing, and a change to the modules needs a pod restart rather than a reload. Every lab node's `frr.conf` streams its pre-policy Adj-RIB-In and Loc-RIB for both families, sourced from `lo` (`source-interface lo`), since the underlay carries loopbacks but not link subnets. Production `frr.conf`s come from infra's renderer, which has to emit the same block for production nodes to stream. A node with no ConfigMap of its own falls back, with a `DeprecatedConfigSource` warning, to key `frr.conf.` of the legacy shared `fabric-config` ConfigMap. **Not** part of the root `config/kustomization.yaml` and not covered by `kubectl apply -k config/` — unlike every other component here, it has no generic default: the deployer must hand-author one `fabric-router.` ConfigMap per matching node before applying `kubectl apply -k config/fabric-router/` (which also applies the agent's ServiceAccount and namespaced Role). - **`config/bmp-collector/`** — a single-replica GoBMP Deployment and its `bmp-collector` Service (TCP `5000`), the BMP (RFC 7854) collector `galactic-router` and `fabric-router` stream to; `galactic-router` does so when `GALACTIC_ROUTER_BMP_STATIONS` names it (`bmp-collector.galactic-system.svc:5000`; see [docs/router/configuration.md](docs/router/configuration.md)). It parses every Peer Up/Down, route-monitoring and statistics message to JSON on stdout, so the cluster's log pipeline is its store; an overlay swapping `--dump=console` for `--kafka-server` publishes to Kafka instead. Opt-in, like `config/fabric-router/`: not part of the root `config/kustomization.yaml`, since nothing streams to it until a router is configured to. The containerlab lab runs one on `iad-worker3`'s host network (`deploy/containerlab/resources/bmp-collector/iad/`), because its three clusters cannot reach each other's Services, and points every node's `fabric-router` and the EVPN route reflector at it. Of the `galactic-router`s, only the reflector streams: its pre-policy Adj-RIB-In already holds every node's advertisements. - **`config/monitoring/`** — Prometheus Operator resources for BGP and NAT: a `fabric-router` PodMonitor (frr-exporter, port `frr-metrics`/`9342`, as `job=fabric-router`; fabric-config-agent, port `agent-metrics`/`9343`, as `job=fabric-config-agent`), a `galactic-router` PodMonitor covering both roles (port `metrics`/`9179`), a `galactic-nat` PodMonitor for the egress shards (port `metrics`/`9182`), each adding a `node` label, and the `galactic-bgp` PrometheusRule: underlay session down, administratively down for an hour, flapping, received prefixes halved, exporter collector or scrape failing, a session announcing an IPv4-mapped next hop, BGP routes not installed, the agent's metrics scrape down, a `galactic-router` BMP station or metrics scrape down, `galactic-router` routes failing to install, an egress shard's session table evicting live sessions, a single stalled NIC receive queue on a `galactic-nat` uplink (#673), and a `galactic-nat` metrics scrape down. Requires the `monitoring.coreos.com` CRDs, which this repo does not install, so it is not part of the root `config/kustomization.yaml`. `task test:alerts` unit-tests the rules with `promtool` (`config/monitoring/tests/`). Every alert carries what infra's alerting standard checks for (`service: galactic`, `team: connect`, `runbook_url` into infra's `docs/runbooks/galactic/README.md`). The PrometheusRule here is the tested source, not where the rules run on Datum's infra: there the PodMonitors are converted and scraped on each edge cluster, but edge vmalert never notifies, so infra evaluates a copy of the rules centrally. Change the rules here first, with their tests, then update infra's copy. -- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an SRv6 transit mesh whose underlay is dual-stack — every transit link and loopback carries an IPv4 and an IPv6 address, with one BGP session per address family (the SRv6 data plane itself stays IPv6-only; tenant IPv4 rides inside the encapsulation). Each site has one `compute` worker and at least one `edge` worker (dfw has two); every worker has its own eBGP uplink to its site's transit router, with FRR running as a hostNetwork DaemonSet. `galactic-router` (plain mode) distributes EVPN paths over iBGP to one route reflector for the whole lab (`iad-worker3`, `config/galactic-router/overlays/control/`) — compute and edge nodes never peer directly. Every `edge` node runs an egress shard (`galactic-nat`, run from the node's XDP dispatcher, `GALACTIC_NAT_XDP_ATTACH=dispatch`); the lab deploys no `galactic-gateway`. Each site's compute node egresses only through its own site's edge shards. A standalone off-cluster nginx host on the fourth transit router gives tenant egress a real destination. See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands. +- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an SRv6 transit mesh whose underlay is dual-stack — every transit link and loopback carries an IPv4 and an IPv6 address, with one BGP session per address family (the SRv6 data plane itself stays IPv6-only; tenant IPv4 rides inside the encapsulation). Each site has one `compute` worker and at least one `edge` worker (dfw has two); every worker has its own eBGP uplink to its site's transit router, with FRR running as a hostNetwork DaemonSet. `galactic-router` (plain mode) distributes EVPN paths over iBGP to one route reflector for the whole lab (`iad-worker3`, `config/galactic-router/overlays/control/`) — compute and edge nodes never peer directly. Every `edge` node runs an egress shard (`galactic-nat`) and a `galactic-gateway`, which share the node's uplinks through its XDP dispatcher (`GALACTIC_NAT_XDP_ATTACH=dispatch`). Each site's compute node egresses only through its own site's edge shards, and each site has its own VIP `/64`, served only through its own edge nodes to one `ns60` backend in that site. A standalone off-cluster host on the fourth transit router gives tenant egress a real destination and is the client for the VIP checks (`verify:gateway-ingress`). See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands. ## New Developer Entry Points diff --git a/README.md b/README.md index b96cfb5a..7777fd1c 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ Each component can also be applied on its own, e.g. `kubectl apply -k config/gal Two components are **not** part of `kubectl apply -k config/` and must be applied separately, each with its own per-node prerequisite: - **`config/fabric-router/`** — the FRR underlay eBGP DaemonSet `galactic-router` depends on. -- **`config/galactic-gateway/`** — the edge XDP DSR load-balancer control plane, a single-container `galactic-gateway` DaemonSet on nodes labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/gateway: enabled` (`galactic-router` runs there too, but as its own independent DaemonSet via `galactic.datumapis.com/galactic: router` — see `config/galactic-router/` above, not bundled into this pod). `kubectl apply -k config/galactic-gateway/` only installs the shared, cluster-safe ServiceAccount/RBAC; `config/galactic-gateway/base/` itself is a template meant to be instantiated once per gateway node by a further overlay (see `deploy/containerlab/resources/galactic-gateway/` as of commit `abdd665b`, the last before the containerlab lab stopped deploying the gateway, for a worked example) — apply that overlay per node instead of `base/` directly. +- **`config/galactic-gateway/`** — the edge XDP DSR load-balancer control plane, a single-container `galactic-gateway` DaemonSet on nodes labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/gateway: enabled` (`galactic-router` runs there too, but as its own independent DaemonSet via `galactic.datumapis.com/galactic: router` — see `config/galactic-router/` above, not bundled into this pod). `kubectl apply -k config/galactic-gateway/` only installs the shared, cluster-safe ServiceAccount/RBAC; `config/galactic-gateway/base/` itself is a template meant to be instantiated once per gateway node by a further overlay (see `deploy/containerlab/resources/galactic-gateway/` for a worked example) — apply that overlay per node instead of `base/` directly. ```bash kubectl apply -k config/fabric-router/ @@ -73,7 +73,7 @@ kubectl apply -k config/galactic-gateway/ A ConfigMap edit doesn't trigger a rollout, and doesn't need one: the `config-agent` sidecar applies a change to a node's `fabric-router.` ConfigMap to that node's running FRR in place. Check the result with `kubectl -n galactic-system get events --field-selector involvedObject.name=fabric-router.`: `Applied` carries the hash of the `frr.conf` the node now runs, `Rejected` means the change failed `vtysh -C` and the node kept its previous config, and `ReloadFailed` means `frr-reload.py` failed and the agent is retrying. A change to `daemons` can't be applied live; it raises `RestartRequired`, and takes effect when that node's pod is deleted. The sidecar also resets any IPv6 session announcing an IPv4-mapped next hop and raises `MappedNextHopReset` on the pod. -- **`config/galactic-gateway/`: per-node public interface and SRv6 address.** `config/galactic-gateway/base/daemonset.yaml`'s `galactic-gateway` container requires `GALACTIC_GATEWAY_PUBLIC_INTERFACE` and `GALACTIC_GATEWAY_SRV6_ADDRESS` — the latter must be unique per gateway node and has no generic default (there's no in-cluster mechanism yet that derives it automatically; see `internal/config/gateway.go`'s `EnvGatewaySRv6Address` doc comment). Applying `base/` as shipped, without pinning both per node, produces a crash-looping container. Instantiate `base/` via a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's values — see `deploy/containerlab/resources/galactic-gateway/` as of commit `abdd665b` for a worked example (the lab no longer deploys the gateway). +- **`config/galactic-gateway/`: per-node public interface and SRv6 address.** `config/galactic-gateway/base/daemonset.yaml`'s `galactic-gateway` container requires `GALACTIC_GATEWAY_PUBLIC_INTERFACE` and `GALACTIC_GATEWAY_SRV6_ADDRESS` — the latter must be unique per gateway node and has no generic default (there's no in-cluster mechanism yet that derives it automatically; see `internal/config/gateway.go`'s `EnvGatewaySRv6Address` doc comment). Applying `base/` as shipped, without pinning both per node, produces a crash-looping container. Instantiate `base/` via a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's values — see `deploy/containerlab/resources/galactic-gateway/` for a worked example. - **Talos: gRPC health port.** `galactic-router` runs `hostNetwork: true` and defaults to gRPC health checks on port `5000`, which collides with Talos's built-in dashboard (`/sbin/dashboard` permanently binds `127.0.0.1:5000` on every Talos node). `config/galactic-router/base/daemonset.yaml` already ships with `GALACTIC_ROUTER_GRPC_HEALTH_PORT=5179` (and matching probe/containerPort) to avoid this; if you run `galactic-router` outside these manifests on Talos, set `GALACTIC_ROUTER_GRPC_HEALTH_PORT` to something other than `5000` yourself. diff --git a/config/galactic-gateway/base/daemonset.yaml b/config/galactic-gateway/base/daemonset.yaml index 847c7076..943d5506 100644 --- a/config/galactic-gateway/base/daemonset.yaml +++ b/config/galactic-gateway/base/daemonset.yaml @@ -123,9 +123,8 @@ spec: # config.GatewayConfig.Validate requires both — # applying this base as-is produces a crash-looping container. # The deployer's per-node overlay must set both; see - # deploy/containerlab/resources/galactic-gateway/ as of - # commit abdd665b for a worked example (four instances, one - # per edge node; the lab no longer deploys the gateway). + # deploy/containerlab/resources/galactic-gateway/ for a worked + # example (four instances, one per edge node). ports: - name: metrics containerPort: 8081 diff --git a/config/galactic-gateway/base/kustomization.yaml b/config/galactic-gateway/base/kustomization.yaml index 7feeeced..01b4a1a5 100644 --- a/config/galactic-gateway/base/kustomization.yaml +++ b/config/galactic-gateway/base/kustomization.yaml @@ -3,8 +3,7 @@ # per gateway node by a further overlay that pins it to one node # (kubernetes.io/hostname) and sets that node's own # GALACTIC_GATEWAY_PUBLIC_INTERFACE/_SRV6_ADDRESS values. See -# deploy/containerlab/resources/galactic-gateway/ as of commit abdd665b for -# a worked example (four instances, one per edge node, each overlay named -# for its node; the lab no longer deploys the gateway). +# deploy/containerlab/resources/galactic-gateway/ for a worked example (four +# instances, one per edge node, each overlay named for its node). resources: - daemonset.yaml diff --git a/containers/remote-host/Dockerfile b/containers/remote-host/Dockerfile index 4844e3ba..30c2a0a1 100644 --- a/containers/remote-host/Dockerfile +++ b/containers/remote-host/Dockerfile @@ -13,13 +13,15 @@ FROM alpine:3.24 # iproute2: the startup script configures addresses and routes itself # (containerlab assigns nothing on a p2p link, and the FRR-based nodes get # theirs from zebra instead). curl/tcpdump/bind-tools are for driving and -# observing traffic from this end. +# observing traffic from this end. socat is the UDP client for the +# gateway's ingress VIP check (scripts/verify-gateway-ingress.sh). RUN apk add --no-cache \ nginx \ iproute2 \ curl \ tcpdump \ - bind-tools + bind-tools \ + socat # Serve something identifiable, so a curl from a tenant pod or another site # proves it reached *this* host rather than any other nginx in the lab. The diff --git a/deploy/containerlab/AGENTS.md b/deploy/containerlab/AGENTS.md index d261d144..4d504eb0 100644 --- a/deploy/containerlab/AGENTS.md +++ b/deploy/containerlab/AGENTS.md @@ -17,7 +17,7 @@ - **Worker SRv6 blackhole route**: A single standalone blackhole route covering the node's `/64` uSID space — its `/48` Block plus its own Node-ID, which is the width `internal/plumbing/ebpf/uformat.go`'s `LocatorBits = BlockBits + NodeIDBits` (64 bits) gives a node, so it covers exactly the SIDs that node can issue (`metric 2048`, no interface — `ip -6 route add blackhole` takes no `dev`) prevents the default route from matching any USID this node could compute, set via `exec` commands in the ContainerLab topology. One aggregate route rather than one per VPC: IPv6 FIB lookup is longest-prefix-match first, so a real seg6local `/128` decap route always wins over the covering blackhole regardless of metric — the aggregate protects every current and future VPC's SID with no per-VPC config needed. The USID is **not** configured as an address on any interface. The seg6local route (`metric 1024`) in the main table handles SRv6 decapsulation. If the USID were an interface address, the kernel's local table route (metric 0) would shadow the seg6local route and break decapsulation. (A dedicated `lo-galactic` dummy interface previously hosted this — removed since it wasn't referenced by any Go code or by the blackhole route itself; `install.sh`'s `all`/`default` sysctl entries already cover the SRv6 kernel knobs it existed for.) - **Node labels**: this lab follows the same label strategy as production (see the root `AGENTS.md`'s "Node label strategy" table) — a primary-role enum (`node`) plus a mode enum per binary family (`galactic`, `fabric`), not one giant enum or a pile of independent booleans: - `galactic.datumapis.com/node: compute` — one per site (`dfw-worker`, `sjc-worker`, `iad-worker`); deploys nothing keyed on this label itself. Each compute node egresses only through its own site's NAT shards (`resources/galactic-cni//egress-shards-patch.yaml`). - - `galactic.datumapis.com/node: edge` plus `galactic.datumapis.com/nat: enabled` — `dfw-worker2`, `dfw-worker3`, `sjc-worker2`, `iad-worker2`, this lab's edge nodes; the NAT base requires both, so these deploy `galactic-nat` (one DaemonSet per site, identity from each node's `EgressShard`), so all four are egress shards — dfw has two. No node carries `galactic.datumapis.com/gateway`, so the lab runs no `galactic-gateway`; every shard still runs from the node's XDP dispatcher (`GALACTIC_NAT_XDP_ATTACH=dispatch`, set by the lab's NAT base). Each of these nodes carries a `galactic.datumapis.com/nat=enabled:NoSchedule` taint. Kind applies labels only when a cluster is created, so a lab brought up before `node=edge` was added needs `kubectl label node galactic.datumapis.com/node=edge` on all four before `deploy:galactic-nat` re-applies the base, or the shards are removed. Adding the label alone, with the old manifests still applied, removes them too. + - `galactic.datumapis.com/node: edge` plus `galactic.datumapis.com/nat: enabled` and `galactic.datumapis.com/gateway: enabled` — `dfw-worker2`, `dfw-worker3`, `sjc-worker2`, `iad-worker2`, this lab's edge nodes. The NAT base requires `node=edge` plus `nat=enabled`, so these deploy `galactic-nat` (one DaemonSet per site, identity from each node's `EgressShard`), so all four are egress shards — dfw has two. The gateway base requires `node=edge` plus `gateway=enabled`, so each also runs `galactic-gateway` (one DaemonSet per node, `galactic-gateway-`). The shard runs with `GALACTIC_NAT_XDP_ATTACH=dispatch`, so the two share the uplinks through the node's XDP dispatcher. Each of these nodes carries a `galactic.datumapis.com/nat=enabled:NoSchedule` taint. Kind applies labels only when a cluster is created, so a lab brought up before a label was added needs it applied by hand on all four: `kubectl label node galactic.datumapis.com/node=edge` before `deploy:galactic-nat` re-applies the base, or the shards are removed (adding the label alone, with the old manifests still applied, removes them too), and `kubectl label node galactic.datumapis.com/gateway=enabled` before `deploy:galactic-gateway`, or no gateway pod schedules. - `galactic.datumapis.com/galactic: router` — every `compute` node and every NAT node (seven in all); deploys `galactic-cni` and plain-mode `galactic-router` together. Mutually exclusive with `galactic: control` below (one label key, one value). - `galactic.datumapis.com/galactic: control` — `iad-worker3` only; deploys `galactic-router-rr`. Not a `galactic.datumapis.com/node` value — a matching `NoSchedule` taint uses the same key/value. - `galactic.datumapis.com/fabric: router` — every worker in the lab (all eight); deploys `fabric-router`, independently of whatever other labels a node carries. Note this is *not* the same as being transit-facing: every worker runs FRR, but only the edge nodes' FRR holds an eBGP session to the transit; the rest run iBGP to their site's edge node. @@ -33,8 +33,9 @@ - **Galactic Router** — galactic-router DaemonSets running on each cluster's workers; handle iBGP EVPN route distribution. Resources live in `resources/galactic-router/`. - **Control** — the route-reflector node's resources (`resources/galactic-control/iad/`); its FRR fabric comes from the same `resources/fabric-router/iad/` DaemonSet as every other iad node. - **iad-control** — `iad-worker3`, the lab's single EVPN route reflector; every galactic-router in every cluster peers with it. galactic-router + BGP CRDs in `resources/galactic-control/iad/` (one `bgppeer-.yaml` per client), fabric in `resources/fabric-router/iad/` (`frr.conf.iad-worker3`). -- **Edge** — the four `nat=enabled` workers (`dfw-worker2`, `dfw-worker3`, `sjc-worker2`, `iad-worker2`): each site's transit-facing nodes, the uplinks its compute node routes through, and the egress shards. Their `BGPRouter` and route-reflector `BGPPeer` live alongside the compute node's in `resources/galactic-router//` (`bgprouter-.yaml`, `bgppeer-.yaml`). The lab deploys no `galactic-gateway`. -- **Egress** — `galactic-nat`, one egress shard per edge node, run from the node's XDP dispatcher (`GALACTIC_NAT_XDP_ATTACH=dispatch`) on that node's `bond0`/`bond1`. One per-site overlay under `resources/galactic-nat//` (uplink override + one `EgressShard` per edge node; dfw has two). Each site's compute node lists only its own site's shard SIDs (`resources/galactic-cni//egress-shards-patch.yaml`), and each edge node's `frr.conf` originates its shard's SID `/64` and masquerade addresses. +- **Edge** — the four `nat=enabled` workers (`dfw-worker2`, `dfw-worker3`, `sjc-worker2`, `iad-worker2`): each site's transit-facing nodes, the uplinks its compute node routes through, the egress shards and the VIP gateways. Their `BGPRouter` and route-reflector `BGPPeer` live alongside the compute node's in `resources/galactic-router//` (`bgprouter-.yaml`, `bgppeer-.yaml`). +- **Egress** — `galactic-nat`, one egress shard per edge node, running from that node's XDP dispatcher on `bond0`/`bond1`. One per-site overlay under `resources/galactic-nat//` (uplink override + one `EgressShard` per edge node; dfw has two). Each site's compute node lists only its own site's shard SIDs (`resources/galactic-cni//egress-shards-patch.yaml`), and each edge node's `frr.conf` originates its shard's SID `/64` and masquerade addresses. +- **Ingress** — `galactic-gateway`, one DaemonSet per edge node (`resources/galactic-gateway//`: hostname pin, `PUBLIC_INTERFACE=bond0`, `INTERNAL_INTERFACES=bond1`, `SRV6_ADDRESS` = the node's own uSID, and its `NetworkGateway`). Each site directory (`resources/galactic-gateway//`) adds the `ns60-tcp`/`ns60-udp` `NetworkRule`s for that site's own VIP and one `ServiceVIPBinding` per rule for the site's single `ns60` backend (`resources/tenants/ns60/`, fixed address from its NAD). Each edge node's `frr.conf` originates its site's VIP `/64` (`2001:db8:6060:{1,2,3}::/64` for dfw, sjc, iad) to `Null0`. One backend per site, because a node's VIP rewrite (`usid.c`'s `vip_xlat_key`) holds no VIP or backend address, so two backends of one VIP port in one VPC on one node would collide (#755). ## References diff --git a/deploy/containerlab/README.md b/deploy/containerlab/README.md index e2e35f20..d980dc43 100644 --- a/deploy/containerlab/README.md +++ b/deploy/containerlab/README.md @@ -44,8 +44,8 @@ two bonds are equal paths — see the BGP design below. Every link that carries a datapath is an 802.3ad (LACP) bond of two veths: each edge node's transit uplink, and each compute node's uplink to its edge node. Production edge uplinks are bonds, and each datapath resolves a bond to its -members before attaching — `galactic-nat` attaches XDP to -the members only, `galactic-cni` attaches TC to the master and its members — +members before attaching — the XDP dispatcher `galactic-nat` and +`galactic-gateway` run from attaches to the members only, `galactic-cni` attaches TC to the master and its members — so the lab presents bonds to exercise those code paths rather than leaving them to be found out in production. The route reflector's link, the `remote-host` link and the transit mesh stay single links: nothing attaches to them. @@ -111,27 +111,28 @@ hanging off `tr4`, the transit router with no site attached. |---------------------|---------------|--------------------------------------------------------------------------------| | `dfw-control-plane` | ext-container | Kind control-plane; runs Cilium, Multus | | `dfw-worker` | ext-container | compute: FRR PE, galactic-router PE, galactic-cni | -| `dfw-worker2` | ext-container | edge: FRR PE, galactic-router PE, galactic-cni, egress shard | +| `dfw-worker2` | ext-container | edge: FRR PE, galactic-router PE, galactic-cni, egress shard, galactic-gateway | | `dfw-worker3` | ext-container | edge: dfw's second edge node, same components as `dfw-worker2` | | `sjc-control-plane` | ext-container | Kind control-plane; runs Cilium, Multus | | `sjc-worker` | ext-container | compute: FRR PE, galactic-router PE, galactic-cni | -| `sjc-worker2` | ext-container | edge: FRR PE, galactic-router PE, galactic-cni, egress shard | +| `sjc-worker2` | ext-container | edge: FRR PE, galactic-router PE, galactic-cni, egress shard, galactic-gateway | | `iad-control-plane` | ext-container | Kind control-plane; runs Cilium, Multus | | `iad-worker` | ext-container | compute: FRR PE, galactic-router PE, galactic-cni | -| `iad-worker2` | ext-container | edge: FRR PE, galactic-router PE, galactic-cni, egress shard | +| `iad-worker2` | ext-container | edge: FRR PE, galactic-router PE, galactic-cni, egress shard, galactic-gateway | | `iad-worker3` | ext-container | EVPN route reflector for all three clusters; FRR PE only | | `tr1`–`tr4` | linux (FRR) | iBGP full mesh, AS 65100 | -| `remote-host` | linux (nginx) | off-fabric host on `tr4`; runs no Galactic component, no BGP | +| `remote-host` | linux (nginx) | off-fabric host on `tr4`; runs no Galactic component, no BGP; VIP client | Every edge node is tainted (`galactic.datumapis.com/nat=enabled:NoSchedule`) and so is the route reflector (`galactic.datumapis.com/galactic=control:NoSchedule`): no tenant pods land on either, only DaemonSets with a blanket toleration. Edge nodes run `galactic-cni` and plain-mode `galactic-router` as their own independent DaemonSets, exactly like compute -nodes, plus `galactic-nat`, the egress shard, which runs from the node's shared XDP -dispatcher on its uplinks (`GALACTIC_NAT_XDP_ATTACH=dispatch`). No node in the lab runs -`galactic-gateway`, so nothing needs the dispatcher to share the hook, but the lab uses it -anyway: it is the mode production edges are moving to, and a shard restart then bounces no -uplink. +nodes, plus `galactic-nat`, the egress shard, and `galactic-gateway`, the VIP load balancer +(one DaemonSet per node, `galactic-gateway-`). Both run from the node's XDP dispatcher +(`GALACTIC_NAT_XDP_ATTACH=dispatch`; the gateway's default), each in its own slot, so either +can restart without detaching the other. Each site has its own VIP `/64` +(`2001:db8:6060:1::/64` for dfw, `:2::` for sjc, `:3::` for iad), originated only by that +site's edge nodes and served by one `ns60` backend on the site's compute node. Compute nodes run no shard: each egresses through its own site's edge shards only. The reflector runs neither `galactic-cni` nor a shard — its `galactic=control` label is mutually exclusive with the `galactic=router` value that pulls @@ -140,12 +141,13 @@ those in, which is why the reflector needs a worker of its own (see veth's native `XDP_TX`/`XDP_REDIRECT` fast path does not deliver a frame to the peer interface's normal receive stack at all unless the peer *also* runs an XDP program, and -both legs of a shard leave from the driver via XDP. A real edge node's uplinks are +both legs of a shard, and both of the gateway's programs, leave from the driver via XDP. A real edge node's uplinks are physical NICs, where this veth-specific behavior doesn't apply, so the lab works around it rather than changing the datapath: `task deploy:lab-xdp-passthrough` loads a trivial pass-through XDP program on every transit port facing an edge node (where a translated -packet leaves), and `task deploy:lab-xdp-passthrough-compute` on the compute-node bond -members a shard redirects un-translated replies onto (`node_files/common/xdp-passthrough.c`), +packet or a backend's VIP reply leaves), and `task deploy:lab-xdp-passthrough-compute` on +the compute-node bond members a shard redirects un-translated replies onto and `edge_lb` +redirects VIP traffic onto (`node_files/common/xdp-passthrough.c`), both already wired into `task deploy`. The compute half runs after `deploy:cni`: loaded before Cilium, `ip` mounts a private bpffs on `/sys/fs/bpf` and Cilium then refuses to start on that node. @@ -294,12 +296,12 @@ node running both tenant delivery and an egress shard needs two Node-IDs, not one. Sharing one would let `galactic-nat`'s XDP program claim that node's own tenant ingress before the TC decap hook ever ran, silently. -| Service | Node-ID range | Identity | -|-------------|-------------------|--------------------------------------------------------------------| -| `0x1` | `0x1000`–`0x1FFF` | Tenant delivery — every node's `BGPRouter` | -| `0x2` | `0x2000`–`0x2FFF` | NAT egress shard (`EgressShard.spec.shardSID`) | -| `0x3` | `0x3000`–`0x3FFF` | Edge gateway (`GALACTIC_GATEWAY_SRV6_ADDRESS`); unused in this lab | -| `0x4`–`0xD` | — | Unallocated | +| Service | Node-ID range | Identity | +|-------------|-------------------|--------------------------------------------------------------------------------------------------------------------| +| `0x1` | `0x1000`–`0x1FFF` | Tenant delivery — every node's `BGPRouter` | +| `0x2` | `0x2000`–`0x2FFF` | NAT egress shard (`EgressShard.spec.shardSID`) | +| `0x3` | `0x3000`–`0x3FFF` | Edge gateway (`GALACTIC_GATEWAY_SRV6_ADDRESS`); unused in this lab, whose gateways use their node's own `0x1` uSID | +| `0x4`–`0xD` | — | Unallocated | `0x0` is left reserved (it is the one short range, since Node-ID `0x0000` is invalid), and `0xE`–`0xF` are the LIB — Function space, not Node-ID space. See @@ -413,16 +415,20 @@ deploy/containerlab/ │ ├── galactic-control/iad/ # the EVPN route reflector + one BGPPeer per client │ ├── bmp-collector/iad/ # the lab's one BMP collector, on the reflector's host network │ ├── galactic-nat/ # egress shards per site, one per edge node (dfw has two) +│ ├── galactic-gateway/ # one gateway overlay per edge node, plus each site's +│ │ # ns60 NetworkRules and ServiceVIPBindings │ └── tenants/ # test VPCs — one shared base/ (Namespace + netshoot │ ├── base/ # Deployment), each tenant patching its namespace and │ ├── ns10/ # default-network annotation; per-site dirs hold each │ ├── ns20/ # site's NAD(s): ns10 (IPv6-only, 3-site), ns20 │ ├── ns30/ # (dual-stack, 3-site), ns30 (IPv6-only, dfw only, 2 -│ └── ns40/ # attachments), ns40 (IPv4-only, iad only, 2 -│ # attachments) — ns30/ns40's two attachments are each -│ # their own NAD+Deployment (distinct vpcattachment, -│ # same vpc), not one NAD scaled to replicas: 2 — see -│ # docs/tenants.md for why. +│ ├── ns40/ # attachments), ns40 (IPv4-only, iad only, 2 +│ │ # attachments) — ns30/ns40's two attachments are each +│ │ # their own NAD+Deployment (distinct vpcattachment, +│ │ # same vpc), not one NAD scaled to replicas: 2 — see +│ │ # docs/tenants.md for why. +│ └── ns60/ # the gateway's VIP backend, one per site at a fixed +│ # address, answering TCP and UDP on port 80 ├── node_files/ │ ├── dfw/ config.yaml │ ├── iad/ config.yaml @@ -475,6 +481,7 @@ task deploy | `build:node` | Build the custom `kindest/node:galactic` image | | `build:galactic-router` | Build the galactic-router container from Go source | | `build:galactic-cni` | Build the galactic-cni installer image | +| `build:galactic-gateway` | Build the galactic-gateway edge load balancer image | | `build:frr` | Build the FRR container from Alpine edge | | `build:remote-host` | Build the off-fabric nginx host image | | `deploy` | Build images, apply host sysctls, and deploy the lab | @@ -486,14 +493,20 @@ task deploy | `deploy:fabric` | Apply FRR DaemonSets to all clusters | | `deploy:bmp-collector` | Install the BMP collector on iad-worker3, where the route reflector streams to it | | `deploy:galactic-router` | Apply galactic-router DaemonSets and BGP CRs | +| `deploy:galactic-gateway` | Install galactic-gateway on every edge node, with each site's VIP rules (after `deploy:scenarios`) | | `deploy:scenarios` | Deploy all VPC test scenarios | | `deploy:ns10` | Deploy ns10 test VPC (IPv6-only, fd20 ULA) | | `deploy:ns20` | Deploy ns20 test VPC (dual-stack, fd20 ULA + IPv4) | | `deploy:ns30` | Deploy ns30 test VPC (dfw only, 2 pods) | | `deploy:ns40` | Deploy ns40 test VPC (iad only, 2 pods) | +| `deploy:ns60` | Deploy ns60 test VPC (one VIP backend per site) | | `verify:fabric-metrics` | Scrape every fabric-router's frr-exporter; fail on a failed collector or a session not Established | | `verify:bmp` | Check the route reflector and every fabric-router stream to the BMP collector, and that it agrees with each on every session's state | | `verify:underlay` | Ping every underlay loopback from tr1 over both IPv4 and IPv6 | +| `verify:gateway` | Check every NetworkGateway is Ready, every NetworkRule Accepted, every ServiceVIPBinding Bound | +| `verify:gateway-ingress` | TCP and UDP from the off-fabric host to each site's VIP, through every gateway node in turn | +| `verify:gateway-restart` | Restart dfw-worker2's gateway and shard in turn while the other carries traffic (#710) | +| `verify:gateway-detach` | Turn dfw-worker2's gateway datapath off: the ingress check must fail, then pass (not in `verify`) | | `verify:nat-datapath` | Full egress round trip to the off-fabric host, IPv6 (NAT66) and IPv4 (NAT64) | | `verify:nat-return-route` | Prove a reply from outside the fabric reaches each shard's masquerade addresses | | `verify:nat-local` | Prove each site's tenants egress through that site's own edge shard | diff --git a/deploy/containerlab/Taskfile.yaml b/deploy/containerlab/Taskfile.yaml index c1622488..0343ba08 100644 --- a/deploy/containerlab/Taskfile.yaml +++ b/deploy/containerlab/Taskfile.yaml @@ -22,6 +22,7 @@ tasks: - "build:galactic-router" - "build:galactic-cni" - "build:galactic-nat" + - "build:galactic-gateway" - "build:remote-host" "build:node": @@ -37,7 +38,7 @@ tasks: # --build-context network=../../../network satisfies go.mod's local # `replace go.datum.net/network => ../network` (see that line's own # comment) inside each Dockerfile's "network" build stage -- see - # containers/galactic-{router,cni,nat}/Dockerfile's own comment on + # containers/galactic-{router,cni,nat,gateway}/Dockerfile's own comment on # that stage for the full mechanism. ../../../network is this Taskfile's # own cwd (deploy/containerlab) relative to the network repo checked out # as galactic's sibling; harmless to pass even once that replace @@ -59,6 +60,11 @@ tasks: cmds: - docker build --network=host --build-context network=../../../network -t galactic-nat:latest -f ../../containers/galactic-nat/Dockerfile ../.. + "build:galactic-gateway": + desc: Build the galactic-gateway edge load balancer image + cmds: + - docker build --network=host --build-context network=../../../network -t galactic-gateway:latest -f ../../containers/galactic-gateway/Dockerfile ../.. + "build:remote-host": desc: Build the off-fabric remote host image (nginx) cmds: @@ -83,6 +89,9 @@ tasks: - task: "deploy:galactic-router" - task: "deploy:galactic-nat" - task: "deploy:scenarios" + # After the scenarios: the gateway's ServiceVIPBindings bind ns60's + # backends, which deploy:scenarios creates. + - task: "deploy:galactic-gateway" "deploy:clusters": desc: Create Kind clusters @@ -220,8 +229,9 @@ tasks: dfw-worker dfw-worker2 dfw-worker3 sjc-worker sjc-worker2 iad-worker iad-worker2 - # Egress shards: one per edge node. + # Egress shards and gateways: one each per edge node. NAT_NODES: "dfw-worker2 dfw-worker3 sjc-worker2 iad-worker2" + GATEWAY_NODES: "dfw-worker2 dfw-worker3 sjc-worker2 iad-worker2" cmds: - | set -e @@ -237,6 +247,7 @@ tasks: load galactic-router:latest {{.FABRIC_NODES}} load galactic-cni:latest {{.CNI_NODES}} load galactic-nat:latest {{.NAT_NODES}} + load galactic-gateway:latest {{.GATEWAY_NODES}} "deploy:system": desc: Install CRDs, namespace, and RBAC @@ -291,6 +302,11 @@ tasks: cmds: - ./scripts/deploy-galactic-nat.sh + "deploy:galactic-gateway": + desc: Install galactic-gateway on every edge node, with each site's VIP rules + cmds: + - ./scripts/deploy-galactic-gateway.sh + "deploy:scenarios": desc: Deploy all VPC test scenarios cmds: @@ -298,6 +314,7 @@ tasks: - task: "deploy:ns20" - task: "deploy:ns30" - task: "deploy:ns40" + - task: "deploy:ns60" - task: "deploy:ns71" "deploy:ns10": @@ -320,6 +337,11 @@ tasks: cmds: - ./scripts/deploy-ns.sh ns40 iad + "deploy:ns60": + desc: Deploy ns60 test VPC (one VIP backend per site) + cmds: + - ./scripts/deploy-ns.sh ns60 dfw sjc iad + "deploy:ns71": desc: Deploy ns71 test VPCs (overlapping ULA on one node, same-node collision fixture) cmds: @@ -337,6 +359,9 @@ tasks: - task: "verify:underlay" - task: "verify:srv6" - task: "verify:evpn" + - task: "verify:gateway" + - task: "verify:gateway-ingress" + - task: "verify:gateway-restart" - task: "verify:nat-sharding" - task: "verify:nat-egress" - task: "verify:nat-return-route" @@ -446,6 +471,56 @@ tasks: - docker exec sjc-control-plane kubectl get bgprouters -A - docker exec dfw-control-plane kubectl get bgprouters -A + "verify:gateway": + desc: Verify every edge node's gateway is ready and every site's VIP rules are accepted and bound + cmds: + - | + set -e + rc=0 + for site in dfw sjc iad; do + cp=${site}-control-plane + echo "--- ${site} ---" + docker exec ${cp} kubectl -n galactic-system get networkgateways,networkrules,servicevipbindings + # Ready: the engine loaded every rule and advertised it. Accepted: + # the rule found a gateway. Bound: the backend's node rewrites the + # VIP for it. + for check in "networkgateway Ready" "networkrule Accepted" "servicevipbinding Bound"; do + set -- ${check} + if ! docker exec ${cp} kubectl -n galactic-system wait $1 --all \ + --for=condition=$2 --timeout=10s >/dev/null; then + echo "FAIL ${site}: not every $1 is $2" >&2 + rc=1 + fi + done + done + exit ${rc} + + # verify:gateway reads CRDs, so it passes on a datapath where no + # connection completes. This sends real TCP and UDP from the off-fabric + # host to each site's VIP, through each of the site's gateway nodes in + # turn, and checks the replies and the datapath's own counters. See the + # script's header for each assertion. + "verify:gateway-ingress": + desc: TCP and UDP from the off-fabric host to each site's VIP, through every gateway node + cmds: + - LAB={{.LAB}} ./scripts/verify-gateway-ingress.sh + + # #710: the gateway and the egress shard share each edge node's uplinks + # through its XDP dispatcher, so restarting either must leave the other's + # traffic flowing. + "verify:gateway-restart": + desc: Restart dfw-worker2's gateway and shard in turn while the other carries traffic + cmds: + - LAB={{.LAB}} ./scripts/verify-gateway-restart.sh + + # Not part of `verify`: dfw-worker2 serves no VIP while it runs. Proves + # verify:gateway-ingress fails when a gateway's XDP program is off its + # uplinks. + "verify:gateway-detach": + desc: Turn dfw-worker2's gateway datapath off and confirm the ingress check fails, then passes again + cmds: + - LAB={{.LAB}} ./scripts/verify-gateway-detach.sh + "verify:nat-sharding": desc: Verify every edge node's EgressShard is programmed and advertised cmds: @@ -1001,6 +1076,7 @@ tasks: - docker rmi kindest/node:galactic || true - docker rmi galactic-router:latest || true - docker rmi galactic-nat:latest || true + - docker rmi galactic-gateway:latest || true - docker rmi galactic-cni:latest || true - docker rmi {{.FABRIC_ROUTER_IMAGE}} || true - docker rmi remote-host:latest || true diff --git a/deploy/containerlab/docs/verification.md b/deploy/containerlab/docs/verification.md index ba966fd1..5b146a95 100644 --- a/deploy/containerlab/docs/verification.md +++ b/deploy/containerlab/docs/verification.md @@ -177,8 +177,10 @@ docker exec dfw-worker2 curl -sS --max-time 5 http://[2001:db8:1:40::2]/ ## Egress shards Four edge nodes across three sites (`dfw-worker2`, `dfw-worker3`, `sjc-worker2`, -`iad-worker2`), each running its own shard from the node's XDP dispatcher on -its uplinks. +`iad-worker2`), each running its own shard. Every one of them also runs +`galactic-gateway`, so the shard runs from its own slot of the node's XDP +dispatcher (`GALACTIC_NAT_XDP_ATTACH=dispatch`) rather than attaching its own +program. ```bash task verify:nat-sharding @@ -187,11 +189,45 @@ task verify:nat-sharding docker exec dfw-control-plane kubectl get pods -n galactic-system -o wide \ -l app.kubernetes.io/name=galactic-nat -# The XDP dispatcher, on every member of both bonds (eth1-eth4); the shard runs -# from one of its slots +# The dispatcher's root program, on every member of both bonds (eth1-eth4); +# the shard and the gateway run from its slots docker exec dfw-worker2 ip -d link show dev eth1 | grep -o 'prog/xdp id [0-9]*' ``` +## Ingress gateway + +The same four edge nodes each run `galactic-gateway` (`galactic-gateway-`) +and a `NetworkGateway` named after the node. Each site has its own VIP +(`2001:db8:6060:1::1` in dfw, `:2::1` in sjc, `:3::1` in iad), in a `/64` only +that site's edge nodes originate, and one `ns60` backend on its compute node +that answers TCP and UDP on port 80 with its pod name and the client address +it saw. + +```bash +# NetworkGateways Ready, NetworkRules Accepted, ServiceVIPBindings Bound +task verify:gateway + +# TCP and UDP from remote-host to every site's VIP, pinned through each +# gateway node in turn; checks the replies and the datapath's counters +task verify:gateway-ingress + +# Restart dfw-worker2's gateway, then its shard, while the other carries +# traffic (#710) +task verify:gateway-restart + +# Turn dfw-worker2's datapath off; the ingress check must fail, then pass +# again. Disruptive -- not part of `task verify` +task verify:gateway-detach + +# By hand: one TCP request to dfw's VIP, from the off-fabric host +echo probe | docker exec -i clab-gvpc-remote-host \ + socat -t2 -T3 - 'TCP6:[2001:db8:6060:1::1]:80' + +# The datapath's own account of it, on a gateway node +docker exec dfw-worker2 curl -s http://localhost:8081/metrics \ + | grep -E '^galactic_edge_(rule|return)_packets_total' +``` + ## EVPN route reflector `iad-worker3` is the single reflector for all three clusters; every @@ -384,11 +420,14 @@ XDP program could not do. ## Automated checks ```bash -task verify # run all verification (bgp-transit, bgp-fabric, bgp-peers, underlay, srv6, evpn, nat, scenarios) +task verify # run all verification (bgp-transit, bgp-fabric, bgp-peers, underlay, srv6, evpn, gateway, nat, scenarios) task verify:bgp-transit task verify:bgp-fabric task verify:bgp-peers task verify:underlay task verify:srv6 task verify:evpn +task verify:gateway +task verify:gateway-ingress +task verify:gateway-restart ``` diff --git a/deploy/containerlab/group_files/common/mkbond.sh b/deploy/containerlab/group_files/common/mkbond.sh index 9a40d82e..72e98afc 100755 --- a/deploy/containerlab/group_files/common/mkbond.sh +++ b/deploy/containerlab/group_files/common/mkbond.sh @@ -6,12 +6,11 @@ # aggregate can never disagree on its LACP parameters. # # Why these parameters: -# mode 802.3ad what production edge uplinks run, and what -# galactic-gateway's per-slave attach gate waits on (it -# reads each slave's LACP actor state -- see -# internal/plumbing/ebpf/edgeattach/gate.go). The lab -# no longer deploys the gateway; galactic-nat attaches -# to the slaves without such a gate. +# mode 802.3ad what production edge uplinks run, and what the +# per-slave XDP attach gate galactic-gateway and +# galactic-nat share waits on (it reads each slave's +# LACP actor state -- see +# internal/plumbing/ebpf/xdpattach/gate.go). # miimon 100 non-zero is load-bearing: with miimon 0 the bonding # driver never re-polls carrier, so a slave that bounces # while a native XDP program attaches stays failed. diff --git a/deploy/containerlab/gvpc.clab.yaml b/deploy/containerlab/gvpc.clab.yaml index e59813d1..2debb239 100644 --- a/deploy/containerlab/gvpc.clab.yaml +++ b/deploy/containerlab/gvpc.clab.yaml @@ -292,7 +292,9 @@ topology: # cluster and running no Galactic component at all: it exists to be the # far end of traffic that leaves the fabric. nginx serves on its transit # addresses (both families), so a tenant pod's egress path through the - # NAT66 shard tier has a real destination to reach. + # NAT66 shard tier has a real destination to reach, and the host itself + # is a real off-fabric client for each site's ingress VIP + # (scripts/verify-gateway-ingress.sh). # It hangs off tr4, the one transit router with no site attached to it. # No BGP: a host doesn't speak it -- tr4 originates the link subnets # (node_files/tr4/frr.conf) and the host just keeps a default route @@ -345,15 +347,16 @@ topology: # them. dfw has two, an active-active pair; sjc and iad one each. # # mtu: 1500 on both ends, unlike containerlab's jumbo-frame veth default - # (~9500) -- required, not cosmetic: the egress shard (galactic-nat) - # attaches the XDP dispatcher to every member of bond0 in native/driver - # mode (xdpattach, XDPDriverMode), and the kernel's veth_xdp_set() rejects native + # (~9500) -- required, not cosmetic: the node's XDP dispatcher, which + # runs the egress shard (galactic-nat) and galactic-gateway's edge_lb, + # attaches to every member of bond0 in native/driver mode, and the + # kernel's veth_xdp_set() rejects native # XDP with -ERANGE whenever the *peer's* MTU exceeds what fits in one # page-sized buffer (SKB_WITH_OVERHEAD(PAGE_SIZE - VETH_XDP_HEADROOM), # ~3.5-3.8KB on a 4K-page host) -- 9500 blows well past that regardless - # of which end asks. Confirmed live: galactic-gateway, which held these - # hooks before the shards did, crash-looped with exactly this ERANGE - # until both ends were dropped to 1500. + # of which end asks. Confirmed live: galactic-gateway, which once held + # these hooks alone, crash-looped with exactly this ERANGE until both + # ends were dropped to 1500. - endpoints: ["dfw-worker2:eth1", "tr1:eth1"] # dfw-worker2 bond0 / tr1 bond1 mtu: 1500 - endpoints: ["dfw-worker2:eth3", "tr1:eth6"] @@ -378,11 +381,13 @@ topology: # resources/fabric-router/*/frr.conf.*). # # mtu: 1500 for the same -ERANGE reason as the edge uplinks above. The - # edge end runs the egress shard's XDP program on every member of these - # bonds, and the compute end runs the lab's pass-through XDP program on - # its members (Taskfile.yaml's deploy:lab-xdp-passthrough-compute) so - # that the shard's redirects of a reply toward the tenant's node are - # received at all. Both native XDP, both subject to the peer-MTU limit. + # edge end runs the node's XDP dispatcher on every member of these + # bonds (the egress shard, and galactic-gateway's edge_return), and the + # compute end runs the lab's pass-through XDP program on its members + # (Taskfile.yaml's deploy:lab-xdp-passthrough-compute) so that the + # shard's redirects of a reply toward the tenant's node, and edge_lb's + # redirects toward a backend, are received at all. Both native XDP, both + # subject to the peer-MTU limit. # (galactic-nat used to attach # here from the compute end, while its shards ran on the compute nodes; # it crash-looped with this ERANGE until these links were dropped to @@ -422,8 +427,9 @@ topology: # The route reflector sits behind iad's edge node exactly the way a # compute node does -- one iBGP session to it, no transit uplink of its - # own. It keeps the jumbo default: it does not run galactic-nat, so no - # XDP program ever attaches to its eth1. Nor is + # own. It keeps the jumbo default: it runs neither galactic-nat nor + # galactic-gateway, and iad-worker2's eth3 is in neither one's interface + # list, so no XDP program ever attaches to either end. Nor is # it bonded -- with no datapath on it, a bond would exercise nothing. - endpoints: ["iad-worker3:eth1", "iad-worker2:eth3"] diff --git a/deploy/containerlab/node_files/common/xdp-passthrough.c b/deploy/containerlab/node_files/common/xdp-passthrough.c index 1568ca3d..56da93a9 100644 --- a/deploy/containerlab/node_files/common/xdp-passthrough.c +++ b/deploy/containerlab/node_files/common/xdp-passthrough.c @@ -13,17 +13,20 @@ // affected traffic stay at exactly zero; attaching this program is what // makes them move). // -// The egress shard's datapath (internal/plumbing/ebpf/natprog) leaves every -// translated packet from the driver: XDP_TX back out the interface it -// arrived on, XDP_REDIRECT out another. On a real edge node's uplinks, -// genuine physical NICs, that is the ordinary high-throughput XDP pattern. -// This program's only job is making this lab's veth-pair simulation of -// those uplinks behave the way a real NIC already does, on whichever node -// sits on the *other* end of a link facing an edge node: the transit ports -// facing each edge node's transit bond, where a translated packet leaves -// toward the off-fabric host, and the compute-node ports facing each edge -// node's compute-facing bond, where a shard redirects an un-translated reply -// back toward the tenant's node. See deploy:lab-xdp-passthrough and +// Both XDP datapaths on an edge node leave their packets from the driver: +// the egress shard (internal/plumbing/ebpf/natprog) and galactic-gateway's +// edge_lb/edge_return (internal/plumbing/ebpf/edgeprog/edgedsr.c), XDP_TX +// back out the interface a packet arrived on, XDP_REDIRECT out another. On a +// real edge node's uplinks, genuine physical NICs, that is the ordinary +// high-throughput XDP pattern. This program's only job is making this lab's +// veth-pair simulation of those uplinks behave the way a real NIC already +// does, on whichever node sits on the *other* end of a link facing an edge +// node: the transit ports facing each edge node's transit bond, where a +// translated packet leaves toward the off-fabric host and edge_return sends +// a backend's reply toward the client, and the compute-node ports facing +// each edge node's compute-facing bond, where a shard redirects an +// un-translated reply back toward the tenant's node and edge_lb redirects a +// client's packet toward its backend. See deploy:lab-xdp-passthrough and // deploy:lab-xdp-passthrough-compute for the exact lists. Not applied // anywhere else in this topology: usid_ingress/usid_egress (usid.c) are // TC-BPF, not XDP, so ordinary tenant traffic never needed this. diff --git a/deploy/containerlab/node_files/dfw/config.yaml b/deploy/containerlab/node_files/dfw/config.yaml index 0d6dd209..be32db8a 100644 --- a/deploy/containerlab/node_files/dfw/config.yaml +++ b/deploy/containerlab/node_files/dfw/config.yaml @@ -14,16 +14,19 @@ nodes: galactic.datumapis.com/node: compute galactic.datumapis.com/galactic: router galactic.datumapis.com/fabric: router - # Two edge nodes, each running its own egress shard: dfw's compute node - # lists both in its shard list, so this is the only site where a tenant's - # egress has a second shard to fall back to. sjc and iad each run a single - # edge node. Every shard runs from the node's XDP dispatcher + # Two edge nodes, each running its own egress shard and galactic-gateway: + # dfw's compute node lists both shards in its shard list, so this is the + # only site where a tenant's egress has a second shard to fall back to, + # and the only site where a VIP has two gateways to enter through. sjc + # and iad each run a single edge node. The shard and the gateway share + # the node's uplinks through its XDP dispatcher # (GALACTIC_NAT_XDP_ATTACH=dispatch). - role: worker labels: topology.kubernetes.io/region: dfw galactic.datumapis.com/node: edge galactic.datumapis.com/nat: enabled + galactic.datumapis.com/gateway: enabled galactic.datumapis.com/galactic: router galactic.datumapis.com/fabric: router kubeadmConfigPatches: @@ -39,6 +42,7 @@ nodes: topology.kubernetes.io/region: dfw galactic.datumapis.com/node: edge galactic.datumapis.com/nat: enabled + galactic.datumapis.com/gateway: enabled galactic.datumapis.com/galactic: router galactic.datumapis.com/fabric: router kubeadmConfigPatches: diff --git a/deploy/containerlab/node_files/iad/config.yaml b/deploy/containerlab/node_files/iad/config.yaml index 3789f1b8..5e580a27 100644 --- a/deploy/containerlab/node_files/iad/config.yaml +++ b/deploy/containerlab/node_files/iad/config.yaml @@ -15,6 +15,7 @@ nodes: topology.kubernetes.io/region: iad galactic.datumapis.com/node: edge galactic.datumapis.com/nat: enabled + galactic.datumapis.com/gateway: enabled galactic.datumapis.com/galactic: router galactic.datumapis.com/fabric: router kubeadmConfigPatches: diff --git a/deploy/containerlab/node_files/sjc/config.yaml b/deploy/containerlab/node_files/sjc/config.yaml index 30bd6b59..47942c20 100644 --- a/deploy/containerlab/node_files/sjc/config.yaml +++ b/deploy/containerlab/node_files/sjc/config.yaml @@ -15,6 +15,7 @@ nodes: topology.kubernetes.io/region: sjc galactic.datumapis.com/node: edge galactic.datumapis.com/nat: enabled + galactic.datumapis.com/gateway: enabled galactic.datumapis.com/galactic: router galactic.datumapis.com/fabric: router kubeadmConfigPatches: diff --git a/deploy/containerlab/resources/fabric-router/dfw/frr.conf.dfw-worker2 b/deploy/containerlab/resources/fabric-router/dfw/frr.conf.dfw-worker2 index d57d9521..3c2c87d1 100644 --- a/deploy/containerlab/resources/fabric-router/dfw/frr.conf.dfw-worker2 +++ b/deploy/containerlab/resources/fabric-router/dfw/frr.conf.dfw-worker2 @@ -20,6 +20,24 @@ interface bond1 ipv6 address 2001:db8:1:10::1/64 ! +# This site's ingress VIP prefix (resources/galactic-gateway/dfw/), the +# /64 every VIP in dfw's NetworkRules sits in. A VIP's own reachability is +# carried as an EVPN Type 5 path inside galactic-router's iBGP mesh, which the +# plain-IPv6-unicast transit never sees, so without this the off-fabric host +# (remote-host on tr4) has no route to any VIP at all. +# +# Each site has its own prefix, originated by that site's edge nodes and no +# others: a load balancer belongs to one region (datum-cloud/enhancements#884), +# so its VIP enters the fabric only through its own site's gateways. Both of +# dfw's edge nodes originate it identically, so tr1 holds two equal-cost +# paths, and Maglev's table, identical on both gateways, picks the backend +# whichever one a flow arrives at. +# +# Null0 for the same reason as the shard prefixes below. The gateway's XDP +# program claims a VIP packet before any FIB lookup, so a packet that reaches +# this static is one no gateway claimed, and dropping it is the point. +ipv6 route 2001:db8:6060:1::/64 Null0 + # This node's egress shard (resources/galactic-nat/dfw/egressshard.yaml), # originated into the underlay: its SID's covering /64 and its two masquerade # addresses. @@ -98,6 +116,8 @@ router bgp 65000 # /128, not the site /48 -- a per-node peering loopback is the only # thing meant to cross the transit as a host route. network fc00:0:a::1/128 + # This site's ingress VIP prefix -- see this file's own comment above. + network 2001:db8:6060:1::/64 # This node's egress shard's SID /64 and IPv6 masquerade address -- see # this file's own comment above. network 2001:db8:ff01:2002::/64 diff --git a/deploy/containerlab/resources/fabric-router/dfw/frr.conf.dfw-worker3 b/deploy/containerlab/resources/fabric-router/dfw/frr.conf.dfw-worker3 index a9ae0e7e..186e8e41 100644 --- a/deploy/containerlab/resources/fabric-router/dfw/frr.conf.dfw-worker3 +++ b/deploy/containerlab/resources/fabric-router/dfw/frr.conf.dfw-worker3 @@ -20,6 +20,11 @@ interface bond1 ipv6 address 2001:db8:1:13::1/64 ! +# This site's ingress VIP prefix -- see +# resources/fabric-router/dfw/frr.conf.dfw-worker2 for why each site +# originates its own, from its own edge nodes only. +ipv6 route 2001:db8:6060:1::/64 Null0 + # This node's egress shard (resources/galactic-nat/dfw/egressshard.yaml), # originated into the underlay: its SID's covering /64 and its two masquerade # addresses. See resources/fabric-router/dfw/frr.conf.dfw-worker2 for the @@ -66,6 +71,8 @@ router bgp 65000 # /128, not the site /48 -- a per-node peering loopback is the only # thing meant to cross the transit as a host route. network fc00:0:b::1/128 + # This site's ingress VIP prefix -- see this file's own comment above. + network 2001:db8:6060:1::/64 # This node's egress shard's SID /64 and IPv6 masquerade address -- see # this file's own comment above. network 2001:db8:ff01:2003::/64 diff --git a/deploy/containerlab/resources/fabric-router/iad/frr.conf.iad-worker2 b/deploy/containerlab/resources/fabric-router/iad/frr.conf.iad-worker2 index 387323cd..27e58df5 100644 --- a/deploy/containerlab/resources/fabric-router/iad/frr.conf.iad-worker2 +++ b/deploy/containerlab/resources/fabric-router/iad/frr.conf.iad-worker2 @@ -25,6 +25,11 @@ interface eth3 ipv6 address 2001:db8:1:31::1/64 ! +# This site's ingress VIP prefix -- see +# resources/fabric-router/dfw/frr.conf.dfw-worker2 for why each site +# originates its own, from its own edge nodes only. +ipv6 route 2001:db8:6060:3::/64 Null0 + # This node's egress shard (resources/galactic-nat/iad/egressshard.yaml), # originated into the underlay: its SID's covering /64 and its two masquerade # addresses. See resources/fabric-router/dfw/frr.conf.dfw-worker2 for the @@ -86,6 +91,8 @@ router bgp 65000 # /128, not the site /48 -- a per-node peering loopback is the only # thing meant to cross the transit as a host route. network fc00:0:9::1/128 + # This site's ingress VIP prefix -- see this file's own comment above. + network 2001:db8:6060:3::/64 # This node's egress shard's SID /64 and IPv6 masquerade address -- see # this file's own comment above. network 2001:db8:ff03:2002::/64 diff --git a/deploy/containerlab/resources/fabric-router/sjc/frr.conf.sjc-worker2 b/deploy/containerlab/resources/fabric-router/sjc/frr.conf.sjc-worker2 index 028b063d..b48e20c9 100644 --- a/deploy/containerlab/resources/fabric-router/sjc/frr.conf.sjc-worker2 +++ b/deploy/containerlab/resources/fabric-router/sjc/frr.conf.sjc-worker2 @@ -20,6 +20,11 @@ interface bond1 ipv6 address 2001:db8:1:20::1/64 ! +# This site's ingress VIP prefix -- see +# resources/fabric-router/dfw/frr.conf.dfw-worker2 for why each site +# originates its own, from its own edge nodes only. +ipv6 route 2001:db8:6060:2::/64 Null0 + # This node's egress shard (resources/galactic-nat/sjc/egressshard.yaml), # originated into the underlay: its SID's covering /64 and its two masquerade # addresses. See resources/fabric-router/dfw/frr.conf.dfw-worker2 for the @@ -65,6 +70,8 @@ router bgp 65000 # /128, not the site /48 -- a per-node peering loopback is the only # thing meant to cross the transit as a host route. network fc00:0:c::1/128 + # This site's ingress VIP prefix -- see this file's own comment above. + network 2001:db8:6060:2::/64 # This node's egress shard's SID /64 and IPv6 masquerade address -- see # this file's own comment above. network 2001:db8:ff02:2002::/64 diff --git a/deploy/containerlab/resources/galactic-gateway/base/gateway-lab-patch.yaml b/deploy/containerlab/resources/galactic-gateway/base/gateway-lab-patch.yaml new file mode 100644 index 00000000..979f3662 --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/base/gateway-lab-patch.yaml @@ -0,0 +1,14 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: galactic-gateway +spec: + template: + spec: + # Image override only. GALACTIC_GATEWAY_XDP_ATTACH stays at the + # base's default, dispatch: every edge node also runs an egress shard, + # and the two share the uplinks through the node's XDP dispatcher. + containers: + - name: galactic-gateway + image: galactic-gateway:latest + imagePullPolicy: Never diff --git a/deploy/containerlab/resources/galactic-gateway/base/kustomization.yaml b/deploy/containerlab/resources/galactic-gateway/base/kustomization.yaml new file mode 100644 index 00000000..b5a1ca6d --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/base/kustomization.yaml @@ -0,0 +1,11 @@ +# The lab's galactic-gateway base: config/galactic-gateway/base, copied in +# as gateway/ by scripts/deploy-galactic-gateway.sh, plus the lab image. +# Each / overlay pins one instance to one edge node and sets that +# node's interfaces and SRv6 address. +resources: + - gateway +patches: + - path: gateway-lab-patch.yaml + target: + kind: DaemonSet + name: galactic-gateway diff --git a/deploy/containerlab/resources/galactic-gateway/dfw-worker2/kustomization.yaml b/deploy/containerlab/resources/galactic-gateway/dfw-worker2/kustomization.yaml new file mode 100644 index 00000000..3e40768a --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/dfw-worker2/kustomization.yaml @@ -0,0 +1,27 @@ +# dfw-worker2's galactic-gateway: one DaemonSet pinned to this node, and the +# NetworkGateway that tells the gateway controller to serve VIPs here. +namespace: galactic-system +resources: + - ../base + - networkgateway.yaml +patches: + - path: node-patch.yaml + target: + kind: DaemonSet + name: galactic-gateway + # Pins this instance to dfw-worker2, on top of the base's own edge-node and + # opt-in terms. + - target: + kind: DaemonSet + name: galactic-gateway + patch: |- + - op: add + path: /spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/- + value: + key: kubernetes.io/hostname + operator: In + values: + - dfw-worker2 + - op: replace + path: /metadata/name + value: galactic-gateway-dfw-worker2 diff --git a/deploy/containerlab/resources/galactic-gateway/dfw-worker2/networkgateway.yaml b/deploy/containerlab/resources/galactic-gateway/dfw-worker2/networkgateway.yaml new file mode 100644 index 00000000..da4cbcdc --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/dfw-worker2/networkgateway.yaml @@ -0,0 +1,9 @@ +apiVersion: network.datumapis.com/v1alpha1 +kind: NetworkGateway +metadata: + name: dfw-worker2 + namespace: galactic-system +spec: + targetRef: + kind: Node + name: dfw-worker2 diff --git a/deploy/containerlab/resources/galactic-gateway/dfw-worker2/node-patch.yaml b/deploy/containerlab/resources/galactic-gateway/dfw-worker2/node-patch.yaml new file mode 100644 index 00000000..833becf9 --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/dfw-worker2/node-patch.yaml @@ -0,0 +1,24 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: galactic-gateway +spec: + template: + spec: + containers: + - name: galactic-gateway + env: + # Transit-facing uplink, where VIP traffic arrives: an LACP bond, + # so edge_lb runs on each of its members. + - name: GALACTIC_GATEWAY_PUBLIC_INTERFACE + value: bond0 + # Compute-facing bond, where backends' VIP-sourced replies cross + # this node; edge_return runs on its members. + - name: GALACTIC_GATEWAY_INTERNAL_INTERFACES + value: bond1 + # The outer source of every packet edge_lb encapsulates toward a + # backend: this node's own uSID, from its BGPRouter's locator and + # nodeID (resources/galactic-router/dfw/). Infra sets it the + # same way. + - name: GALACTIC_GATEWAY_SRV6_ADDRESS + value: "2001:db8:ff01:1002::" diff --git a/deploy/containerlab/resources/galactic-gateway/dfw-worker3/kustomization.yaml b/deploy/containerlab/resources/galactic-gateway/dfw-worker3/kustomization.yaml new file mode 100644 index 00000000..12e2ef1e --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/dfw-worker3/kustomization.yaml @@ -0,0 +1,27 @@ +# dfw-worker3's galactic-gateway: one DaemonSet pinned to this node, and the +# NetworkGateway that tells the gateway controller to serve VIPs here. +namespace: galactic-system +resources: + - ../base + - networkgateway.yaml +patches: + - path: node-patch.yaml + target: + kind: DaemonSet + name: galactic-gateway + # Pins this instance to dfw-worker3, on top of the base's own edge-node and + # opt-in terms. + - target: + kind: DaemonSet + name: galactic-gateway + patch: |- + - op: add + path: /spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/- + value: + key: kubernetes.io/hostname + operator: In + values: + - dfw-worker3 + - op: replace + path: /metadata/name + value: galactic-gateway-dfw-worker3 diff --git a/deploy/containerlab/resources/galactic-gateway/dfw-worker3/networkgateway.yaml b/deploy/containerlab/resources/galactic-gateway/dfw-worker3/networkgateway.yaml new file mode 100644 index 00000000..b32d3edd --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/dfw-worker3/networkgateway.yaml @@ -0,0 +1,9 @@ +apiVersion: network.datumapis.com/v1alpha1 +kind: NetworkGateway +metadata: + name: dfw-worker3 + namespace: galactic-system +spec: + targetRef: + kind: Node + name: dfw-worker3 diff --git a/deploy/containerlab/resources/galactic-gateway/dfw-worker3/node-patch.yaml b/deploy/containerlab/resources/galactic-gateway/dfw-worker3/node-patch.yaml new file mode 100644 index 00000000..4725dbb8 --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/dfw-worker3/node-patch.yaml @@ -0,0 +1,24 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: galactic-gateway +spec: + template: + spec: + containers: + - name: galactic-gateway + env: + # Transit-facing uplink, where VIP traffic arrives: an LACP bond, + # so edge_lb runs on each of its members. + - name: GALACTIC_GATEWAY_PUBLIC_INTERFACE + value: bond0 + # Compute-facing bond, where backends' VIP-sourced replies cross + # this node; edge_return runs on its members. + - name: GALACTIC_GATEWAY_INTERNAL_INTERFACES + value: bond1 + # The outer source of every packet edge_lb encapsulates toward a + # backend: this node's own uSID, from its BGPRouter's locator and + # nodeID (resources/galactic-router/dfw/). Infra sets it the + # same way. + - name: GALACTIC_GATEWAY_SRV6_ADDRESS + value: "2001:db8:ff01:1003::" diff --git a/deploy/containerlab/resources/galactic-gateway/dfw/kustomization.yaml b/deploy/containerlab/resources/galactic-gateway/dfw/kustomization.yaml new file mode 100644 index 00000000..e5652c8d --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/dfw/kustomization.yaml @@ -0,0 +1,11 @@ +# dfw's ingress tier: a galactic-gateway and NetworkGateway on each of +# its edge nodes, and the NetworkRules and ServiceVIPBindings for ns60, the +# site's backend tenant (resources/tenants/ns60/dfw/). The VIP is +# dfw's own, and so are its backends: a load balancer belongs to one +# region (datum-cloud/enhancements#884). +namespace: galactic-system +resources: + - ../dfw-worker2 + - ../dfw-worker3 + - networkrules.yaml + - servicevipbindings.yaml diff --git a/deploy/containerlab/resources/galactic-gateway/dfw/networkrules.yaml b/deploy/containerlab/resources/galactic-gateway/dfw/networkrules.yaml new file mode 100644 index 00000000..ee82f62a --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/dfw/networkrules.yaml @@ -0,0 +1,33 @@ +# ns60's port 80 over TCP, served by every ns60 backend in dfw. +apiVersion: network.datumapis.com/v1alpha1 +kind: NetworkRule +metadata: + name: ns60-tcp + namespace: galactic-system +spec: + vpcRef: "60" + vpcAttachmentRef: "60" + vipAddresses: + - 2001:db8:6060:1::1 + protocol: tcp + port: 80 + backends: + - address: fd20:60:ff01:a::100 + port: 80 +--- +# ns60's port 80 over UDP, served by every ns60 backend in dfw. +apiVersion: network.datumapis.com/v1alpha1 +kind: NetworkRule +metadata: + name: ns60-udp + namespace: galactic-system +spec: + vpcRef: "60" + vpcAttachmentRef: "60" + vipAddresses: + - 2001:db8:6060:1::1 + protocol: udp + port: 80 + backends: + - address: fd20:60:ff01:a::100 + port: 80 diff --git a/deploy/containerlab/resources/galactic-gateway/dfw/servicevipbindings.yaml b/deploy/containerlab/resources/galactic-gateway/dfw/servicevipbindings.yaml new file mode 100644 index 00000000..bce4ad78 --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/dfw/servicevipbindings.yaml @@ -0,0 +1,37 @@ +# Lets ns60/backend on dfw-worker answer TCP for the VIP: its node +# rewrites the VIP to the backend's address on the way in, and back on the +# way out. +apiVersion: network.datumapis.com/v1alpha1 +kind: ServiceVIPBinding +metadata: + name: ns60-tcp-backend + namespace: galactic-system +spec: + targetRef: + kind: Node + name: dfw-worker + vipAddress: 2001:db8:6060:1::1 + port: 80 + protocol: tcp + backendAddress: fd20:60:ff01:a::100 + backendPort: 80 + egressKind: veth +--- +# Lets ns60/backend on dfw-worker answer UDP for the VIP: its node +# rewrites the VIP to the backend's address on the way in, and back on the +# way out. +apiVersion: network.datumapis.com/v1alpha1 +kind: ServiceVIPBinding +metadata: + name: ns60-udp-backend + namespace: galactic-system +spec: + targetRef: + kind: Node + name: dfw-worker + vipAddress: 2001:db8:6060:1::1 + port: 80 + protocol: udp + backendAddress: fd20:60:ff01:a::100 + backendPort: 80 + egressKind: veth diff --git a/deploy/containerlab/resources/galactic-gateway/iad-worker2/kustomization.yaml b/deploy/containerlab/resources/galactic-gateway/iad-worker2/kustomization.yaml new file mode 100644 index 00000000..dbaabb4f --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/iad-worker2/kustomization.yaml @@ -0,0 +1,27 @@ +# iad-worker2's galactic-gateway: one DaemonSet pinned to this node, and the +# NetworkGateway that tells the gateway controller to serve VIPs here. +namespace: galactic-system +resources: + - ../base + - networkgateway.yaml +patches: + - path: node-patch.yaml + target: + kind: DaemonSet + name: galactic-gateway + # Pins this instance to iad-worker2, on top of the base's own edge-node and + # opt-in terms. + - target: + kind: DaemonSet + name: galactic-gateway + patch: |- + - op: add + path: /spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/- + value: + key: kubernetes.io/hostname + operator: In + values: + - iad-worker2 + - op: replace + path: /metadata/name + value: galactic-gateway-iad-worker2 diff --git a/deploy/containerlab/resources/galactic-gateway/iad-worker2/networkgateway.yaml b/deploy/containerlab/resources/galactic-gateway/iad-worker2/networkgateway.yaml new file mode 100644 index 00000000..6477661e --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/iad-worker2/networkgateway.yaml @@ -0,0 +1,9 @@ +apiVersion: network.datumapis.com/v1alpha1 +kind: NetworkGateway +metadata: + name: iad-worker2 + namespace: galactic-system +spec: + targetRef: + kind: Node + name: iad-worker2 diff --git a/deploy/containerlab/resources/galactic-gateway/iad-worker2/node-patch.yaml b/deploy/containerlab/resources/galactic-gateway/iad-worker2/node-patch.yaml new file mode 100644 index 00000000..bdfddd4b --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/iad-worker2/node-patch.yaml @@ -0,0 +1,24 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: galactic-gateway +spec: + template: + spec: + containers: + - name: galactic-gateway + env: + # Transit-facing uplink, where VIP traffic arrives: an LACP bond, + # so edge_lb runs on each of its members. + - name: GALACTIC_GATEWAY_PUBLIC_INTERFACE + value: bond0 + # Compute-facing bond, where backends' VIP-sourced replies cross + # this node; edge_return runs on its members. + - name: GALACTIC_GATEWAY_INTERNAL_INTERFACES + value: bond1 + # The outer source of every packet edge_lb encapsulates toward a + # backend: this node's own uSID, from its BGPRouter's locator and + # nodeID (resources/galactic-router/iad/). Infra sets it the + # same way. + - name: GALACTIC_GATEWAY_SRV6_ADDRESS + value: "2001:db8:ff03:1002::" diff --git a/deploy/containerlab/resources/galactic-gateway/iad/kustomization.yaml b/deploy/containerlab/resources/galactic-gateway/iad/kustomization.yaml new file mode 100644 index 00000000..2f6de20e --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/iad/kustomization.yaml @@ -0,0 +1,10 @@ +# iad's ingress tier: a galactic-gateway and NetworkGateway on each of +# its edge nodes, and the NetworkRules and ServiceVIPBindings for ns60, the +# site's backend tenant (resources/tenants/ns60/iad/). The VIP is +# iad's own, and so are its backends: a load balancer belongs to one +# region (datum-cloud/enhancements#884). +namespace: galactic-system +resources: + - ../iad-worker2 + - networkrules.yaml + - servicevipbindings.yaml diff --git a/deploy/containerlab/resources/galactic-gateway/iad/networkrules.yaml b/deploy/containerlab/resources/galactic-gateway/iad/networkrules.yaml new file mode 100644 index 00000000..0cacfeca --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/iad/networkrules.yaml @@ -0,0 +1,33 @@ +# ns60's port 80 over TCP, served by every ns60 backend in iad. +apiVersion: network.datumapis.com/v1alpha1 +kind: NetworkRule +metadata: + name: ns60-tcp + namespace: galactic-system +spec: + vpcRef: "60" + vpcAttachmentRef: "60" + vipAddresses: + - 2001:db8:6060:3::1 + protocol: tcp + port: 80 + backends: + - address: fd20:60:ff03:a::100 + port: 80 +--- +# ns60's port 80 over UDP, served by every ns60 backend in iad. +apiVersion: network.datumapis.com/v1alpha1 +kind: NetworkRule +metadata: + name: ns60-udp + namespace: galactic-system +spec: + vpcRef: "60" + vpcAttachmentRef: "60" + vipAddresses: + - 2001:db8:6060:3::1 + protocol: udp + port: 80 + backends: + - address: fd20:60:ff03:a::100 + port: 80 diff --git a/deploy/containerlab/resources/galactic-gateway/iad/servicevipbindings.yaml b/deploy/containerlab/resources/galactic-gateway/iad/servicevipbindings.yaml new file mode 100644 index 00000000..89f3e21a --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/iad/servicevipbindings.yaml @@ -0,0 +1,37 @@ +# Lets ns60/backend on iad-worker answer TCP for the VIP: its node +# rewrites the VIP to the backend's address on the way in, and back on the +# way out. +apiVersion: network.datumapis.com/v1alpha1 +kind: ServiceVIPBinding +metadata: + name: ns60-tcp-backend + namespace: galactic-system +spec: + targetRef: + kind: Node + name: iad-worker + vipAddress: 2001:db8:6060:3::1 + port: 80 + protocol: tcp + backendAddress: fd20:60:ff03:a::100 + backendPort: 80 + egressKind: veth +--- +# Lets ns60/backend on iad-worker answer UDP for the VIP: its node +# rewrites the VIP to the backend's address on the way in, and back on the +# way out. +apiVersion: network.datumapis.com/v1alpha1 +kind: ServiceVIPBinding +metadata: + name: ns60-udp-backend + namespace: galactic-system +spec: + targetRef: + kind: Node + name: iad-worker + vipAddress: 2001:db8:6060:3::1 + port: 80 + protocol: udp + backendAddress: fd20:60:ff03:a::100 + backendPort: 80 + egressKind: veth diff --git a/deploy/containerlab/resources/galactic-gateway/sjc-worker2/kustomization.yaml b/deploy/containerlab/resources/galactic-gateway/sjc-worker2/kustomization.yaml new file mode 100644 index 00000000..5d040c0d --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/sjc-worker2/kustomization.yaml @@ -0,0 +1,27 @@ +# sjc-worker2's galactic-gateway: one DaemonSet pinned to this node, and the +# NetworkGateway that tells the gateway controller to serve VIPs here. +namespace: galactic-system +resources: + - ../base + - networkgateway.yaml +patches: + - path: node-patch.yaml + target: + kind: DaemonSet + name: galactic-gateway + # Pins this instance to sjc-worker2, on top of the base's own edge-node and + # opt-in terms. + - target: + kind: DaemonSet + name: galactic-gateway + patch: |- + - op: add + path: /spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/- + value: + key: kubernetes.io/hostname + operator: In + values: + - sjc-worker2 + - op: replace + path: /metadata/name + value: galactic-gateway-sjc-worker2 diff --git a/deploy/containerlab/resources/galactic-gateway/sjc-worker2/networkgateway.yaml b/deploy/containerlab/resources/galactic-gateway/sjc-worker2/networkgateway.yaml new file mode 100644 index 00000000..acf6b6f4 --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/sjc-worker2/networkgateway.yaml @@ -0,0 +1,9 @@ +apiVersion: network.datumapis.com/v1alpha1 +kind: NetworkGateway +metadata: + name: sjc-worker2 + namespace: galactic-system +spec: + targetRef: + kind: Node + name: sjc-worker2 diff --git a/deploy/containerlab/resources/galactic-gateway/sjc-worker2/node-patch.yaml b/deploy/containerlab/resources/galactic-gateway/sjc-worker2/node-patch.yaml new file mode 100644 index 00000000..48d8b219 --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/sjc-worker2/node-patch.yaml @@ -0,0 +1,24 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: galactic-gateway +spec: + template: + spec: + containers: + - name: galactic-gateway + env: + # Transit-facing uplink, where VIP traffic arrives: an LACP bond, + # so edge_lb runs on each of its members. + - name: GALACTIC_GATEWAY_PUBLIC_INTERFACE + value: bond0 + # Compute-facing bond, where backends' VIP-sourced replies cross + # this node; edge_return runs on its members. + - name: GALACTIC_GATEWAY_INTERNAL_INTERFACES + value: bond1 + # The outer source of every packet edge_lb encapsulates toward a + # backend: this node's own uSID, from its BGPRouter's locator and + # nodeID (resources/galactic-router/sjc/). Infra sets it the + # same way. + - name: GALACTIC_GATEWAY_SRV6_ADDRESS + value: "2001:db8:ff02:1002::" diff --git a/deploy/containerlab/resources/galactic-gateway/sjc/kustomization.yaml b/deploy/containerlab/resources/galactic-gateway/sjc/kustomization.yaml new file mode 100644 index 00000000..cb496a96 --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/sjc/kustomization.yaml @@ -0,0 +1,10 @@ +# sjc's ingress tier: a galactic-gateway and NetworkGateway on each of +# its edge nodes, and the NetworkRules and ServiceVIPBindings for ns60, the +# site's backend tenant (resources/tenants/ns60/sjc/). The VIP is +# sjc's own, and so are its backends: a load balancer belongs to one +# region (datum-cloud/enhancements#884). +namespace: galactic-system +resources: + - ../sjc-worker2 + - networkrules.yaml + - servicevipbindings.yaml diff --git a/deploy/containerlab/resources/galactic-gateway/sjc/networkrules.yaml b/deploy/containerlab/resources/galactic-gateway/sjc/networkrules.yaml new file mode 100644 index 00000000..4244bea4 --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/sjc/networkrules.yaml @@ -0,0 +1,33 @@ +# ns60's port 80 over TCP, served by every ns60 backend in sjc. +apiVersion: network.datumapis.com/v1alpha1 +kind: NetworkRule +metadata: + name: ns60-tcp + namespace: galactic-system +spec: + vpcRef: "60" + vpcAttachmentRef: "60" + vipAddresses: + - 2001:db8:6060:2::1 + protocol: tcp + port: 80 + backends: + - address: fd20:60:ff02:a::100 + port: 80 +--- +# ns60's port 80 over UDP, served by every ns60 backend in sjc. +apiVersion: network.datumapis.com/v1alpha1 +kind: NetworkRule +metadata: + name: ns60-udp + namespace: galactic-system +spec: + vpcRef: "60" + vpcAttachmentRef: "60" + vipAddresses: + - 2001:db8:6060:2::1 + protocol: udp + port: 80 + backends: + - address: fd20:60:ff02:a::100 + port: 80 diff --git a/deploy/containerlab/resources/galactic-gateway/sjc/servicevipbindings.yaml b/deploy/containerlab/resources/galactic-gateway/sjc/servicevipbindings.yaml new file mode 100644 index 00000000..5adc687a --- /dev/null +++ b/deploy/containerlab/resources/galactic-gateway/sjc/servicevipbindings.yaml @@ -0,0 +1,37 @@ +# Lets ns60/backend on sjc-worker answer TCP for the VIP: its node +# rewrites the VIP to the backend's address on the way in, and back on the +# way out. +apiVersion: network.datumapis.com/v1alpha1 +kind: ServiceVIPBinding +metadata: + name: ns60-tcp-backend + namespace: galactic-system +spec: + targetRef: + kind: Node + name: sjc-worker + vipAddress: 2001:db8:6060:2::1 + port: 80 + protocol: tcp + backendAddress: fd20:60:ff02:a::100 + backendPort: 80 + egressKind: veth +--- +# Lets ns60/backend on sjc-worker answer UDP for the VIP: its node +# rewrites the VIP to the backend's address on the way in, and back on the +# way out. +apiVersion: network.datumapis.com/v1alpha1 +kind: ServiceVIPBinding +metadata: + name: ns60-udp-backend + namespace: galactic-system +spec: + targetRef: + kind: Node + name: sjc-worker + vipAddress: 2001:db8:6060:2::1 + port: 80 + protocol: udp + backendAddress: fd20:60:ff02:a::100 + backendPort: 80 + egressKind: veth diff --git a/deploy/containerlab/resources/galactic-nat/README.md b/deploy/containerlab/resources/galactic-nat/README.md index 92c5c62d..9f255515 100644 --- a/deploy/containerlab/resources/galactic-nat/README.md +++ b/deploy/containerlab/resources/galactic-nat/README.md @@ -3,8 +3,9 @@ What's here: a per-site containerlab overlay for `galactic-nat`, the sharded egress translation datapath control plane (`config/galactic-nat/`). Every edge node runs a shard — `dfw-worker2` and `dfw-worker3` in dfw, -`sjc-worker2` in sjc, `iad-worker2` in iad. Every shard runs from the -node's shared XDP dispatcher on its uplinks. Compute nodes run none. +`sjc-worker2` in sjc, `iad-worker2` in iad. Every edge node also runs +`galactic-gateway`, so every shard runs from its own slot of the node's XDP +dispatcher, sharing the uplinks with the gateway. Compute nodes run none. - `base/` — the lab's patch onto `config/galactic-nat/base` (image override for Kind's locally-built images; see `base/kustomization.yaml` @@ -20,7 +21,7 @@ node's shared XDP dispatcher on its uplinks. Compute nodes run none. DaemonSet. Each site's `node-patch.yaml` sets only `GALACTIC_NAT_UPLINK_INTERFACES=bond0,bond1` — an edge node's transit bond (replies arrive there) and its compute-facing bond (tenant egress - arrives there), the two the shard runs from the XDP dispatcher on. iad's + arrives there), the two the shard's XDP program runs on. iad's also sets `GALACTIC_NAT_ECHO_RESPONDER=true`, so `verify:nat-icmp` can ping one shard's masquerade addresses and see the others refuse. The shard's identity lives in that site's `egressshard.yaml`, one `EgressShard` per edge node (dfw's holds two), @@ -40,16 +41,14 @@ Every shard translates for the same `nat64Prefix`, `2001:db8:64::/96`. The Node-ID is service `0x2` over the edge node's own index, beside its router (`0x1NNN`) identity. -## Run from the dispatcher +## Run from the XDP dispatcher -The lab base sets `GALACTIC_NAT_XDP_ATTACH=dispatch`, overriding the -binary's default, `direct`: the shard runs from the node's shared, pinned -XDP dispatcher on every member of `bond0` and `bond1`. No lab node runs -`galactic-gateway`, so nothing else needs the hook, but dispatch is the -mode production edges are moving to. The first deploy onto a lab that ran -`direct` bounces each uplink: the old shard's unpinned program detaches when -its pod exits, and the dispatcher's first attach bounces it again. After -that, a shard restart bounces no uplink. See +The lab base sets `GALACTIC_NAT_XDP_ATTACH=dispatch`: an interface takes +one native XDP program, and every edge node also runs `galactic-gateway` +(`resources/galactic-gateway/`), so the shard and the gateway each run from +their own slot of the node's pinned XDP dispatcher on every member of +`bond0` and `bond1`. Either can then restart without detaching the other, +which `task verify:gateway-restart` checks. See [docs/nat/configuration.md](../../../../docs/nat/configuration.md) for both modes. @@ -69,15 +68,17 @@ yet. The script waits for every shard's `Programmed` condition, not just the rollout: a shard reports ready once attached, before its identity is programmed. -It also migrates a lab brought up while the shards ran chained behind -`galactic-gateway`. It deletes every `galactic-gateway` DaemonSet, the -`NetworkGateway`/`NetworkRule`/`ServiceVIPBinding` objects behind them, and -the `ns60` backend tenant, waiting for the gateway pods to exit before the -shard DaemonSet rolls out: the shard cannot attach while a -gateway still holds an uplink's hook. The gateway's XDP links are not -pinned, so its exit releases them. Re-run `scripts/deploy-fabric.sh` too, -so the edge nodes stop originating the anycast VIP aggregate. A fresh lab -has nothing to delete. +`task deploy:galactic-gateway` runs later, after `deploy:scenarios`: the +gateway's `ServiceVIPBinding`s bind the `ns60` backends that step creates. +Because the shard and the gateway share the uplinks through the dispatcher, +neither has to be deployed first for the other to attach. + +A lab brought up while the shards ran in direct mode moves to dispatch when +this script re-applies the base. That first deploy bounces each uplink: the +old shard's unpinned program detaches when its pod exits, and the +dispatcher's first attach bounces it again. After that, a restart bounces no +uplink. Re-run `scripts/deploy-fabric.sh` as well, so each edge node +originates its site's VIP `/64`. `task build`/`task deploy:images` build and load `galactic-nat:latest` onto the edge nodes the same way the other lab images are; RBAC @@ -137,6 +138,6 @@ Unlike `shardAddressIPv6`, the IPv4 address is not advertised into the fabric by anything in this repo — a NAT64 reply arrives over the IPv4 underlay, which carries no EVPN, so each shard's edge node originates its own `/32` in `resources/fabric-router//frr.conf.`. The reply -reaches the shard because its XDP program sits directly on `bond0`. +reaches the shard because its XDP program runs on `bond0`'s members. `task verify:nat-datapath` drives both families end to end to the off-fabric host. diff --git a/deploy/containerlab/resources/galactic-nat/base/nat-lab-patch.yaml b/deploy/containerlab/resources/galactic-nat/base/nat-lab-patch.yaml index 6984a1c9..cbce63d0 100644 --- a/deploy/containerlab/resources/galactic-nat/base/nat-lab-patch.yaml +++ b/deploy/containerlab/resources/galactic-nat/base/nat-lab-patch.yaml @@ -9,9 +9,11 @@ spec: - name: galactic-nat image: galactic-nat:latest imagePullPolicy: Never - # dispatch, not the base's default direct: the lab runs the shards - # from the node's shared XDP dispatcher, the mode production edges - # move to, so every lab deploy exercises it. env: + # Every edge node also runs galactic-gateway + # (resources/galactic-gateway/), and an interface takes one + # native XDP program, so the shard runs from its own slot of the + # node's XDP dispatcher alongside the gateway's. Either can then + # restart without detaching the other. - name: GALACTIC_NAT_XDP_ATTACH value: dispatch diff --git a/deploy/containerlab/resources/tenants/ns60/base/backend.yaml b/deploy/containerlab/resources/tenants/ns60/base/backend.yaml new file mode 100644 index 00000000..1d64a9d2 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns60/base/backend.yaml @@ -0,0 +1,63 @@ +# ns60's VIP backend, one per site, behind that site's NetworkRules +# (resources/galactic-gateway//). It answers TCP and UDP on port 80 +# with its own pod name and the address it saw the client as, which is how +# scripts/verify-gateway-ingress.sh tells which backend answered and checks +# that the client's address arrived unchanged. +# +# One per site rather than two, though two would give Maglev a choice: each +# site has one compute node, and a node can bind a VIP's port to only one +# backend per VPC (#755: usid.c's vip_xlat_key holds no backend address). +# +# Its address is fixed in the site's NAD rather than taken from the pool, +# because the NetworkRule and ServiceVIPBinding name it. Recreate, not +# RollingUpdate, so a new pod never asks for the address while the old +# one still holds it. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: backend + labels: + app: backend +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: backend + template: + metadata: + labels: + app: backend + annotations: + v1.multus-cni.io/default-network: ns60/backend + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: galactic.datumapis.com/node + operator: In + values: + - compute + containers: + - name: backend + image: ghcr.io/datum-cloud/galactic-debug:latest + imagePullPolicy: IfNotPresent + # The image sets HOSTNAME itself, so the pod's name comes from the + # downward API instead. + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + # socat sets SOCAT_PEERADDR to the client's address for each + # connection or datagram. + command: + - sh + - -c + - | + reply='echo "$POD_NAME $SOCAT_PEERADDR"' + socat TCP6-LISTEN:80,reuseaddr,fork SYSTEM:"$reply" & + exec socat UDP6-RECVFROM:80,reuseaddr,fork SYSTEM:"$reply" diff --git a/deploy/containerlab/resources/tenants/ns60/base/kustomization.yaml b/deploy/containerlab/resources/tenants/ns60/base/kustomization.yaml new file mode 100644 index 00000000..b81bbb04 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns60/base/kustomization.yaml @@ -0,0 +1,4 @@ +namespace: ns60 +resources: + - namespace.yaml + - backend.yaml diff --git a/deploy/containerlab/resources/tenants/ns60/base/namespace.yaml b/deploy/containerlab/resources/tenants/ns60/base/namespace.yaml new file mode 100644 index 00000000..05dcf955 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns60/base/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: ns60 diff --git a/deploy/containerlab/resources/tenants/ns60/dfw/kustomization.yaml b/deploy/containerlab/resources/tenants/ns60/dfw/kustomization.yaml new file mode 100644 index 00000000..ee81d25b --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns60/dfw/kustomization.yaml @@ -0,0 +1,3 @@ +resources: + - ../base + - nad.yaml diff --git a/deploy/containerlab/resources/tenants/ns60/dfw/nad.yaml b/deploy/containerlab/resources/tenants/ns60/dfw/nad.yaml new file mode 100644 index 00000000..6bb80d84 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns60/dfw/nad.yaml @@ -0,0 +1,31 @@ +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + name: backend + namespace: ns60 +spec: + config: |- + { + "cniVersion": "1.0.0", + "name": "backend", + "plugins": [ + { + "type": "galactic-veth", + "vpc": "60", + "vpcattachment": "60", + "namespace": "galactic-system", + "ipam": { + "type": "galactic-ipam", + "addresses": [ + { "address": "fd20:60:ff01:a::100/96", "gateway": "fd20:60:ff01:a::1" } + ] + } + }, + { + "type": "galactic-bgp", + "vpc": "60", + "vpcattachment": "60", + "namespace": "galactic-system" + } + ] + } diff --git a/deploy/containerlab/resources/tenants/ns60/iad/kustomization.yaml b/deploy/containerlab/resources/tenants/ns60/iad/kustomization.yaml new file mode 100644 index 00000000..ee81d25b --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns60/iad/kustomization.yaml @@ -0,0 +1,3 @@ +resources: + - ../base + - nad.yaml diff --git a/deploy/containerlab/resources/tenants/ns60/iad/nad.yaml b/deploy/containerlab/resources/tenants/ns60/iad/nad.yaml new file mode 100644 index 00000000..998df576 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns60/iad/nad.yaml @@ -0,0 +1,31 @@ +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + name: backend + namespace: ns60 +spec: + config: |- + { + "cniVersion": "1.0.0", + "name": "backend", + "plugins": [ + { + "type": "galactic-veth", + "vpc": "60", + "vpcattachment": "60", + "namespace": "galactic-system", + "ipam": { + "type": "galactic-ipam", + "addresses": [ + { "address": "fd20:60:ff03:a::100/96", "gateway": "fd20:60:ff03:a::1" } + ] + } + }, + { + "type": "galactic-bgp", + "vpc": "60", + "vpcattachment": "60", + "namespace": "galactic-system" + } + ] + } diff --git a/deploy/containerlab/resources/tenants/ns60/sjc/kustomization.yaml b/deploy/containerlab/resources/tenants/ns60/sjc/kustomization.yaml new file mode 100644 index 00000000..ee81d25b --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns60/sjc/kustomization.yaml @@ -0,0 +1,3 @@ +resources: + - ../base + - nad.yaml diff --git a/deploy/containerlab/resources/tenants/ns60/sjc/nad.yaml b/deploy/containerlab/resources/tenants/ns60/sjc/nad.yaml new file mode 100644 index 00000000..9cf0fb1d --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns60/sjc/nad.yaml @@ -0,0 +1,31 @@ +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + name: backend + namespace: ns60 +spec: + config: |- + { + "cniVersion": "1.0.0", + "name": "backend", + "plugins": [ + { + "type": "galactic-veth", + "vpc": "60", + "vpcattachment": "60", + "namespace": "galactic-system", + "ipam": { + "type": "galactic-ipam", + "addresses": [ + { "address": "fd20:60:ff02:a::100/96", "gateway": "fd20:60:ff02:a::1" } + ] + } + }, + { + "type": "galactic-bgp", + "vpc": "60", + "vpcattachment": "60", + "namespace": "galactic-system" + } + ] + } diff --git a/deploy/containerlab/scripts/deploy-galactic-gateway.sh b/deploy/containerlab/scripts/deploy-galactic-gateway.sh new file mode 100755 index 00000000..097aaaf0 --- /dev/null +++ b/deploy/containerlab/scripts/deploy-galactic-gateway.sh @@ -0,0 +1,60 @@ +#!/bin/bash +# deploy-galactic-gateway.sh — Install galactic-gateway on every edge node, one +# NetworkGateway per node, and each site's ns60 NetworkRules and +# ServiceVIPBindings (resources/galactic-gateway/). Requires deploy:system +# (the gateway's RBAC/ServiceAccount), deploy:images (galactic-gateway:latest +# on every edge node), deploy:galactic-router (each node's BGPRouter, which +# the gateway advertises its VIPs through), and deploy:ns60 (the backends the +# ServiceVIPBindings bind). +# +# The gateway shares each edge node's uplinks with the egress shard through +# the node's XDP dispatcher, so the two can be deployed in either order. +set -euo pipefail + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +source "${SCRIPT_DIR}/lib.sh" + +GALACTIC_GATEWAY_BASE_DIR=$(cd "${SCRIPT_DIR}/../../../config/galactic-gateway/base" && pwd) + +declare -A GATEWAY_NODES=( + [dfw]="dfw-worker2 dfw-worker3" + [sjc]="sjc-worker2" + [iad]="iad-worker2" +) + +# copy_gateway_config NODE copies config/galactic-gateway/base onto NODE as +# resources/galactic-gateway/base/gateway/, which the lab base names as its +# resource. rm -rf first: docker cp nests SRC inside an existing DEST dir +# rather than replacing it. +copy_gateway_config() { + local node="$1" + docker exec "${node}" rm -rf /galactic/resources/galactic-gateway/base/gateway + docker cp "${GALACTIC_GATEWAY_BASE_DIR}" "${node}:/galactic/resources/galactic-gateway/base/gateway" +} + +# A VIP route verify-gateway-ingress.sh pins on a transit router is removed +# when the check exits, unless it was killed outright. Clear any left over +# from such a run, so the transit forwards by BGP again. +"${SCRIPT_DIR}/verify-gateway-ingress.sh" --unpin + +for site in dfw sjc iad; do + node=$(control_plane "${site}") + echo "Applying galactic-gateway/${site} to ${node}..." + copy_to "${node}" galactic-gateway + copy_gateway_config "${node}" + apply_k "${node}" "/galactic/resources/galactic-gateway/${site}/" + for gw in ${GATEWAY_NODES[${site}]}; do + docker exec "${node}" kubectl -n galactic-system rollout status daemonset "galactic-gateway-${gw}" --timeout=180s + done + # Ready means the engine loaded every rule and advertised it, Accepted that + # the rule found a gateway, and Bound that the backend's node rewrites the + # VIP for it. + docker exec "${node}" kubectl -n galactic-system wait networkgateway --all \ + --for=condition=Ready --timeout=120s + docker exec "${node}" kubectl -n galactic-system wait networkrule --all \ + --for=condition=Accepted --timeout=60s + docker exec "${node}" kubectl -n galactic-system wait servicevipbinding --all \ + --for=condition=Bound --timeout=60s +done + +echo "Done." diff --git a/deploy/containerlab/scripts/deploy-galactic-nat.sh b/deploy/containerlab/scripts/deploy-galactic-nat.sh index cce53f0a..19c46b98 100755 --- a/deploy/containerlab/scripts/deploy-galactic-nat.sh +++ b/deploy/containerlab/scripts/deploy-galactic-nat.sh @@ -5,8 +5,9 @@ # galactic-cni/galactic-router's own), deploy:images (galactic-nat:latest # loaded onto every edge node), and deploy:galactic-router: the shard # advertises its SID through the edge node's BGPRouter, which that step -# creates. Each shard runs from the node's XDP dispatcher on its uplinks -# (GALACTIC_NAT_XDP_ATTACH=dispatch, set in resources/galactic-nat/base). +# creates. Every edge node also runs galactic-gateway +# (deploy-galactic-gateway.sh), so each shard runs from its own slot of the +# node's XDP dispatcher (GALACTIC_NAT_XDP_ATTACH=dispatch). set -euo pipefail SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) @@ -49,27 +50,9 @@ delete_stale_shards() { done } -# delete_gateway NODE removes everything the lab used to deploy for -# galactic-gateway: its per-node DaemonSets, the NetworkGateway/NetworkRule/ -# ServiceVIPBinding objects that drove them, and ns60, their backend tenant. -# The shards used to run chained behind the gateway and now run from the -# node's XDP dispatcher, which cannot attach while an old gateway still holds -# an uplink's hook. The gateway's XDP links are not pinned, so deleting the -# pod releases them. --wait makes sure that has happened before the shard -# DaemonSet below rolls out. A fresh lab has none to delete. -delete_gateway() { - local node="$1" - docker exec "${node}" kubectl -n galactic-system delete daemonset \ - -l app.kubernetes.io/name=galactic-gateway --ignore-not-found --wait - docker exec "${node}" kubectl -n galactic-system delete \ - networkrules,servicevipbindings,networkgateways --all --ignore-not-found --wait - docker exec "${node}" kubectl delete namespace ns60 --ignore-not-found --wait -} - for site in dfw sjc iad; do node=$(control_plane "${site}") echo "Applying galactic-nat/${site} to ${node}..." - delete_gateway "${node}" delete_stale_shards "${node}" copy_to "${node}" galactic-nat copy_nat_config "${node}" diff --git a/deploy/containerlab/scripts/deploy-system.sh b/deploy/containerlab/scripts/deploy-system.sh index cec12c63..282fe3e2 100755 --- a/deploy/containerlab/scripts/deploy-system.sh +++ b/deploy/containerlab/scripts/deploy-system.sh @@ -1,7 +1,8 @@ #!/bin/bash # deploy-system.sh — Install BGP CRDs (datum-cloud/network) and VPC CRDs # (datum-cloud/cloud), then apply the galactic-system namespace and shared -# RBAC (galactic-cni, galactic-router) to every cluster. The namespace and +# RBAC (galactic-cni, galactic-router, galactic-nat, galactic-gateway) to +# every cluster. The namespace and # ServiceAccount/RBAC manifests are applied straight from the repo's # config/ — the same ones used in production — so the lab never forks them. set -euo pipefail @@ -78,6 +79,8 @@ for site in dfw sjc iad; do apply_f "${node}" /galactic/config/galactic-router/rbac.yaml apply_f "${node}" /galactic/config/galactic-nat/serviceaccount.yaml apply_f "${node}" /galactic/config/galactic-nat/rbac.yaml + apply_f "${node}" /galactic/config/galactic-gateway/serviceaccount.yaml + apply_f "${node}" /galactic/config/galactic-gateway/rbac.yaml done echo "Done." diff --git a/deploy/containerlab/scripts/verify-bonds.sh b/deploy/containerlab/scripts/verify-bonds.sh index 600b8f98..84f97f41 100755 --- a/deploy/containerlab/scripts/verify-bonds.sh +++ b/deploy/containerlab/scripts/verify-bonds.sh @@ -8,9 +8,10 @@ # disagree about who they are talking to, and the bond keeps forwarding over # whichever member is left. Every datapath attached to the missing member then # goes unexercised without anything failing. So for every member this checks -# LACP actor state collecting and distributing (the same test -# galactic-gateway's attach gate applies, internal/plumbing/ebpf/edgeattach/ -# gate.go, though the lab no longer deploys the gateway). It also checks that the member count matches the topology and +# LACP actor state collecting and distributing (the same test the XDP attach +# gate galactic-gateway and galactic-nat share applies, +# internal/plumbing/ebpf/xdpattach/gate.go). It also checks that the member +# count matches the topology and # that every member joined the bond's active aggregator with a real partner. # # Keep BONDS in step with gvpc.clab.yaml's mkbond.sh exec lines and diff --git a/deploy/containerlab/scripts/verify-gateway-detach.sh b/deploy/containerlab/scripts/verify-gateway-detach.sh new file mode 100755 index 00000000..999ac8ca --- /dev/null +++ b/deploy/containerlab/scripts/verify-gateway-detach.sh @@ -0,0 +1,54 @@ +#!/bin/bash +# verify-gateway-detach.sh — prove verify-gateway-ingress.sh fails when a +# gateway's XDP program is off its uplinks, and passes again once it is back. +# +# Turning dfw-worker2's datapath off (GALACTIC_GATEWAY_DATAPATH_ENABLED=false) +# clears its slot in the node's XDP dispatcher, so nothing on its uplinks +# claims a VIP packet any more. The NAT shard's slot is untouched. The ingress +# check through that node must then fail; then the datapath is turned back +# on, and the same check must pass. +# +# Not part of `task verify`: dfw-worker2 serves no VIP while it runs. +set -euo pipefail + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +source "${SCRIPT_DIR}/lib.sh" + +CP=$(control_plane dfw) +NODE=dfw-worker2 +DS="galactic-gateway-${NODE}" + +set_datapath() { + docker exec "${CP}" kubectl -n galactic-system set env daemonset "${DS}" \ + "GALACTIC_GATEWAY_DATAPATH_ENABLED=$1" + docker exec "${CP}" kubectl -n galactic-system rollout status daemonset "${DS}" --timeout=120s +} + +restore() { + docker exec "${CP}" kubectl -n galactic-system set env daemonset "${DS}" \ + GALACTIC_GATEWAY_DATAPATH_ENABLED- >/dev/null 2>&1 || true +} +trap restore EXIT + +echo "--- turn ${NODE}'s gateway datapath off ---" +set_datapath false +docker exec "${CP}" kubectl -n galactic-system wait networkgateway "${NODE}" \ + --for=jsonpath='{.status.conditions[?(@.type=="Ready")].reason}'=DatapathDisabled --timeout=60s + +echo "--- the ingress check through ${NODE} must fail ---" +if "${SCRIPT_DIR}/verify-gateway-ingress.sh" --node "${NODE}"; then + echo "FAIL: verify-gateway-ingress.sh passed with ${NODE}'s datapath off" >&2 + exit 1 +fi +echo " ok it failed, as it should" + +echo "--- turn it back on ---" +docker exec "${CP}" kubectl -n galactic-system set env daemonset "${DS}" GALACTIC_GATEWAY_DATAPATH_ENABLED- +docker exec "${CP}" kubectl -n galactic-system rollout status daemonset "${DS}" --timeout=120s +trap - EXIT +docker exec "${CP}" kubectl -n galactic-system wait networkgateway "${NODE}" \ + --for=condition=Ready --timeout=120s + +echo "--- the same check must pass again ---" +"${SCRIPT_DIR}/verify-gateway-ingress.sh" --node "${NODE}" +echo "PASS: the ingress check fails while ${NODE}'s datapath is off and passes once it is back" diff --git a/deploy/containerlab/scripts/verify-gateway-ingress.sh b/deploy/containerlab/scripts/verify-gateway-ingress.sh new file mode 100755 index 00000000..93c78414 --- /dev/null +++ b/deploy/containerlab/scripts/verify-gateway-ingress.sh @@ -0,0 +1,237 @@ +#!/bin/bash +# verify-gateway-ingress.sh — prove each site's VIP serves real TCP and UDP +# through every one of the site's gateway nodes, from outside the fabric. +# +# The client is remote-host on tr4, the one host outside every cluster. Each +# site's transit router splits a VIP's traffic across the site's edge nodes by +# hash, so for each gateway node in turn this pins the VIP to that node with a +# temporary /128 on the transit router (more specific than the /64 the edge +# nodes originate) and removes it on exit. Then, per protocol, FLOWS flows from +# distinct source ports, each of which must: +# +# - draw a reply from an ns60 backend in the VIP's own site (the reply +# carries the backend's pod name), and +# - reach that backend with the client's own address as its source. That is +# DSR's defining property: the gateway encapsulates the client's packet +# and changes nothing in it. +# +# Across the flows through each node: +# +# - the node's own galactic_edge_rule_packets_total for the VIP rises. A +# node whose XDP datapath is gone can still hand a VIP packet to another +# gateway through an EVPN-installed route and get an answer, so a reply +# alone does not prove the pinned node served it. +# - galactic_edge_return_packets_total rises on some gateway in the site: +# the backend's reply crossed edge_return rather than the kernel's +# forwarding path, which survives only while conntrack happens to allow +# it. +# - no gateway in the site counts a datapath drop, and none drops a packet +# as ctstate INVALID in KUBE-FORWARD (#554). +# +# Usage: +# verify-gateway-ingress.sh every gateway node in every site +# verify-gateway-ingress.sh --node NODE one gateway node only +# verify-gateway-ingress.sh --unpin remove any leftover pinned route +set -euo pipefail + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +source "${SCRIPT_DIR}/lib.sh" + +LAB=${LAB:-gvpc} +CLIENT="2001:db8:1:40::2" +PORT=80 +FLOWS=8 +GATEWAY_METRICS_PORT=8081 +SITES=(dfw sjc iad) +declare -A VIP=([dfw]=2001:db8:6060:1::1 [sjc]=2001:db8:6060:2::1 [iad]=2001:db8:6060:3::1) +declare -A GATEWAYS=([dfw]="dfw-worker2 dfw-worker3" [sjc]="sjc-worker2" [iad]="iad-worker2") +# " " +# for each gateway node; keep in step with gvpc.clab.yaml and +# node_files/tr*/frr.conf. +declare -A UPLINK=( + [dfw-worker2]="tr1 2001:db8:1:11::2 bond1" + [dfw-worker3]="tr1 2001:db8:1:12::2 bond2" + [sjc-worker2]="tr2 2001:db8:1:21::2 bond1" + [iad-worker2]="tr3 2001:db8:1:32::2 bond1" +) + +only="" +case "${1:-}" in + --node) only="${2:?--node needs a gateway node name}" ;; + --unpin) ;; + "") ;; + *) + echo "unknown argument: $1" >&2 + exit 2 + ;; +esac + +transit() { docker exec "clab-${LAB}-$1" "${@:2}"; } + +unpin_all() { + local site tr + for site in "${SITES[@]}"; do + for tr in tr1 tr2 tr3; do + transit "${tr}" ip -6 route del "${VIP[${site}]}/128" 2>/dev/null || true + done + done +} + +if [ "${1:-}" = "--unpin" ]; then + unpin_all + exit 0 +fi +trap 'unpin_all; rm -rf "${work:-}"' EXIT + +rc=0 +fail() { + echo " FAIL $*" >&2 + rc=1 +} + +# metric NODE NAME VIP sums NAME's samples labelled with VIP on NODE. +metric() { + docker exec "$1" curl -s --max-time 4 "http://localhost:${GATEWAY_METRICS_PORT}/metrics" | + awk -v name="$2" -v vip="vip=\"$3\"" \ + 'index($0, name"{") == 1 && index($0, vip) {s += $NF} END {print s + 0}' +} + +drops() { + docker exec "$1" curl -s --max-time 4 "http://localhost:${GATEWAY_METRICS_PORT}/metrics" | + awk '/^galactic_edge_drops_total/ {s += $NF} END {print s + 0}' +} + +# invalid NODE is KUBE-FORWARD's ctstate INVALID drop count, matched on the +# rule's text: kube-proxy owns the chain and its order. +invalid() { + docker exec "$1" ip6tables -t filter -L KUBE-FORWARD -n -v -x 2>/dev/null | + awk '/ctstate INVALID/ {print $1; exit}' | grep . || echo 0 +} + +# expand6 ADDR prints ADDR as eight four-digit groups, the form socat reports +# a peer in, so the two can be compared. +expand6() { + local addr="$1" head tail groups=() g missing + if [[ "${addr}" == *::* ]]; then + head=${addr%%::*} + tail=${addr##*::} + local h=() t=() + [ -n "${head}" ] && IFS=: read -ra h <<<"${head}" + [ -n "${tail}" ] && IFS=: read -ra t <<<"${tail}" + missing=$((8 - ${#h[@]} - ${#t[@]})) + groups=("${h[@]}") + for _ in $(seq "${missing}"); do groups+=(0); done + groups+=("${t[@]}") + else + IFS=: read -ra groups <<<"${addr}" + fi + for g in "${groups[@]}"; do printf '%04x:' "0x${g}"; done | sed 's/:$//' + echo +} + +# Prometheus counters are floats, so compare in awk. +rose() { awk -v a="$1" -v b="$2" 'BEGIN {exit !(b > a)}'; } + +# request PROTO VIP SPORT prints the backend's one-line answer, or nothing. +# socat's own errors go to ${work}/socat.err. +request() { + local addr + case "$1" in + tcp) addr="TCP6:[$2]:${PORT},sourceport=$3,reuseaddr" ;; + udp) addr="UDP6:[$2]:${PORT},sourceport=$3,reuseaddr" ;; + esac + echo probe | docker exec -i "clab-${LAB}-remote-host" \ + timeout 5 socat -t2 -T3 - "${addr}" 2>"${work}/socat.err" | head -1 || true +} + +# A random source-port base per run, so the flows are new ones: a port an +# earlier run used can still be in TIME_WAIT on the client. +sport=$((20000 + RANDOM % 30000)) +work=$(mktemp -d) + +checked=0 +for site in "${SITES[@]}"; do + cp=$(control_plane "${site}") + vip=${VIP[${site}]} + backends=$(pod_names "${cp}" ns60 app=backend) + if [ -z "${backends}" ]; then + fail "${site}: no ns60 backend pod (run deploy:ns60)" + continue + fi + + for gw in ${GATEWAYS[${site}]}; do + [ -z "${only}" ] || [ "${only}" = "${gw}" ] || continue + checked=$((checked + 1)) + read -r tr nexthop dev <<<"${UPLINK[${gw}]}" + echo "--- ${site}: ${vip} through ${gw} (pinned on ${tr}) ---" + transit "${tr}" ip -6 route replace "${vip}/128" via "${nexthop}" dev "${dev}" + + declare -A was_ret=() was_drp=() was_inv=() + for n in ${GATEWAYS[${site}]}; do + was_ret[${n}]=$(metric "${n}" galactic_edge_return_packets_total "${vip}") + was_drp[${n}]=$(drops "${n}") + was_inv[${n}]=$(invalid "${n}") + done + was_fwd=$(metric "${gw}" galactic_edge_rule_packets_total "${vip}") + + for proto in tcp udp; do + answered=0 + for _ in $(seq "${FLOWS}"); do + sport=$((sport + 1)) + read -r pod peer <<<"$(request "${proto}" "${vip}" "${sport}")" + peer=${peer#[} + peer=${peer%]} + if [ -z "${pod:-}" ]; then + fail "${proto} from port ${sport}: no reply$(sed -n '1s/^.*socat\[[0-9]*\] [A-Z] / (client: /p' "${work}/socat.err" | sed 's/$/)/')" + elif ! grep -qw -- "${pod}" <<<"${backends}"; then + fail "${proto} from port ${sport}: answered by '${pod}', not one of ${site}'s backends (${backends})" + elif [ "$(expand6 "${peer}")" != "$(expand6 "${CLIENT}")" ]; then + fail "${proto} from port ${sport}: ${pod} saw the client as '${peer}', want ${CLIENT}" + else + answered=$((answered + 1)) + fi + done + echo " ${proto}: ${answered}/${FLOWS} flows answered by ${site}'s backends, client address intact" + done + + now_fwd=$(metric "${gw}" galactic_edge_rule_packets_total "${vip}") + if rose "${was_fwd}" "${now_fwd}"; then + echo " ok ${gw} matched the VIP (rule packets ${was_fwd} -> ${now_fwd})" + else + fail "${gw} never matched the VIP (galactic_edge_rule_packets_total ${was_fwd} -> ${now_fwd}):" \ + "its XDP datapath did not see the traffic" + fi + + returned="" + for n in ${GATEWAYS[${site}]}; do + now=$(metric "${n}" galactic_edge_return_packets_total "${vip}") + rose "${was_ret[${n}]}" "${now}" && returned="${returned} ${n}" + now=$(drops "${n}") + if rose "${was_drp[${n}]}" "${now}"; then + fail "${n} counted datapath drops (galactic_edge_drops_total ${was_drp[${n}]} -> ${now})" + fi + now=$(invalid "${n}") + if [ "${now}" -gt "${was_inv[${n}]}" ]; then + fail "${n} dropped $((now - was_inv[${n}])) packet(s) as ctstate INVALID: a reply took the" \ + "kernel's forwarding path instead of edge_return (#554)" + fi + done + if [ -n "${returned}" ]; then + echo " ok replies forwarded by edge_return on${returned}" + else + fail "no gateway in ${site} counted a reply (galactic_edge_return_packets_total did not move)" + fi + + transit "${tr}" ip -6 route del "${vip}/128" + done +done + +if [ "${checked}" -eq 0 ]; then + echo "FAIL: no gateway node named ${only}" >&2 + exit 2 +fi +if [ "${rc}" -ne 0 ]; then + echo "FAIL: a VIP did not serve TCP and UDP through every gateway node" >&2 + exit 1 +fi +echo "PASS: every VIP served TCP and UDP through each of its site's gateway nodes" diff --git a/deploy/containerlab/scripts/verify-gateway-restart.sh b/deploy/containerlab/scripts/verify-gateway-restart.sh new file mode 100755 index 00000000..cfd3ea13 --- /dev/null +++ b/deploy/containerlab/scripts/verify-gateway-restart.sh @@ -0,0 +1,136 @@ +#!/bin/bash +# verify-gateway-restart.sh — prove the gateway and the egress shard on one +# edge node restart without cutting each other's traffic (#710). +# +# Both run from the node's XDP dispatcher, each in its own slot, so restarting +# one detaches nothing the other uses. On dfw-worker2: +# +# 1. Restart the gateway while dfw's tenant (ns10) sends to the off-fabric +# host through dfw-worker2's shard, its first. Every request must succeed. +# 2. Restart the shard while the off-fabric host sends to dfw's VIP, pinned +# through dfw-worker2's gateway. Every request must succeed. +# +# dfw-worker2's uplink members must also keep their carrier. A restart that +# reattached XDP would reset a real NIC; veths do not show that, so this half +# is weak here and was confirmed on staging hardware instead. +set -euo pipefail + +if [ "$#" -ne 0 ]; then + echo "usage: $(basename "$0") (takes no arguments)" >&2 + exit 2 +fi + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +source "${SCRIPT_DIR}/lib.sh" + +LAB=${LAB:-gvpc} +CP=$(control_plane dfw) +NODE=dfw-worker2 +MEMBERS="eth1 eth2 eth3 eth4" +HOST="2001:db8:1:40::2" +VIP="2001:db8:6060:1::1" +# tr1's link to dfw-worker2: see verify-gateway-ingress.sh's UPLINK. +PIN="via 2001:db8:1:11::2 dev bond1" +# Marks the request loops' command lines, so they can be stopped by name. The +# loops run inside the tenant pod and on remote-host, where killing the local +# docker exec leaves them running, and their traffic would land in the next +# check's capture. +MARKER=gw-restart-probe + +work=$(mktemp -d) +# stop_loops ends both request loops wherever they run. +stop_loops() { + if [ -n "${pod:-}" ]; then + docker exec "${CP}" kubectl -n ns10 exec "${pod}" -- pkill -f "${MARKER}" 2>/dev/null || true + fi + docker exec "clab-${LAB}-remote-host" pkill -f "${MARKER}" 2>/dev/null || true + jobs -p | xargs -r kill 2>/dev/null || true +} + +cleanup() { + stop_loops + docker exec "clab-${LAB}-tr1" ip -6 route del "${VIP}/128" 2>/dev/null || true + rm -rf "${work}" +} +trap cleanup EXIT + +rc=0 + +carrier() { + local m + for m in ${MEMBERS}; do + docker exec "${NODE}" cat "/sys/class/net/${m}/carrier_changes" + done | paste -sd' ' +} + +# watch NAME CMD... runs CMD in the background, recording one "ok" or "fail" +# line per request in ${work}/NAME. +watch() { + local name="$1" + shift + "$@" >"${work}/${name}" 2>/dev/null & +} + +# tally NAME prints " " for the requests recorded so far. +tally() { + awk '/^ok$/ {o++} /^fail$/ {f++} END {print o + 0, f + 0}' "${work}/$1" +} + +# settle NAME stops the background requests and fails the check on any +# failed request, or if too few ran to cover the restart. +settle() { + local name="$1" what="$2" ok failed + sleep 3 + stop_loops + wait 2>/dev/null || true + read -r ok failed <<<"$(tally "${name}")" + if [ "${ok}" -lt 10 ]; then + echo " FAIL ${what}: only ${ok} requests completed, too few to cover the restart" >&2 + rc=1 + elif [ "${failed}" -ne 0 ]; then + echo " FAIL ${what}: ${failed} of $((ok + failed)) requests failed during the restart" >&2 + rc=1 + else + echo " ok ${what}: ${ok} requests, none failed" + fi +} + +before=$(carrier) + +echo "--- restart ${NODE}'s gateway while ns10 egresses through its shard ---" +pod=$(pod_name "${CP}" ns10) +watch egress docker exec "${CP}" kubectl -n ns10 exec "${pod}" -- timeout 120 sh -c \ + ": ${MARKER}; while :; do curl -s -o /dev/null -m2 http://[${HOST}]/ && echo ok || echo fail; sleep 0.2; done" +sleep 3 +docker exec "${CP}" kubectl -n galactic-system rollout restart daemonset "galactic-gateway-${NODE}" +docker exec "${CP}" kubectl -n galactic-system rollout status daemonset "galactic-gateway-${NODE}" --timeout=120s +docker exec "${CP}" kubectl -n galactic-system wait networkgateway "${NODE}" \ + --for=condition=Ready --timeout=60s +settle egress "egress through ${NODE}'s shard" + +echo "--- restart ${NODE}'s shard while the VIP is served through its gateway ---" +docker exec "clab-${LAB}-tr1" ip -6 route replace "${VIP}/128" ${PIN} +watch vip docker exec "clab-${LAB}-remote-host" timeout 120 sh -c \ + ": ${MARKER}; while :; do echo p | socat -t1 -T2 - TCP6:[${VIP}]:80 2>/dev/null | grep -q . && echo ok || echo fail; sleep 0.2; done" +sleep 3 +shard=$(docker exec "${CP}" kubectl -n galactic-system get pods -l app.kubernetes.io/name=galactic-nat \ + --field-selector "spec.nodeName=${NODE}" -o jsonpath='{.items[0].metadata.name}') +docker exec "${CP}" kubectl -n galactic-system delete pod "${shard}" --wait +docker exec "${CP}" kubectl -n galactic-system rollout status daemonset galactic-nat --timeout=120s +docker exec "${CP}" kubectl -n galactic-system wait egressshard "${NODE}-egress" \ + --for=condition=Programmed --timeout=60s +settle vip "VIP through ${NODE}'s gateway" + +after=$(carrier) +if [ "${before}" != "${after}" ]; then + echo " FAIL ${NODE}'s uplink members lost carrier (carrier_changes ${before} -> ${after})" >&2 + rc=1 +else + echo " ok ${NODE}'s uplink members kept carrier (${MEMBERS}: ${after})" +fi + +if [ "${rc}" -ne 0 ]; then + echo "FAIL: restarting one datapath on ${NODE} interrupted the other" >&2 + exit 1 +fi +echo "PASS: the gateway and the shard on ${NODE} each restarted without interrupting the other" diff --git a/docs/agents/ARCHITECTURE-GATEWAY.md b/docs/agents/ARCHITECTURE-GATEWAY.md index 54b0c827..8ae428b4 100644 --- a/docs/agents/ARCHITECTURE-GATEWAY.md +++ b/docs/agents/ARCHITECTURE-GATEWAY.md @@ -179,25 +179,33 @@ public-interface/SRv6-address values. ### Worked ContainerLab example -The containerlab lab no longer deploys `galactic-gateway`; its edge nodes run -only the egress shards, run from each node's XDP dispatcher. Until commit -`abdd665b`, `deploy/containerlab/resources/galactic-gateway/` ran this role on -four edge nodes across all three lab clusters — `dfw-worker2`/`dfw-worker3`, -`sjc-worker2`, and `iad-worker2`. Each node's overlay directory, named for the -node itself, carries: - -- `node-patch.yaml` — pins the DaemonSet to one node via - `kubernetes.io/hostname` and sets `GALACTIC_GATEWAY_PUBLIC_INTERFACE` - (`eth1` in this lab) and `GALACTIC_GATEWAY_SRV6_ADDRESS` (a distinct uFMT - 48+16 uSID per node). -- `bgprouter.yaml`/`bgppeer.yaml` — this node's tenant `BGPRouter`/`BGPPeer` - (the separate `galactic-router` DaemonSet's own config, running - independently on this same node). +`deploy/containerlab/resources/galactic-gateway/` runs this role on all four +lab edge nodes: `dfw-worker2`/`dfw-worker3`, `sjc-worker2`, and +`iad-worker2`. Each node shares its uplinks with that node's egress shard +through the XDP dispatcher (the shard runs with +`GALACTIC_NAT_XDP_ATTACH=dispatch`). Each node's overlay directory, named for +the node itself, carries: + +- `node-patch.yaml` — sets `GALACTIC_GATEWAY_PUBLIC_INTERFACE` (`bond0`, the + transit-facing bond), `GALACTIC_GATEWAY_INTERNAL_INTERFACES` (`bond1`, the + compute-facing bond) and `GALACTIC_GATEWAY_SRV6_ADDRESS` (the node's own + uSID, `:::` from its `BGPRouter`, e.g. + `2001:db8:ff01:1002::` on `dfw-worker2`). The overlay's `kustomization.yaml` + pins the DaemonSet to the node via `kubernetes.io/hostname` and renames it + `galactic-gateway-`. - `networkgateway.yaml` — the `NetworkGateway` object itself (just `spec.targetRef.name`, see the [samples](#the-two-crds) above). -`iad/kustomization.yaml` additionally carries a `networkrule-ns60.yaml` -sample rule. +The node's `BGPRouter`/`BGPPeer` live with the separate `galactic-router` +DaemonSet's config, in `resources/galactic-router//`. + +Each site directory (`dfw/`, `sjc/`, `iad/`) adds the site's +`networkrules.yaml` (`ns60-tcp` and `ns60-udp`, port 80) and +`servicevipbindings.yaml`. Each site has its own VIP, in a `/64` only that +site's edge nodes originate into the underlay (`2001:db8:6060:1::/64` for +dfw, `:2::` for sjc, `:3::` for iad), and one `ns60` backend on its compute +node. `task verify:gateway-ingress` drives TCP and UDP through each gateway +node in turn; see [Testing](#testing). --- @@ -746,7 +754,7 @@ that code. | --------------- | -------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Unit | `task test:unit` | `go test -race` | `internal/config` (`gateway_test.go`), `internal/controller` (`networkgateway_controller_test.go`, `networkrule_controller_test.go`, `usidresolver_test.go`), `internal/gateway` (`engine_test.go`, `diff_test.go`, `kerneldatapath_test.go`, `quota_test.go`, `recovery_test.go`, `telemetry_test.go`), `internal/maglev` (`table_test.go`), `internal/plumbing/bond` (`bond_test.go`, faked `netlink.Link`s, no kernel required — shared bond-master/slave detection used by both this package's `edgeattach.ResolveTargets` and `internal/plumbing/ebpf/attach`'s CNI TC-BPF path), `internal/plumbing/ebpf/edgemap` (`viptable_test.go`, in-memory fake `Table`, no kernel required), `internal/plumbing/ebpf/edgemetrics` (`collector_test.go`), `internal/plumbing/ebpf/edgepreflight` (`preflight_test.go`, `kernel_prober_test.go`); `edgeattach.ResolveTargets` also has its own faked-netlink `TestResolveTargets` (non-bond passthrough, bond-expands-to-slaves-only, no-slaves error) | | Kernel-required | `task test:unit` (root/CAP_BPF gated) | `go test` + `BPF_PROG_TEST_RUN` | `internal/plumbing/ebpf/edgeprog/edgedsr_test.go` — exercises the compiled program directly, including the version-nibble/payload_len regression tests; `internal/plumbing/ebpf/edgeattach/attach_test.go` | -| E2E | — | — | **Not covered.** `deploy/containerlab/` no longer deploys the gateway (until `abdd665b` its four edge nodes were a manifests-and-live-pod canary, not a scripted e2e test) — see Known Constraints. No bonded-uplink scenario exists there either, deliberately: those nodes' uplinks are veth pairs, and (confirmed while adding `TestResolveTargetsAndAttach_RealBondDevice` above) veth slaves accept a native XDP attach on a real bond master directly on at least some kernels, since veth itself supports native XDP and some kernels' bonding driver forwards the attach through to slaves that do — the opposite of the real igb/tg3 failure this whole feature exists for. A containerlab veth-bond scenario would not exercise the bug it would be built to guard against; the root-gated real-bond-device unit test does, without that false sense of coverage. | +| E2E | `task verify:gateway-ingress` (in `deploy/containerlab/`) | bash, live containerlab lab | Real TCP and UDP from the off-fabric host to each site's VIP, pinned through each gateway node in turn by a temporary `/128` on the transit router. Asserts the pinned node's `galactic_edge_rule_packets_total` rises, `edge_return` counts the replies, no gateway counts a drop or a `ctstate INVALID` drop, and the backend sees the client's own address. `verify:gateway-restart` restarts a node's gateway and its egress shard in turn while the other carries traffic (#710). `verify:gateway-detach`, outside `task verify`, turns one node's datapath off and confirms the ingress check fails. The lab's uplinks are LACP bonds of veths, which accept a native XDP attach and so do not reproduce the real igb/tg3 bond failure; the root-gated `TestResolveTargetsAndAttach_RealBondDevice` above covers that. | --- @@ -778,11 +786,11 @@ that tag into `config/galactic-gateway/base`. ## Known Constraints -- **No e2e coverage yet.** `deploy/containerlab/` no longer deploys the gateway at all. Its former canary (removed after `abdd665b`) validated manifests and a live pod/eBPF attach, but predated any live underlay BGP peering that would deliver real traffic through it — see the `gatewayDatapathKeepAlive` incident described in [Entry Points](#cmd-galactic-gatewaygateway-go--datapath-setup) above, which was only discovered because ingress traffic silently stopped being intercepted, not caught by any test. +- **E2E coverage is lab-only.** `task verify:gateway-ingress` in `deploy/containerlab/` sends real traffic through every lab gateway node (see [Testing](#testing)). Before it, the `gatewayDatapathKeepAlive` incident described in [Entry Points](#cmd-galactic-gatewaygateway-go--datapath-setup) above was only discovered because ingress traffic silently stopped being intercepted. Nothing runs that check in CI, since it needs the containerlab lab. - **No `NetworkRule` admission webhook is deployed in this repo.** The CRD's own doc comment and `NetworkRuleReconciler`'s doc comment both describe an admission webhook that verifies VPC/VPCAttachment ownership before setting the `Accepted` condition — no `ValidatingWebhookConfiguration` exists in `config/` yet (`config/webhook/` doesn't exist). `updateAcceptedCondition` currently sets `Accepted=True` unconditionally once gateway nodes exist for the namespace, not gated on any ownership check. - **The uSID backend resolver's tenant-ownership check depends on `BGPVRFInstance` naming staying deterministic.** `verifyTenantOwnership` closes the ambiguous-match gap an earlier version of this resolver had, but it is only as strong as `crdnames.BGPVRFInstanceName(vpc, nodeName)` staying the exact name `galactic-bgp` writes — a divergence between the two would fail closed (a real backend never resolving) rather than open (a wrong-tenant resolve), which is the safer failure direction but still worth knowing about when either side of that naming contract changes. - **`GALACTIC_GATEWAY_SRV6_ADDRESS` has no in-cluster derivation mechanism.** It is operator-supplied per gateway node today; nothing in this repo yet computes it automatically from a node's own `BGPRouter` locator/node-ID. See [SRv6 encap-source address](#srv6-encap-source-address) above. -- **Only the forward path sends Packet Too Big.** When the route to a backend refuses a client's packet for size once the 40-byte outer header is on (`BPF_FIB_LKUP_RET_FRAG_NEEDED`), `edge_lb` counts the drop as `fib_frag_needed` and answers the client with an ICMPv6 Packet Too Big from the VIP, reporting the route's MTU less 40 and quoting the packet's IPv6 header and first 8 transport bytes (#700). A route MTU under 1320 gets no Packet Too Big, since it would report less than the 1280 an IPv6 sender never goes below. The messages draw on a per-CPU token bucket (`icmp_rate_bucket`, 1000 a second with a burst of 100 per CPU, the egress shard's limits), and a refusal is counted as `icmp_rate_limited`. `edge_return` still drops a reply too big for its route as `fib_frag_needed` without telling the backend, and sends no Time Exceeded for an expired hop limit. +- **The uSID TC-BPF/XDP FIB-lookup PMTUD gap applies here too.** When `bpf_fib_lookup()` returns `BPF_FIB_LKUP_RET_FRAG_NEEDED`, `edgedsr.c` counts `DROP_REASON_FIB_FRAG_NEEDED` and drops rather than emitting an ICMPv6 Packet Too Big — the same gap `internal/plumbing/ebpf/prog/usid.c`'s `usid_ingress` has for its FIB lookup (see [ARCHITECTURE-CNI.md#known-constraints](ARCHITECTURE-CNI.md#known-constraints)). `usid_egress` and the egress shard both send Packet Too Big now; closing it here is #700. - **`bpf_fib_lookup()` requires IPv6 forwarding sysctls on the public uplink, not just XDP driver support.** `setupGatewayDatapath` calls `sysctl.ConfigureFIBLookupUplinkSysctls` before attaching the datapath — without `net.ipv6.conf..forwarding` and `net.ipv6.conf.all.forwarding` both set, the kernel returns `BPF_FIB_LKUP_RET_NOT_FWDED` for every lookup regardless of anything this program does, which `edgedsr.c`'s own drop-reason accounting cannot distinguish from a generic FIB lookup failure. Found via live-kernel investigation of a pre-existing containerlab veth/XDP_TX blocker: the sysctl gap, not `XDP_TX` itself, was the actual cause. A related, lab-only characteristic the same investigation turned up: native `XDP_TX` on a veth pair only promotes a frame into the peer's normal receive stack (visible to `tcpdump`) if the peer *also* runs an XDP program — otherwise delivery uses a raw fast-path invisible to normal tools. This does not apply to a real physical NIC uplink in production, where there is no "peer's own XDP program" question to begin with. - **A bonded public uplink attaches per-slave, with per-slave FIB-lookup sysctls to match.** Native-mode XDP against a Linux bonding master is not reliable: confirmed failing outright with "operation not supported" on a real gateway node (802.3ad over an igb/tg3 slave pair). Not every kernel's bonding driver categorically lacks `ndo_bpf` — some do implement it by forwarding the attach to every slave — but that still requires each slave's own driver to support native XDP itself, which not every NIC driver does (tg3 is a commonly cited example that doesn't), so this codebase never relies on attaching to the master working, on any kernel. `edgeattach.ResolveTargets` expands a bonding-master `GALACTIC_GATEWAY_PUBLIC_INTERFACE` to its slave interfaces (never the master itself — see `internal/plumbing/bond`, shared with `internal/plumbing/ebpf/attach`'s TC-BPF path, which attaches to the master *and* its slaves instead), and `setupGatewayDatapath` attaches to and configures FIB-lookup sysctls on every one of them. This is not cosmetic: `edgedsr.c`'s `push_outer_header` calls `bpf_fib_lookup()` with `ctx->ingress_ifindex` — confirmed against the source, not assumed — which for a native XDP program attached to a bond slave is that slave's own ifindex, not the bond master's, so `net.ipv6.conf..forwarding` (not `net.ipv6.conf..forwarding`) is what the kernel actually checks. A bonding master with no resolvable slaves is a hard startup error, not a silent fallback to attaching the master (which would only risk repeating the same failure). `edgeattach.Attach` is all-or-nothing across every resolved slave in one call — if any single slave's driver can't accept a native XDP attach (a real possibility per the tg3 note above, not independently confirmed against that node specifically), the whole datapath startup fails rather than running in a degraded, missing-that-slave's-traffic state. The attach step itself lives in `internal/plumbing/ebpf/xdpattach`, shared with `natattach`: every target is checked for native XDP support through the netdev generic netlink family before any is touched, and each bond slave is waited back into its aggregate (MII up, plus LACP collecting|distributing on an 802.3ad bond) before the next is attached (#583); this has not been exercised against real igb/tg3 hardware, only veth in tests, so whether all of a real bonded pair's slaves actually accept native XDP on the affected class of hardware is still open. Attaching per-slave rather than to the bond as a whole means the slave set has to be kept current after startup: `watchTargets` (`cmd/galactic-gateway/gateway.go`) re-runs `ResolveTargets` on every netlink link or route change, and every 30s with none, and hands the result to an `xdpattach.Set` per program (#647). The set attaches a member enslaved or replaced (new ifindex under the same name) since startup through the same gate, one at a time, and skips one whose bounce would leave its bond with no member carrying traffic; it never detaches a member that still exists, since detaching bounces the link as attaching does. Unlike the startup attach, one member failing there does not undo the rest: it is retried on the next pass, and while any resolved target lacks the program the gateway's `readiness` gRPC health service (`config.GRPCReadinessService`, the readinessProbe's `service`) reports NOT_SERVING, without failing liveness. - **`XDP_REDIRECT` into a veth needs NAPI enabled on the *peer*, which is a lab-only concern.** `edgedsr.c` returns `XDP_REDIRECT` whenever the route to the backend egresses an interface other than the ingress one. In native mode that calls the egress device's `ndo_xdp_xmit`, and veth's implementation silently discards the frame unless the peer end has NAPI enabled — which for a veth means the peer runs its own XDP program or has GRO turned on. A containerlab peer inside an FRR/transit container has neither by default, so a redirect there can fail exactly the way this datapath's original bug did: no drop counter moves, because `bpf_redirect()` itself succeeded and the discard happens later in `xdp_do_redirect`. Both programs are affected, and `edge_return` more so: it attaches to compute-facing links, which are veth pairs in the lab and may be veth or a plain NIC in production. `bpftool prog tracelog` and the `xdp:xdp_redirect_err` tracepoint are the diagnostics; `ethtool -K gro on` is the lab fix. A production uplink (physical NIC, or a bond slave per the bullet above) implements `ndo_xdp_xmit` natively and has no peer to ask about. This is the redirect-side analogue of the `XDP_TX`-on-veth observability quirk noted above. diff --git a/docs/gateway/configuration.md b/docs/gateway/configuration.md index e80f639c..0fd56f44 100644 --- a/docs/gateway/configuration.md +++ b/docs/gateway/configuration.md @@ -10,8 +10,8 @@ that doc wherever the mechanics matter. > Last verified: 2026-09-09 against the current working tree of > `cmd/galactic-gateway/`, `internal/config/gateway.go`, `config/galactic-gateway/`, > `config/galactic-router/`, and `deploy/containerlab/resources/galactic-gateway/`. -> The containerlab lab no longer deploys the gateway; the worked example below -> is that directory as of commit `abdd665b`. +> The worked example below is the lab's own gateway deployment in that +> directory. ## What `galactic-gateway` is, and when you need it @@ -246,43 +246,75 @@ to the gateway datapath — see ## Step 3: Worked example, a per-node overlay -`deploy/containerlab/resources/galactic-gateway/`, as of commit `abdd665b`, -is a real, working instantiation pattern to copy for production — four edge -nodes across three lab clusters, each with its own overlay directory named -for the node. The lab has since stopped deploying the gateway and the -directory is gone from the tree; recover it with -`git archive abdd665b deploy/containerlab/resources/galactic-gateway | tar -x`: +`deploy/containerlab/resources/galactic-gateway/` is a real, working +instantiation pattern to copy for production: four edge nodes across three +lab clusters, each with its own overlay directory named for the node. ``` deploy/containerlab/resources/galactic-gateway/ ├── base/ # kustomize base pointing at config/galactic-gateway/base, │ # plus a lab-only image-tag patch (gateway-lab-patch.yaml) ├── dfw-worker2/ -│ ├── kustomization.yaml -│ ├── node-patch.yaml # pins to one node, sets PUBLIC_INTERFACE/SRV6_ADDRESS -│ ├── bgprouter.yaml # this node's tenant BGPRouter -│ ├── bgppeer.yaml # this node's iBGP session to the route reflector +│ ├── kustomization.yaml # pins to one node, renames the DaemonSet +│ ├── node-patch.yaml # sets PUBLIC_INTERFACE/INTERNAL_INTERFACES/SRV6_ADDRESS │ └── networkgateway.yaml # the NetworkGateway object itself ├── dfw-worker3/ # same shape, this node's own values ├── sjc-worker2/ # " ├── iad-worker2/ # " ├── dfw/ -│ ├── kustomization.yaml # composes this site's edge nodes + its rule -│ ├── networkrule-ns60.yaml # sample NetworkRule -│ └── servicevipbinding-ns60.yaml # sample ServiceVIPBinding for the backend +│ ├── kustomization.yaml # composes this site's edge nodes + its rules +│ ├── networkrules.yaml # ns60-tcp and ns60-udp, port 80 +│ └── servicevipbindings.yaml # one ServiceVIPBinding per rule for the backend ├── sjc/ # same shape, one edge node └── iad/ # " ``` -All three sites bind the same anycast VIP to their own site-local backend, so -the per-site directories differ only in which backend they point at. +Each site has its own VIP, in a `/64` only that site's edge nodes originate +(`2001:db8:6060:1::/64` for dfw, `:2::` for sjc, `:3::` for iad), and one +`ns60` backend of its own, so the per-site directories differ in both. -### `node-patch.yaml` — pin to one node, set the per-node values +### `kustomization.yaml` and `node-patch.yaml` — pin to one node, set the per-node values -`dfw-worker2/node-patch.yaml` adds a `kubernetes.io/hostname` match to the -DaemonSet's node affinity (so this overlay's copy of the DaemonSet only -ever schedules onto exactly one node) and sets the two required -gateway-container env vars: +`dfw-worker2/kustomization.yaml` adds a `kubernetes.io/hostname` match to +the DaemonSet's node affinity, on top of the base's own `node=edge` and +`gateway=enabled` terms, so this overlay's copy of the DaemonSet only ever +schedules onto exactly one node. It also renames the DaemonSet to +`galactic-gateway-dfw-worker2` (a strategic-merge patch can't rename a +resource), so two gateway nodes in the same cluster don't collide under one +name in one namespace: + +```yaml +# dfw-worker2's galactic-gateway: one DaemonSet pinned to this node, and the +# NetworkGateway that tells the gateway controller to serve VIPs here. +namespace: galactic-system +resources: + - ../base + - networkgateway.yaml +patches: + - path: node-patch.yaml + target: + kind: DaemonSet + name: galactic-gateway + # Pins this instance to dfw-worker2, on top of the base's own edge-node and + # opt-in terms. + - target: + kind: DaemonSet + name: galactic-gateway + patch: |- + - op: add + path: /spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/- + value: + key: kubernetes.io/hostname + operator: In + values: + - dfw-worker2 + - op: replace + path: /metadata/name + value: galactic-gateway-dfw-worker2 +``` + +`dfw-worker2/node-patch.yaml` sets the gateway container's per-node env +vars: ```yaml apiVersion: apps/v1 @@ -292,63 +324,56 @@ metadata: spec: template: spec: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: node-role.kubernetes.io/control-plane - operator: DoesNotExist - - key: galactic.datumapis.com/node - operator: In - values: [edge] - - key: kubernetes.io/hostname - operator: In - values: [dfw-worker2] containers: - name: galactic-gateway env: + # Transit-facing uplink, where VIP traffic arrives: an LACP bond, + # so edge_lb runs on each of its members. - name: GALACTIC_GATEWAY_PUBLIC_INTERFACE - value: eth1 + value: bond0 + # Compute-facing bond, where backends' VIP-sourced replies cross + # this node; edge_return runs on its members. + - name: GALACTIC_GATEWAY_INTERNAL_INTERFACES + value: bond1 + # The outer source of every packet edge_lb encapsulates toward a + # backend: this node's own uSID, from its BGPRouter's locator and + # nodeID (resources/galactic-router/dfw/). Infra sets it the + # same way. - name: GALACTIC_GATEWAY_SRV6_ADDRESS - value: "2001:db8:ff01:3002:e000::" + value: "2001:db8:ff01:1002::" ``` -`dfw-worker3`'s own `node-patch.yaml` repeats this shape with its own -hostname and a distinct `GALACTIC_GATEWAY_SRV6_ADDRESS` -(`2001:db8:ff01:3003:e000::`) — every gateway node needs its own unique value. -`dfw-worker2/kustomization.yaml` additionally JSON6902-patches the -DaemonSet's own `metadata.name` to `galactic-gateway-dfw-worker2` (a -strategic-merge patch can't rename a resource), so two gateway nodes in the -same cluster don't collide under one name in one namespace. +`dfw-worker3`'s own overlay repeats this shape with its own hostname and its +own uSID (`2001:db8:ff01:1003::`), since every gateway node needs its own +unique `GALACTIC_GATEWAY_SRV6_ADDRESS`. **To generalize this to a real deployment:** create one overlay directory -per gateway node, each with its own `node-patch.yaml` pinning -`kubernetes.io/hostname` to that node and setting that node's own public -uplink interface name and a unique SRv6-reachable IPv6 address (a uSID -computed the same way `internal/plumbing/ebpf/uformat.Encode` derives one -for this node's `BGPRouter`'s locator/nodeID — see the lab's own -`node-patch.yaml` comment for the exact encoding used there). There is no -generic default for either value and nothing in this repo computes them -for you. +per gateway node, each pinning `kubernetes.io/hostname` to that node and +setting that node's own public uplink interface name and a unique +SRv6-reachable IPv6 address. The lab uses the node's own uSID, the +`:::` address of its `BGPRouter` (`nodeID: 4098` is +`0x1002`). There is no generic default for either value and nothing in this +repo computes them for you. -### `bgprouter.yaml`/`bgppeer.yaml` — this node's tenant BGP +### The node's tenant BGP The co-located `galactic-router` pod needs its own `BGPRouter`/ `BGPPeer` CRDs, exactly like every other `galactic-router` node — a gateway -node is not exempt from the normal tenant-BGP setup: +node is not exempt from the normal tenant-BGP setup. The lab keeps them with +the rest of the router config, in +`deploy/containerlab/resources/galactic-router/dfw/bgprouter-dfw-worker2.yaml` +and `bgppeer-dfw-worker2.yaml`: ```yaml apiVersion: network.datumapis.com/v1alpha1 kind: BGPRouter metadata: - name: dfw-worker2-tenant + name: galactic-router-dfw-worker2 namespace: galactic-system spec: targetRef: kind: Node name: dfw-worker2 - roles: [tenant] localASN: 65000 routerID: "10.0.1.2" srv6Locator: "2001:db8:ff01::/48" @@ -360,11 +385,11 @@ spec: apiVersion: network.datumapis.com/v1alpha1 kind: BGPPeer metadata: - name: dfw-worker2-tenant-to-rr + name: galactic-control-dfw-worker2 namespace: galactic-system spec: routerRef: - name: dfw-worker2-tenant + name: galactic-router-dfw-worker2 peerASN: 65000 address: "fc00:0:8::1" remotePort: 1790 @@ -426,23 +451,23 @@ gateway node, named after that node (see the worked example above). | `spec.backends` | Yes | `[]{address, port}` (1–64) | Backend `address:port` targets traffic is load-balanced to. | | `status.conditions` | — | — | `Accepted` condition — currently set `True` unconditionally once gateway nodes exist for the namespace (see the admission-webhook caveat below). | -Example, from `deploy/containerlab/resources/galactic-gateway/iad/networkrule-ns60.yaml` at `abdd665b`: +Example, the TCP rule from `deploy/containerlab/resources/galactic-gateway/iad/networkrules.yaml`: ```yaml apiVersion: network.datumapis.com/v1alpha1 kind: NetworkRule metadata: - name: ns60-web + name: ns60-tcp namespace: galactic-system spec: vpcRef: "60" vpcAttachmentRef: "60" vipAddresses: - - 2001:db8:6060::1 + - 2001:db8:6060:3::1 protocol: tcp port: 80 backends: - - address: fd20:60:ff03::100:0 + - address: fd20:60:ff03:a::100 port: 80 ``` @@ -474,23 +499,23 @@ naming that `(node, VIP, backend)` triple — this is reconciled by `ServiceVIPBindingReconciler` running inside **`galactic-router`** (not `galactic-gateway`), so it's out of this document's direct scope, but worth knowing about since it's required for the datapath to work -end-to-end. Example, from -`deploy/containerlab/resources/galactic-gateway/iad/servicevipbinding-ns60.yaml` at `abdd665b`: +end-to-end. Example, the TCP binding from +`deploy/containerlab/resources/galactic-gateway/iad/servicevipbindings.yaml`: ```yaml apiVersion: network.datumapis.com/v1alpha1 kind: ServiceVIPBinding metadata: - name: ns60-web-iad-worker + name: ns60-tcp-backend namespace: galactic-system spec: targetRef: kind: Node name: iad-worker - vipAddress: 2001:db8:6060::1 + vipAddress: 2001:db8:6060:3::1 port: 80 protocol: tcp - backendAddress: fd20:60:ff03::100:0 + backendAddress: fd20:60:ff03:a::100 backendPort: 80 egressKind: veth ``` diff --git a/docs/node-labels.md b/docs/node-labels.md index 2e4d8d9a..d1f69e09 100644 --- a/docs/node-labels.md +++ b/docs/node-labels.md @@ -245,17 +245,17 @@ Mutually exclusive with `fabric=router` on the same node, the same way ## Worked example: the containerlab lab -| Node | `node` | `galactic` | `fabric` | Runs | -|------------------------------------------------------------|-----------|------------|----------|------------------------------------------------------------------------------------| -| `dfw-worker`, `sjc-worker`, `iad-worker` | `compute` | `router` | `router` | `galactic-cni`, `galactic-router`, `fabric-router` | -| `dfw-worker2`, `dfw-worker3`, `sjc-worker2`, `iad-worker2` | `edge` | `router` | `router` | `galactic-nat` (`nat=enabled`), `galactic-cni`, `galactic-router`, `fabric-router` | -| `iad-worker3` | — | `control` | `router` | `galactic-router-rr`, `fabric-router` | - -The lab's edge workers carry `node=edge` plus `nat=enabled`, which together -place a `galactic-nat` shard on each; the lab deploys no `galactic-vrf` -yet. No worker carries `gateway=enabled`: the lab deploys no -`galactic-gateway`; each shard still runs from the node's XDP dispatcher -(`GALACTIC_NAT_XDP_ATTACH=dispatch`). +| Node | `node` | `galactic` | `fabric` | Runs | +|------------------------------------------------------------|-----------|------------|----------|----------------------------------------------------------------------------------------------------------------------------| +| `dfw-worker`, `sjc-worker`, `iad-worker` | `compute` | `router` | `router` | `galactic-cni`, `galactic-router`, `fabric-router` | +| `dfw-worker2`, `dfw-worker3`, `sjc-worker2`, `iad-worker2` | `edge` | `router` | `router` | `galactic-gateway` (`gateway=enabled`), `galactic-nat` (`nat=enabled`), `galactic-cni`, `galactic-router`, `fabric-router` | +| `iad-worker3` | — | `control` | `router` | `galactic-router-rr`, `fabric-router` | + +The lab's edge workers carry `node=edge` plus both `gateway=enabled` and +`nat=enabled`, which place a `galactic-gateway` and a `galactic-nat` shard on +each; the lab deploys no `galactic-vrf` yet. The shard runs with +`GALACTIC_NAT_XDP_ATTACH=dispatch`, so it and the gateway share the uplinks +through the node's XDP dispatcher. The reflector row is the one that shows why `galactic` is a mode enum rather than a boolean: `iad-worker3` carries no `node` value at all. Both `node`