Skip to content

feat(containerlab): Test VIP traffic through every lab gateway node - #761

Merged
privateip merged 2 commits into
mainfrom
feat/issue-699-gateway-lab
Oct 7, 2026
Merged

privateip merged 2 commits into
mainfrom
feat/issue-699-gateway-lab

Conversation

@privateip

@privateip privateip commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The lab stopped running galactic-gateway in #637, so no test sent traffic through a VIP, and the one ingress outage we know of was found because traffic stopped.

Every edge node now runs the gateway beside its egress shard, and each site has its own VIP served by a backend in that site. task verify sends TCP and UDP from outside the fabric through each gateway node in turn, and restarts a node's gateway and shard while the other carries traffic.

One backend per site, because a node can translate only one binding per VPC, protocol and port (#755).

Merges after #760. If #758 or #738 lands first, this rebases onto it.

Test plan

  • task verify passes on a lab running the gateway on all four edge nodes
  • Turning one gateway's datapath off makes the VIP check fail, and turning it back on makes it pass

Stacked on #760.

Fixes #699

🤖 Generated with Claude Code

@privateip privateip self-assigned this Oct 7, 2026
privateip and others added 2 commits October 6, 2026 21:20
The lab stopped deploying galactic-gateway in #637, so nothing sent real traffic through a VIP and the one known ingress outage was found because traffic stopped. Every edge node now runs the gateway beside its egress shard, both in their own slots of the node's XDP dispatcher.

Each site has its own VIP /64, originated only by that site's edge nodes, and TCP and UDP NetworkRules on port 80 that name one ns60 backend in the same site. The backend sits at a fixed address from its NAD and answers with its pod name and the client address it saw. The gateway's SRv6 encap source is the node's own uSID, which is how infra configures it. There is one backend per site because a node translates only one binding per VPC, protocol and port (#755).

New checks:
- verify:gateway: every NetworkGateway is Ready, every NetworkRule Accepted, every ServiceVIPBinding Bound.
- verify:gateway-ingress: TCP and UDP from the off-fabric host to each site's VIP, through each gateway node in turn, pinned with a temporary /128 on the transit router. Each pass needs the pinned node's rule counter to rise, edge_return to count the replies, no drops or ctstate INVALID drops, and the backend to see the client's real address.
- verify:gateway-restart: restarts dfw-worker2's gateway while its shard carries egress, then the shard while the gateway serves the VIP, and fails on any lost request (#710).
- verify:gateway-detach, outside verify: turns one node's datapath off and requires the ingress check to fail, then to pass once it is back.

deploy-galactic-nat.sh no longer deletes the gateway and ns60, and deploy-system.sh applies the gateway's ServiceAccount and RBAC again. remote-host gains socat for the UDP client.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lab docs, the architecture and configuration docs, and the gateway base's comments said the lab no longer deploys galactic-gateway and pointed at commit abdd665 for a worked per-node overlay. They now describe the overlays under deploy/containerlab/resources/galactic-gateway/, the per-site VIPs, the shards running beside the gateway in dispatch mode, and the new verify tasks. The gate path in two lab comments is corrected to internal/plumbing/ebpf/xdpattach/gate.go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@privateip
privateip force-pushed the feat/issue-699-gateway-lab branch from cdf7d60 to 16cd0ed Compare October 7, 2026 01:20
@privateip
privateip marked this pull request as ready for review October 7, 2026 01:27
@privateip
privateip requested a review from a team as a code owner October 7, 2026 01:27
@privateip

Copy link
Copy Markdown
Collaborator Author

First pass on head cdf7d60. Since then the session reworded two commit subjects (now 16cd0ed); the tree is identical, git diff cdf7d602 16cd0ed0 is empty.

pr-adversary: "VERDICT: hold. The lab scenarios behind the PR's claims were never run, and CI is pending." Its one decision finding was that it could not run the lab and that kustomize build fails on the per-node overlays without the gateway base. The build failure is by design, the same copy-at-deploy pattern the lab's NAT, router and fabric overlays use. The lab was run by hand instead (evidence below), and that settled the finding. It checked SRv6 addresses against BGPRouter nodeIDs, the DatapathDisabled reason, ns60 backend addresses against the NADs, and DaemonSet names against rollout waits, and found them consistent.

pr-conventions-reviewer: "VERDICT: hold. (the subject length, plus noting merge order)". Both subject findings are fixed by the reword. Merge order is now in the body: #760 first, then retarget to main; if #758 or #738 lands first, this rebases onto it and the two gateway docs will conflict. A render of config/ is unchanged, and kustomize build with the base copied in renders cleanly for dfw, iad, sjc and ns60.

RAN: neither reviewer's RAN list was supplied to the fixer beyond the checks named above (kustomize builds with the base copied in, cross-checks of addresses, names and NADs); no lab run by the reviewers.

Applied findings: the two commit subject rewords, done by the session before this run, and the PR body (merge-order line added, both test-plan boxes ticked). No new commits from this run.

Lab evidence, from a full task clean and task deploy on this branch with no manual steps. The lab converged about 75 s after deploy, and verify ran after a 3-minute settle.

  • verify:gateway and verify:gateway-ingress passed on every run. TCP and UDP from the off-fabric host reach each site's VIP through all four gateway nodes, each pinned node's rule counter rises, edge_return forwards replies, there are no drops or ctstate INVALID drops, and the backend sees the client's own address.
  • verify:gateway-detach: with one node's datapath off the ingress check fails (no replies, that node's counter at 0); with it back on, it passes.
  • verify:gateway-restart passed 6 of 7 runs. One run lost 1 of 45 egress requests during a gateway restart. 30 further instrumented restarts (4,148 requests at 100 ms) lost none, with no shard drop-counter movement and a 1.9 ms worst case, so that loss is not attributed to the restart. The check stays strict.
  • All NAT checks, MSS clamp, PMTU, NAT64 non-global and the ns10 to ns40 scenarios passed with the gateway running beside the shards. verify:nat-icmp failed once at its NAT64 Packet Too Big step right after the fresh deploy, then passed 3 of 3 runs.

CI: all checks green on 16cd0ed (Build, Lint, Unit Tests, Unit Tests (root), E2E Tests, image and bundle publishes, CLA).

Merge: this PR is stacked on #760, so auto-merge is not enabled; it would land this into #760's branch after #760 was approved without it. Auto-merge will be enabled once #760 merges and this PR is retargeted to main, with the merge method (merge) the main ruleset allows. That ruleset requires one approving review, code-owner review and approval of the last push, and dismisses stale reviews on push, so a human approval is needed before it merges.

Base automatically changed from fix/lab-bmp-logs-tenant-redeploy to main October 7, 2026 01:29
@privateip
privateip enabled auto-merge October 7, 2026 01:44
@privateip
privateip merged commit f33aeab into main Oct 7, 2026
14 checks passed
@privateip
privateip deleted the feat/issue-699-gateway-lab branch October 7, 2026 01:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run galactic-gateway in the containerlab lab with an end-to-end VIP check

2 participants