Network Load Balancer #698

Summary

Engineering plan for the per-region Layer 4 network load balancer proposed in datum-cloud/enhancements#884. A tenant exposes a set of VPC backends in one region behind a public address: IPv6 by default, with optional reserved static IPv4. TCP and UDP connections are spread across healthy backends without a proxy, keep the client's source address, stay on one backend for the life of the connection, and survive the loss of any single gateway node.

Most of the datapath already exists. galactic-gateway runs a stateless Maglev consistent-hash, Direct Server Return (DSR) XDP datapath, driven by NetworkGateway and NetworkRule. Infra already deploys it onto production edge nodes. This plan covers what stands between that datapath and a load balancer tenants can use: hardening, tenant safety, the missing features, and the public surface.

Architecture reference: docs/agents/ARCHITECTURE-GATEWAY.md.

Scope against enhancements#884

#884 has two phases of its own. Every phase in this plan (0 to 3 below) delivers #884 Phase 1: regional unicast. #884 Phase 2 (one global anycast VIP) is out of scope here, but nothing in this plan may preclude it (see Non-goals).

A load balancer belongs to one region. Its VIPs are announced to the internet only from that region, and every backend must be in that region. Inside the region, every gateway node advertises every VIP at equal preference and ECMP spreads clients across them. That is safe because Maglev is deterministic. It is still regional unicast in #884's terms: one region announces the address, not the global anycast of #884 Phase 2.

Requirements from enhancements#884 and how they map to the design

Non-goals

Carried over from #884:

• L7 inspection: TLS termination, HTTP routing, header insertion. Those belong to the application load balancer.
• Anycast VIPs (fix(vrf): Tear down routes left behind by a restarted sidecar #884 Phase 2). Phase 1 VIPs are regional unicast. To keep Phase 2 open: the public API (Public API for network load balancer #696) records a load balancer's region as a field rather than implying it, VIP pools are per region (Allocate load balancer VIPs from a platform-managed pool #702), and cross-cluster rule fan-out (Keep NetworkRules consistent across clusters so backend replies have a return path #706) is written so it can extend to backends in several regions.
• Non-IP protocols.
• ESP, GRE, ICMP and SCTP support. The API must be able to express them (Phase 2), but the datapath for them comes later.

Added for this plan:

• Egress from VPC workloads. That's galactic-nat (Internet NAT66 Gateway #683, Internet NAT64 gateway #684).
• Full-NAT or a second load-balancing mode. DSR is the only mode.
• Per-tenant bandwidth or packet-rate limits. The gateway's per-node quota keeps its count-based caps.
• Sessions that survive a change to the backend set. Maglev bounds the disruption to about 1/N of flows. That's accepted.
• Backend port remapping. DSR rewrites nothing, so a backend has to listen on the VIP's port (NetworkRule backend port is accepted but ignored #737).

Phases

Phase 0: Harden the existing datapath

Make what's deployed trustworthy before more load goes on it.

• Edge gateway starts without the forwarding sysctls its FIB lookup needs #586 The gateway starts without its forwarding sysctls and reports healthy while dropping everything. Give the pod a writable /proc/sys and fail visibly when forwarding can't be enabled.
• Run galactic-gateway in the containerlab lab with an end-to-end VIP check #699 Bring the gateway back into the containerlab lab, chained with the egress shards. Add a task verify check that sends real client traffic through a VIP to backends in the VIP's own site. Keep a load balancer's VIPs and backends in its own region #736 extends it to prove a VIP isn't reachable through another site. Today there is no e2e coverage at all.
• Send Packet Too Big instead of dropping oversized VIP traffic #700 Return Packet Too Big when the next hop needs fragmentation. Today it's counted and dropped. Reuse the generator from Non-TCP packets over 1460 bytes are silently dropped crossing the fabric #641 / feat(nat): Tell a sender when its reply is too big to re-encapsulate #669.
• Remove the stale galactic-router image stamp for the gateway from the publish workflow #701 Remove the stale galactic-router image stamp into the gateway's base manifests from the publish workflow.
• Gateway advertises rules it failed to load #712 Advertise only the rules the datapath loaded.
• One unresolved backend takes a VIP offline #713 Keep a VIP serving when one of its backends can't be resolved.
• Removing a gateway node can withdraw another's routes #714 Withdraw a departed gateway node's routes without touching other nodes' routes.
• Rule deletion can strand an advertisement #715 Withdraw a deleted rule's route before any node stops serving it.
• Gateway docs describe removed features #720 Update the gateway docs to describe the load balancer as built.

Sharing edge nodes with galactic-nat and galactic-vrf

• Gateway restart detaches XDP from uplinks #710 Keep the XDP program on the uplinks across a gateway restart.
• Decouple gateway and NAT XDP attachment #717 Decouple the gateway's XDP attachment from the egress shard's.
• Egress shard can capture tenant ingress #711 Refuse an egress shard SID that overlaps a router's.
• Run gateway and NAT on edge nodes #718 Let the gateway and egress shards run on the edge nodes alongside galactic-vrf.

Exit: the lab runs the gateway and the egress shards on the same edge nodes, task verify proves VIP traffic end to end within one site, and a gateway restart leaves egress running.

Phase 1: Tenant safety and regional scope

Nothing tenant-facing ships until one tenant can't steer another tenant's traffic, and a load balancer can't reach outside its region.

• Rule authorization is wired but never enforced #366 Replace the allow-all authorizer with one that resolves vpcRef/vpcAttachmentRef against the companion operator. Deploy the ValidatingWebhookConfiguration, Service and certificate. Derive Accepted from the admission result, or rename it.
• Allocate load balancer VIPs from a platform-managed pool #702 VIP allocation, per region. Tenants must not pick their own public addresses.
  • Platform-managed IPv6 and IPv4 pools per region, which Allocate a default IPv6 address and reserved static IPv4 addresses for load balancers #735 allocates from.
  • The allocation is recorded on the resource, and admission rejects a VIP outside the tenant's allocations.
  • VIPs are exported under the same controls as other platform GUA (default-deny, RPKI ROA).
• Allocate a default IPv6 address and reserved static IPv4 addresses for load balancers #735 Default IPv6 and reserved static IPv4. Every new load balancer gets an IPv6 address or subnet from its region's pool automatically. A tenant reserves one or more static IPv4 addresses and binds them to a load balancer. A reservation outlives the load balancer it's bound to, so the address stays stable for firewall allowlists.
• Keep a load balancer's VIPs and backends in its own region #736 Regional scope. Admission refuses a backend outside the load balancer's region, and a VIP is exported to the internet only from its own region.
• Test that NetworkRule backends never resolve into another tenant's VPC #703 Backend scoping. Confirm a backend never resolves into another tenant's VPC, including with overlapping ULA between two tenants.
• NetworkRule backend port is accepted but ignored #737 Refuse a backend port that differs from the rule's port. DSR can't remap it, and today the mismatch is accepted and silently ignored.

Exit: admission refuses a NetworkRule that names another tenant's VPC, a VIP not allocated to the tenant, or a backend outside the region. A VIP can't be reached from the internet through another region.

Phase 2: Feature completeness

• Health-check load balancer backends and drop unhealthy ones from the Maglev table #704 Backend health. Add active L4 health checks (TCP connect or UDP probe), run per gateway node or per region. Drop unhealthy backends from the input to the Maglev table. Every gateway node has to reach the same healthy set or the tables diverge, so the health result is written to a shared object (rule status or a new CR), not decided locally. This is the main design decision in this phase and needs a short design doc first.
• Serve IPv4 VIPs with the client's IPv4 address preserved #705 IPv4 VIPs. fix(vrf): Tear down routes left behind by a restarted sidecar #884 requires reserved static IPv4, so this is in launch scope. Rejecting IPv4 at admission is acceptable only as a stopgap until it ships. Add IPv4 parsing and VIP matching to the load balancer, IPv4 handling to the reply path, and IPv4 support to ServiceVIPBinding. The client's IPv4 source address must reach the backend unchanged (fix(vrf): Tear down routes left behind by a restarted sidecar #884 requires source preservation), so a design that translates the client to IPv6 on the way in doesn't meet the requirement. The issue has to settle what that asks of a backend that has only IPv6.
• Configurable session affinity for load balancer rules #733 Session affinity. Add an affinity setting to NetworkRule: None (5-tuple, the default and today's behaviour), ClientIP (client address only), and ClientIPAndProtocol (client address and protocol). The setting changes only the hash input, so it stays stateless and gives the same answer on every gateway node. Affinity holds only while the backend set is unchanged, the same 1/N bound as above.
• Make NetworkRule's protocol field extensible to protocols without ports #734 Protocol extensibility in the API. Before Public API for network load balancer #696 freezes the public API, change NetworkRule so that ESP, GRE, ICMP and SCTP can be added later without a breaking change. The protocol stays an enum that can grow, and the port becomes optional and required only for protocols that have ports. The datapath keeps rejecting what it doesn't support, and admission refuses a protocol the datapath can't serve yet.
• Keep NetworkRules consistent across clusters so backend replies have a return path #706 Rule consistency across clusters within a region. A backend in another cluster replies through that cluster's edge node, so the NetworkRule has to exist there as well. Test that NetworkRule backends never resolve into another tenant's VPC #703 keeps every backend in the load balancer's region, so this matters only where a region has more than one cluster, and again for fix(vrf): Tear down routes left behind by a restarted sidecar #884 Phase 2. The controller that writes rules (Phase 3) should fan out to every cluster in the region that holds a backend. Galactic should surface a status condition when the reply path is missing.
• Derive the gateway's SRv6 encap source address from its BGPRouter #707 Derive the gateway's SRv6 address from the node's BGPRouter locator and node ID instead of setting it by hand on each node.
• Gateway rebuilds every rule on any route change #719 Stop rebuilding every rule when an unrelated route changes.

Exit: a backend that dies stops receiving new flows without operator action, IPv4 VIPs serve IPv4 clients with their source address preserved, session affinity is configurable, and the API can take new protocols without a breaking change.

Phase 3: Product surface

• Public API for network load balancer #696 Public API: a tenant-facing, per-region load balancer resource in the Datum Cloud API (region, addresses, protocol and port, backend set, session affinity, status), plus a resource for reserving static IPv4 addresses. Its controller allocates the VIPs and writes NetworkRules into the region's clusters. Galactic keeps NetworkRule as the internal contract. The protocol field must follow the extensible shape from Phase 2.
• datumctl integration for network load balancer #694 datumctl support (depends on Public API for network load balancer #696).
• Portal integration for network load balancer #695 Portal support (depends on Public API for network load balancer #696).
• Define tenant-facing status for the network load balancer #708 Status a tenant can act on: provisioned, advertised and reachable, per-backend health from Phase 2.
• Attribute per-VIP traffic counters to the owning VPC #709 Per-VIP packet and byte counters, attributed to the owning VPC, to feed usage metering in Expose network stats from Galactic for usage metering #589.

Exit: a tenant can create a load balancer in a region through the API, CLI or portal, get an IPv6 address automatically, bind a reserved IPv4 address, see that it's healthy, and reach it from the internet over both.

Sequencing

Phase 0 and Phase 1 can run in parallel. Phase 2's health design can start alongside them. #696 can be designed now, but its protocol and affinity fields depend on #733 and #734, and it must not reach tenants before Phase 1 is done. #735 needs #696 for its tenant-facing resources. #694 and #695 follow #696.

Risks

• Maglev tables diverge between nodes. Anything that makes two gateway nodes see a different backend list or affinity setting, such as per-node health results or a partial rule fan-out, breaks the guarantee that a flow sticks to one backend. Every input to the table has to be shared state.
• XDP slot contention. The gateway and galactic-nat share each uplink through a chained XDP program. Any change to what the load balancer leaves unclaimed affects egress. Decouple gateway and NAT XDP attachment #717 removes that dependency.
• IPv4 is in launch scope. fix(vrf): Tear down routes left behind by a restarted sidecar #884 requires reserved static IPv4, and preserving the client's IPv4 source address rules out the cheapest translation-based designs. Serve IPv4 VIPs with the client's IPv4 address preserved #705 is the largest datapath change in this plan.
• Regional leakage. Gateway routes reach every site over the EVPN mesh. If the internet export isn't filtered by region (Keep a load balancer's VIPs and backends in its own region #736), a VIP is announced from regions that don't hold its backends, which breaks fix(vrf): Tear down routes left behind by a restarted sidecar #884's regional unicast model.

Related

• Proposal: Per Region Network Load Balancer for Public Connectivity to gVPC Workloads enhancements#884
• Egress counterparts: Internet NAT66 Gateway #683 (NAT66), Internet NAT64 gateway #684 (NAT64)
• Metering: Expose network stats from Galactic for usage metering #589
• Path MTU work: Non-TCP packets over 1460 bytes are silently dropped crossing the fabric #641, feat(nat): Tell a sender when its reply is too big to re-encapsulate #669