From 17df64300f882ac123270a7a688194b29d3bcea2 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sun, 11 Oct 2026 17:24:33 -0500 Subject: [PATCH 01/25] docs: propose network load balancer API --- docs/enhancements/network-load-balancers.md | 153 ++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 docs/enhancements/network-load-balancers.md diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md new file mode 100644 index 00000000..302fdb91 --- /dev/null +++ b/docs/enhancements/network-load-balancers.md @@ -0,0 +1,153 @@ +# Network Load Balancers + +| | | +|---|---| +| **Status** | Proposed | +| **Author** | Engineering | +| **Created** | 2026-10-11 | + +## Summary + +A project can expose a `NetworkService` on public TCP ports through a +`NetworkLoadBalancer`. The load balancer owns listeners and public addresses; a +separate `NetworkLoadBalancerRoute` connects a listener to a service. Each address +request chooses whether the platform allocates one address per location or one +address shared across locations. + +The API is specific to Datum. It borrows the useful separation of listeners and +routes from Gateway API without making consumers configure Gateway API objects. + +## What a consumer writes + +This example publishes the same TCP service on a distinct IPv4 address in every +eligible location and on one shared IPv4 address announced from all of them: + +```yaml +apiVersion: networking.datumapis.com/v1alpha +kind: NetworkService +metadata: + name: game +spec: + networkInterfaces: + selector: + matchLabels: + compute.datumapis.com/workload-name: game + ports: + - name: game + port: 27015 + protocol: TCP +--- +apiVersion: networking.datumapis.com/v1alpha +kind: NetworkLoadBalancer +metadata: + name: game +spec: + networkRef: + name: default + exposure: + addresses: + - name: regional + mode: Unicast + family: IPv4 + locations: + from: All + - name: global + mode: Anycast + family: IPv4 + locations: + from: All + listeners: + - name: game + protocol: TCP + port: 27015 + addressRefs: [regional, global] +--- +apiVersion: networking.datumapis.com/v1alpha +kind: NetworkLoadBalancerRoute +metadata: + name: game +spec: + parentRef: + name: game + listener: game + backend: + networkService: + name: game + port: game +``` + +`Unicast` allocates a different address for each selected location. `Anycast` +allocates one address and announces it from each selected location. These modes +describe the address *across locations*: within a location, more than one edge +node may announce the same address. `locations.from: All` follows every current +and future location eligible to serve this load balancer. A `Names` selector +with explicit location names limits the request to those locations. Adding an +eligible location to `All` allocates another unicast address, while the anycast +address stays the same. + +A listener can name several address requests when they serve the same route. +Consumers use separate listeners when addresses need different ports or routes. +The route is a separate object so a service owner can change backends without +changing address ownership or listener configuration. A route attaches to one +named listener and initially has one `NetworkService` backend. Only one route +may claim a listener in the first release; a conflicting route reports a +condition rather than taking traffic unpredictably. + +## What a consumer reads + +The load balancer reports each allocated IP as one status entry, regardless of +address mode. For two locations, the example above could report: + +```yaml +status: + addresses: + - requestName: regional + family: IPv4 + value: 192.0.2.10 + announcements: + - location: us-central-1 + state: Announced + - requestName: regional + family: IPv4 + value: 192.0.2.11 + announcements: + - location: eu-west-1 + state: Announced + - requestName: global + family: IPv4 + value: 198.51.100.20 + announcements: + - location: us-central-1 + state: Announced + - location: eu-west-1 + state: Announced +``` + +These are illustrative addresses. An allocation remains visible when its route +or backends are unavailable; its announcement state changes instead. Conditions +on the load balancer distinguish address allocation and programming. Route +conditions report attachment, backend resolution, and programming. `Ready` +means at least one selected location is serving; per-location status shows which +ones are not. A missing backend does not silently release a consumer's IP. + +## Delivery scope + +The first release forwards TCP connections to healthy members of a +`NetworkService` in the ingress location. It supports one catch-all route per +listener, per-location unicast, multi-location anycast, and their combination. +The address allocation and location selection shape is intended for later use +by Application Load Balancers too. `HTTPProxy` keeps its current API until that +capability is implemented. + +NSO must resolve the service's members, allocate VIPs, select eligible +locations, and report status. Cell controllers translate the resolved intent +into Galactic `NetworkRule` and `ServiceVIPBinding` objects. Galactic already +supports VIP-to-backend TCP/UDP forwarding and announcements, but the controllers +do not yet create these objects from project intent. Announcing a location must +be gated on a programmed rule and serving backends; losing the last usable +backend withdraws that location without releasing its address. Anycast needs +one globally unique allocation propagated to every selected location. + +UDP, explicit health probes, multiple weighted backends, source-based matches, +cross-location backend fallback, and advanced routing policies require separate +data plane and API work. They are not implied by the first route shape. From c535635ec3dfcb387defa8399a74993289fcc8b7 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sun, 11 Oct 2026 17:27:00 -0500 Subject: [PATCH 02/25] docs: align NLB proposal with enhancement template --- docs/enhancements/network-load-balancers.md | 226 ++++++++++++++------ 1 file changed, 162 insertions(+), 64 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index 302fdb91..f25e1914 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -1,26 +1,95 @@ +--- +status: provisional +stage: alpha +latest-milestone: "v0.x" +--- + # Network Load Balancers -| | | -|---|---| -| **Status** | Proposed | -| **Author** | Engineering | -| **Created** | 2026-10-11 | +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories](#user-stories) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed](#infrastructure-needed) ## Summary -A project can expose a `NetworkService` on public TCP ports through a -`NetworkLoadBalancer`. The load balancer owns listeners and public addresses; a -separate `NetworkLoadBalancerRoute` connects a listener to a service. Each address -request chooses whether the platform allocates one address per location or one -address shared across locations. +A project can publish a `NetworkService` on a public TCP port through a +`NetworkLoadBalancer`. The load balancer owns listeners and public address +requests; a separate `NetworkLoadBalancerRoute` connects a listener to the +service. Consumers choose distinct IPs in each location, one IP shared across +locations, or both on the same listener. Status shows the allocated IPs and +where each is announced. + +## Motivation + +`NetworkService` identifies workloads and their ports, but does not give them a +public Layer 4 entry point. Consumers need a stable address and port without +managing Galactic resources. They also need to choose whether clients use a +location-specific address or a shared address reachable from several locations. + +### Goals + +- Publish a TCP `NetworkService` through a named listener and separate route. +- Support per-location unicast and multi-location anycast address requests, + including a mix of both on one load balancer. +- Show allocated addresses and serving locations without releasing an address + when its backends are temporarily unavailable. +- Use an address request shape that can later serve Application Load Balancers. + +### Non-Goals + +- HTTP routing, TLS termination, or changing the current `HTTPProxy` API. +- UDP, active health probes, weighted backends, source-based matches, or + cross-location backend fallback in the first release. +- Requiring consumers to create Gateway API or Galactic resources. + +## Proposal -The API is specific to Datum. It borrows the useful separation of listeners and -routes from Gateway API without making consumers configure Gateway API objects. +A `NetworkLoadBalancer` names a network, requests addresses, and defines TCP +listeners. Each listener names the address requests on which it serves. A +`NetworkLoadBalancerRoute` attaches to a named listener and points to a named +port on a `NetworkService`. The first release accepts one catch-all route per +listener; a conflicting route reports a condition. -## What a consumer writes +`Unicast` allocates one IP per selected location. `Anycast` allocates one IP +announced from all selected locations. These names describe addressing *across +locations*: multiple edge nodes may announce an IP within one location. +`locations.from: All` follows current and future eligible locations. An +explicit `Names` selection limits the request to listed locations. Adding a +location to `All` creates another unicast IP while the anycast IP stays stable. -This example publishes the same TCP service on a distinct IPv4 address in every -eligible location and on one shared IPv4 address announced from all of them: +### User Stories + +- A game operator publishes the same TCP port on a different IP in each region + so clients can choose a region. +- The operator also publishes one global IP on that listener so clients can use + a single endpoint; the status shows which regions currently announce it. +- If a region loses its serving backends, its IP remains allocated while its + announcement is withdrawn. + +### Risks and Mitigations + +- **An announced IP without serving backends drops traffic.** Announce a + location only after its rule is programmed with usable backends; withdraw it + when the last usable backend disappears. +- **Reusing an anycast IP for another load balancer misroutes traffic.** Hold + one globally unique allocation and propagate that allocation to every + selected location. +- **An `All` request can expand over time.** Expose each added location and IP + in status, and bound the number of locations and addresses per load balancer. + +## Design Details + +The consumer-facing shape is: ```yaml apiVersion: networking.datumapis.com/v1alpha @@ -76,27 +145,8 @@ spec: port: game ``` -`Unicast` allocates a different address for each selected location. `Anycast` -allocates one address and announces it from each selected location. These modes -describe the address *across locations*: within a location, more than one edge -node may announce the same address. `locations.from: All` follows every current -and future location eligible to serve this load balancer. A `Names` selector -with explicit location names limits the request to those locations. Adding an -eligible location to `All` allocates another unicast address, while the anycast -address stays the same. - -A listener can name several address requests when they serve the same route. -Consumers use separate listeners when addresses need different ports or routes. -The route is a separate object so a service owner can change backends without -changing address ownership or listener configuration. A route attaches to one -named listener and initially has one `NetworkService` backend. Only one route -may claim a listener in the first release; a conflicting route reports a -condition rather than taking traffic unpredictably. - -## What a consumer reads - -The load balancer reports each allocated IP as one status entry, regardless of -address mode. For two locations, the example above could report: +The load balancer reports one entry per allocated IP. Unicast produces an entry +per location; anycast produces one entry with several announcements: ```yaml status: @@ -123,31 +173,79 @@ status: state: Announced ``` -These are illustrative addresses. An allocation remains visible when its route -or backends are unavailable; its announcement state changes instead. Conditions -on the load balancer distinguish address allocation and programming. Route -conditions report attachment, backend resolution, and programming. `Ready` -means at least one selected location is serving; per-location status shows which -ones are not. A missing backend does not silently release a consumer's IP. - -## Delivery scope - -The first release forwards TCP connections to healthy members of a -`NetworkService` in the ingress location. It supports one catch-all route per -listener, per-location unicast, multi-location anycast, and their combination. -The address allocation and location selection shape is intended for later use -by Application Load Balancers too. `HTTPProxy` keeps its current API until that -capability is implemented. - -NSO must resolve the service's members, allocate VIPs, select eligible -locations, and report status. Cell controllers translate the resolved intent -into Galactic `NetworkRule` and `ServiceVIPBinding` objects. Galactic already -supports VIP-to-backend TCP/UDP forwarding and announcements, but the controllers -do not yet create these objects from project intent. Announcing a location must -be gated on a programmed rule and serving backends; losing the last usable -backend withdraws that location without releasing its address. Anycast needs -one globally unique allocation propagated to every selected location. - -UDP, explicit health probes, multiple weighted backends, source-based matches, -cross-location backend fallback, and advanced routing policies require separate -data plane and API work. They are not implied by the first route shape. +These IPs are illustrative. An allocation remains in status when a route or +backend is unavailable; its announcement state changes instead. Load balancer +conditions distinguish allocation and programming. Route conditions report +attachment, backend resolution, and programming. `Ready` means at least one +selected location serves traffic; location status identifies any that do not. + +NSO resolves service membership, allocates IPs, selects eligible locations, +and reports status. Cell controllers turn resolved intent into Galactic +`NetworkRule` and `ServiceVIPBinding` objects. The initial route uses healthy +backends in the ingress location. The same anycast IP is installed in every +selected cell; the control plane must not allocate one IP per cell for it. + +## Production Readiness Review Questionnaire + +### Feature Enablement and Rollback + +Enable the feature only in locations with a capable Galactic gateway and IP +pool. Disabling reconciliation alone must not strand announced IPs: rollback +needs an ordered withdrawal and release path that preserves allocations until +advertisements are gone. + +### Rollout, Upgrade and Rollback Planning + +Start with single-location TCP, then test multi-location unicast and anycast. +Exercise route updates, loss of the last backend, location removal, and +withdrawal before IP release. Existing `NetworkService` and `HTTPProxy` +behavior should be unaffected. + +### Monitoring Requirements + +Operators need counts of allocated VIPs, programmed rules, and announced +locations, plus failures to allocate, program, or withdraw. Consumers read +load balancer and route conditions and the per-address announcement status. + +### Dependencies + +IPAM must provide globally unique anycast VIPs and location-specific unicast +VIPs. Selected cells need Galactic gateways, backend VIP bindings, and a way +to report programming and announcement state. + +### Scalability + +Bound listeners, routes, address requests, and selected locations. An `All` +request adds an allocation and cell configuration when a new location becomes +eligible; reconciliation must handle those additions without changing existing +IPs. + +### Troubleshooting + +Report failures at the affected route, address request, and location. In +particular, distinguish an address that is allocated but not announced from an +address that could not be allocated. + +## Implementation History + +- 2026-10-11: Initial proposal. + +## Drawbacks + +Separate routes add an object to a simple single-backend setup. Unicast across +many locations consumes an IP in each one, and anycast requires coordinated +withdrawal across cells. + +## Alternatives + +- **Put exposure and backends on `NetworkService`.** This ties workload + membership to public address and listener lifecycle. +- **Put the backend on each listener.** This is compact for one backend but + leaves no independent route lifecycle or place for future route policy. +- **Expose Gateway API directly.** It adds concepts consumers do not need for + this use case and does not express Datum's location and address policies. + +## Infrastructure Needed + +Public VIP pools, cell-level programming and status propagation, and a +multi-location traffic test environment are required before rollout. From cda5e55d4b9c1cab804e8806d31cc01c346372df Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sun, 11 Oct 2026 17:30:39 -0500 Subject: [PATCH 03/25] docs: outline future NLB route controls --- docs/enhancements/network-load-balancers.md | 44 +++++++++++++++++++-- 1 file changed, 40 insertions(+), 4 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index f25e1914..98af6e5c 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -14,6 +14,7 @@ latest-milestone: "v0.x" - [User Stories](#user-stories) - [Risks and Mitigations](#risks-and-mitigations) - [Design Details](#design-details) + - [Route Controls](#route-controls) - [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) - [Implementation History](#implementation-history) - [Drawbacks](#drawbacks) @@ -44,6 +45,8 @@ location-specific address or a shared address reachable from several locations. - Show allocated addresses and serving locations without releasing an address when its backends are temporarily unavailable. - Use an address request shape that can later serve Application Load Balancers. +- Leave room for common L4 traffic controls without requiring them in the first + release. ### Non-Goals @@ -86,6 +89,9 @@ location to `All` creates another unicast IP while the anycast IP stays stable. selected location. - **An `All` request can expand over time.** Expose each added location and IP in status, and bound the number of locations and addresses per load balancer. +- **Overlapping routes could select different backends for the same flow.** When + match-based routes are added, require one default and deterministic precedence + or reject ambiguous overlaps. ## Design Details @@ -139,12 +145,15 @@ spec: parentRef: name: game listener: game - backend: - networkService: - name: game - port: game + backends: + - networkService: + name: game + port: game ``` +`backends` is a list so a later weighted split or ordered failover does not +require replacing the field. The first release requires exactly one entry. + The load balancer reports one entry per allocated IP. Unicast produces an entry per location; anycast produces one entry with several announcements: @@ -185,6 +194,33 @@ and reports status. Cell controllers turn resolved intent into Galactic backends in the ingress location. The same anycast IP is installed in every selected cell; the control plane must not allocate one IP per cell for it. +### Route Controls + +The listener selects destination IP, protocol, and port. The route selects a +backend for each new connection. The following controls are candidates for +later releases, after the single-backend catch-all route works: + +- **Weighted backends:** Split new connections among `NetworkService` backends + by weight. Existing connections do not move when weights change. +- **Priority failover:** Use a standby backend when the preferred one lacks + enough healthy members, with an explicit threshold. +- **Backend locality:** Select `LocalOnly`, `PreferLocal`, or `Global` backends + relative to the ingress location. This does not change which locations + announce the VIP. Cross-location forwarding needs a verified DSR return path. +- **Source CIDR match:** Steer client networks to different backends. Multiple + routes on one listener need a default and unambiguous precedence. This is + traffic selection, not an access-control substitute. +- **Client affinity:** Keep new connections from one client on a consistent + backend when needed. Clients sharing a NAT IP can create uneven load. + +SNI-based TLS passthrough is a separate TLS route capability, not a match on an +ordinary TCP route. Backend health checks belong with `NetworkService` or a +backend policy. Connection draining, source-IP preservation, and PROXY protocol +are delivery settings rather than route matches; they need their own data plane +contracts. Galactic's current rule has a backend list but no weights, route +matches, or priority policy, so these controls need implementation before the +API accepts them. + ## Production Readiness Review Questionnaire ### Feature Enablement and Rollback From 3546d1c3e2012fbc54d40209a460dca824b97da3 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sun, 11 Oct 2026 17:34:13 -0500 Subject: [PATCH 04/25] docs: include UDP in NLB proposal --- docs/enhancements/network-load-balancers.md | 70 ++++++++++++--------- 1 file changed, 42 insertions(+), 28 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index 98af6e5c..b5ebc4df 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -23,7 +23,7 @@ latest-milestone: "v0.x" ## Summary -A project can publish a `NetworkService` on a public TCP port through a +A project can publish a `NetworkService` on a public TCP or UDP port through a `NetworkLoadBalancer`. The load balancer owns listeners and public address requests; a separate `NetworkLoadBalancerRoute` connects a listener to the service. Consumers choose distinct IPs in each location, one IP shared across @@ -39,7 +39,8 @@ location-specific address or a shared address reachable from several locations. ### Goals -- Publish a TCP `NetworkService` through a named listener and separate route. +- Publish a TCP or UDP `NetworkService` through a named listener and separate + route. - Support per-location unicast and multi-location anycast address requests, including a mix of both on one load balancer. - Show allocated addresses and serving locations without releasing an address @@ -51,14 +52,14 @@ location-specific address or a shared address reachable from several locations. ### Non-Goals - HTTP routing, TLS termination, or changing the current `HTTPProxy` API. -- UDP, active health probes, weighted backends, source-based matches, or +- Active health probes, weighted backends, source-based matches, or cross-location backend fallback in the first release. - Requiring consumers to create Gateway API or Galactic resources. ## Proposal A `NetworkLoadBalancer` names a network, requests addresses, and defines TCP -listeners. Each listener names the address requests on which it serves. A +or UDP listeners. Each listener names the address requests on which it serves. A `NetworkLoadBalancerRoute` attaches to a named listener and points to a named port on a `NetworkService`. The first release accepts one catch-all route per listener; a conflicting route reports a condition. @@ -72,7 +73,7 @@ location to `All` creates another unicast IP while the anycast IP stays stable. ### User Stories -- A game operator publishes the same TCP port on a different IP in each region +- A game operator publishes the same UDP port on a different IP in each region so clients can choose a region. - The operator also publishes one global IP on that listener so clients can use a single endpoint; the status shows which regions currently announce it. @@ -110,7 +111,7 @@ spec: ports: - name: game port: 27015 - protocol: TCP + protocol: UDP --- apiVersion: networking.datumapis.com/v1alpha kind: NetworkLoadBalancer @@ -123,17 +124,17 @@ spec: addresses: - name: regional mode: Unicast - family: IPv4 + family: IPv6 locations: from: All - name: global mode: Anycast - family: IPv4 + family: IPv6 locations: from: All listeners: - name: game - protocol: TCP + protocol: UDP port: 27015 addressRefs: [regional, global] --- @@ -161,20 +162,20 @@ per location; anycast produces one entry with several announcements: status: addresses: - requestName: regional - family: IPv4 - value: 192.0.2.10 + family: IPv6 + value: 2001:db8:100::10 announcements: - location: us-central-1 state: Announced - requestName: regional - family: IPv4 - value: 192.0.2.11 + family: IPv6 + value: 2001:db8:200::11 announcements: - location: eu-west-1 state: Announced - requestName: global - family: IPv4 - value: 198.51.100.20 + family: IPv6 + value: 2001:db8:300::20 announcements: - location: us-central-1 state: Announced @@ -182,8 +183,11 @@ status: state: Announced ``` -These IPs are illustrative. An allocation remains in status when a route or -backend is unavailable; its announcement state changes instead. Load balancer +These IPs are illustrative. IPv6 is shown because Galactic's current public +ingress packet path handles IPv6. IPv4 address requests need IPv4 ingress +support before they can be served and must be rejected where unavailable. An +allocation remains in status when a route or backend is unavailable; its +announcement state changes instead. Load balancer conditions distinguish allocation and programming. Route conditions report attachment, backend resolution, and programming. `Ready` means at least one selected location serves traffic; location status identifies any that do not. @@ -194,14 +198,21 @@ and reports status. Cell controllers turn resolved intent into Galactic backends in the ingress location. The same anycast IP is installed in every selected cell; the control plane must not allocate one IP per cell for it. +For UDP, packets from the same client address and port select the same backend +while the backend set is unchanged. A backend change may move that flow; +the first release offers no UDP session or draining guarantee. Backend +availability initially follows `NetworkService` membership and its holder +availability signal, not an active UDP probe. + ### Route Controls The listener selects destination IP, protocol, and port. The route selects a -backend for each new connection. The following controls are candidates for -later releases, after the single-backend catch-all route works: +backend for each TCP connection or UDP flow. The following controls are +candidates for later releases, after the single-backend catch-all route works: -- **Weighted backends:** Split new connections among `NetworkService` backends - by weight. Existing connections do not move when weights change. +- **Weighted backends:** Split new TCP connections or UDP flows among + `NetworkService` backends by weight. Changing weights must not interrupt + established TCP connections; UDP flow stability needs an explicit contract. - **Priority failover:** Use a standby backend when the preferred one lacks enough healthy members, with an explicit threshold. - **Backend locality:** Select `LocalOnly`, `PreferLocal`, or `Global` backends @@ -210,7 +221,7 @@ later releases, after the single-backend catch-all route works: - **Source CIDR match:** Steer client networks to different backends. Multiple routes on one listener need a default and unambiguous precedence. This is traffic selection, not an access-control substitute. -- **Client affinity:** Keep new connections from one client on a consistent +- **Client affinity:** Keep new flows from one client on a consistent backend when needed. Clients sharing a NAT IP can create uneven load. SNI-based TLS passthrough is a separate TLS route capability, not a match on an @@ -232,10 +243,10 @@ advertisements are gone. ### Rollout, Upgrade and Rollback Planning -Start with single-location TCP, then test multi-location unicast and anycast. -Exercise route updates, loss of the last backend, location removal, and -withdrawal before IP release. Existing `NetworkService` and `HTTPProxy` -behavior should be unaffected. +Start with single-location TCP and UDP, then test multi-location unicast and +anycast. Exercise route updates, UDP flow stability, loss of the last backend, +location removal, and withdrawal before IP release. Existing `NetworkService` +and `HTTPProxy` behavior should be unaffected. ### Monitoring Requirements @@ -246,8 +257,11 @@ load balancer and route conditions and the per-address announcement status. ### Dependencies IPAM must provide globally unique anycast VIPs and location-specific unicast -VIPs. Selected cells need Galactic gateways, backend VIP bindings, and a way -to report programming and announcement state. +VIPs. NSO must allow UDP `NetworkService` ports and require the route listener +and backend port protocols to match; `HTTPProxy` must continue to reject UDP +backends. Selected cells need Galactic gateways, backend VIP bindings, and a +way to report programming and announcement state. IPv4 VIPs additionally need +Galactic IPv4 ingress support. ### Scalability From f3c72d5b415165392357b4c8f5eea466fd6ee266 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sun, 11 Oct 2026 17:40:55 -0500 Subject: [PATCH 05/25] docs: add TLS and backend health to NLB proposal --- docs/enhancements/network-load-balancers.md | 104 ++++++++++++++------ 1 file changed, 76 insertions(+), 28 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index b5ebc4df..58b9158a 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -14,6 +14,8 @@ latest-milestone: "v0.x" - [User Stories](#user-stories) - [Risks and Mitigations](#risks-and-mitigations) - [Design Details](#design-details) + - [Backend Health](#backend-health) + - [TLS](#tls) - [Route Controls](#route-controls) - [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) - [Implementation History](#implementation-history) @@ -28,7 +30,8 @@ A project can publish a `NetworkService` on a public TCP or UDP port through a requests; a separate `NetworkLoadBalancerRoute` connects a listener to the service. Consumers choose distinct IPs in each location, one IP shared across locations, or both on the same listener. Status shows the allocated IPs and -where each is announced. +where each is announced. TCP listeners can pass encrypted traffic to the +backend, and active health checks keep failed backends out of rotation. ## Motivation @@ -41,6 +44,10 @@ location-specific address or a shared address reachable from several locations. - Publish a TCP or UDP `NetworkService` through a named listener and separate route. +- Pass TLS through a TCP listener without requiring a certificate on the load + balancer, and actively check backend health before serving traffic. +- Define managed TLS termination as a future listener mode for non-HTTP + applications. - Support per-location unicast and multi-location anycast address requests, including a mix of both on one load balancer. - Show allocated addresses and serving locations without releasing an address @@ -51,9 +58,10 @@ location-specific address or a shared address reachable from several locations. ### Non-Goals -- HTTP routing, TLS termination, or changing the current `HTTPProxy` API. -- Active health probes, weighted backends, source-based matches, or - cross-location backend fallback in the first release. +- HTTP routing, managed TLS termination, SNI routing, or changing the current + `HTTPProxy` API in the first release. +- Weighted backends, source-based matches, or cross-location backend fallback + in the first release. - Requiring consumers to create Gateway API or Galactic resources. ## Proposal @@ -79,12 +87,15 @@ location to `All` creates another unicast IP while the anycast IP stays stable. a single endpoint; the status shows which regions currently announce it. - If a region loses its serving backends, its IP remains allocated while its announcement is withdrawn. +- An operator exposes a TLS application through a TCP listener; the backend + retains its certificate and the load balancer forwards the encrypted stream. ### Risks and Mitigations - **An announced IP without serving backends drops traffic.** Announce a - location only after its rule is programmed with usable backends; withdraw it - when the last usable backend disappears. + location only after its rule is programmed with healthy backends; withdraw it + when the last healthy backend disappears. Workload readiness alone does not + prove the application answers on its port. - **Reusing an anycast IP for another load balancer misroutes traffic.** Hold one globally unique allocation and propagate that allocation to every selected location. @@ -93,6 +104,9 @@ location to `All` creates another unicast IP while the anycast IP stays stable. - **Overlapping routes could select different backends for the same flow.** When match-based routes are added, require one default and deterministic precedence or reject ambiguous overlaps. +- **Probe failure can withdraw a healthy location.** Report probe errors + separately from an unhealthy backend and avoid making one probe runner a + single point of failure. ## Design Details @@ -112,6 +126,9 @@ spec: - name: game port: 27015 protocol: UDP + - name: health + port: 8080 + protocol: TCP --- apiVersion: networking.datumapis.com/v1alpha kind: NetworkLoadBalancer @@ -150,6 +167,10 @@ spec: - networkService: name: game port: game + healthCheck: + protocol: HTTP + port: health + path: /healthz ``` `backends` is a list so a later weighted split or ordered failover does not @@ -187,9 +208,9 @@ These IPs are illustrative. IPv6 is shown because Galactic's current public ingress packet path handles IPv6. IPv4 address requests need IPv4 ingress support before they can be served and must be rejected where unavailable. An allocation remains in status when a route or backend is unavailable; its -announcement state changes instead. Load balancer -conditions distinguish allocation and programming. Route conditions report -attachment, backend resolution, and programming. `Ready` means at least one +announcement state changes instead. Load balancer conditions distinguish +allocation and programming. Route conditions report attachment, backend +resolution, and programming. `Ready` means at least one selected location serves traffic; location status identifies any that do not. NSO resolves service membership, allocates IPs, selects eligible locations, @@ -199,10 +220,37 @@ backends in the ingress location. The same anycast IP is installed in every selected cell; the control plane must not allocate one IP per cell for it. For UDP, packets from the same client address and port select the same backend -while the backend set is unchanged. A backend change may move that flow; -the first release offers no UDP session or draining guarantee. Backend -availability initially follows `NetworkService` membership and its holder -availability signal, not an active UDP probe. +while the backend set is unchanged. A backend change may move that flow; the +first release offers no UDP session or draining guarantee. + +### Backend Health + +Health checks are configured per route backend. A TCP backend defaults to a +TCP connect probe on its serving port; it may instead name a TCP, HTTP, or +HTTPS probe on another `NetworkService` port. A UDP backend must name a TCP, +HTTP, or HTTPS health endpoint that reports whether the UDP application can +serve. There is no generic UDP probe with reliable success semantics. Probe +interval, timeout, and healthy and unhealthy thresholds have platform defaults +and can be set on the health check. + +An endpoint serves only when its workload reports `HolderAvailable` and its +active probe passes. Probes run from each serving location. The cell publishes +one healthy backend set for all of its gateway nodes so anycast nodes make +consistent selections. A location with no healthy backends withdraws its +announcement but keeps its IP allocation. Status reports healthy backend +counts and probe failures by location. + +### TLS + +A TCP listener forwards TLS bytes unchanged. The application terminates TLS, +owns its certificate, and can use any TCP port. No TLS setting is needed on +the first-release listener. + +A future `TLS` listener may terminate TLS with a certificate reference for +non-HTTP applications. SNI-based dispatch would use a distinct TLS route +capability. Both need a proxy-capable data plane and certificate lifecycle; +Galactic's current DSR rule cannot terminate or inspect TLS. HTTP termination +remains with `HTTPProxy`. ### Route Controls @@ -224,11 +272,9 @@ candidates for later releases, after the single-backend catch-all route works: - **Client affinity:** Keep new flows from one client on a consistent backend when needed. Clients sharing a NAT IP can create uneven load. -SNI-based TLS passthrough is a separate TLS route capability, not a match on an -ordinary TCP route. Backend health checks belong with `NetworkService` or a -backend policy. Connection draining, source-IP preservation, and PROXY protocol -are delivery settings rather than route matches; they need their own data plane -contracts. Galactic's current rule has a backend list but no weights, route +Connection draining, source-IP preservation, and PROXY protocol are delivery +settings rather than route matches; they need their own data plane contracts. +Galactic's current rule has a backend list but no weights, route matches, or priority policy, so these controls need implementation before the API accepts them. @@ -243,25 +289,27 @@ advertisements are gone. ### Rollout, Upgrade and Rollback Planning -Start with single-location TCP and UDP, then test multi-location unicast and -anycast. Exercise route updates, UDP flow stability, loss of the last backend, -location removal, and withdrawal before IP release. Existing `NetworkService` -and `HTTPProxy` behavior should be unaffected. +Start with single-location TCP and UDP with active probes, then test +multi-location unicast and anycast. Exercise route updates, UDP flow stability, +probe failures, loss of the last backend, location removal, and withdrawal +before IP release. Existing `NetworkService` and `HTTPProxy` behavior should +be unaffected. ### Monitoring Requirements -Operators need counts of allocated VIPs, programmed rules, and announced -locations, plus failures to allocate, program, or withdraw. Consumers read -load balancer and route conditions and the per-address announcement status. +Operators need counts of allocated VIPs, programmed rules, healthy backends, +and announced locations, plus failures to allocate, probe, program, or +withdraw. Consumers read load balancer and route conditions, backend health, +and per-address announcement status. ### Dependencies IPAM must provide globally unique anycast VIPs and location-specific unicast VIPs. NSO must allow UDP `NetworkService` ports and require the route listener and backend port protocols to match; `HTTPProxy` must continue to reject UDP -backends. Selected cells need Galactic gateways, backend VIP bindings, and a -way to report programming and announcement state. IPv4 VIPs additionally need -Galactic IPv4 ingress support. +backends. Selected cells need a health probe runner, Galactic gateways, +backend VIP bindings, and a way to report programming and announcement state. +IPv4 VIPs additionally need Galactic IPv4 ingress support. ### Scalability From 357304733ede21f726f5b666e36253d45049f4e8 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sun, 11 Oct 2026 17:47:53 -0500 Subject: [PATCH 06/25] docs: define SNI routing and managed TLS options --- docs/enhancements/network-load-balancers.md | 136 +++++++++++++++----- 1 file changed, 107 insertions(+), 29 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index 58b9158a..1b5a25a7 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -30,8 +30,9 @@ A project can publish a `NetworkService` on a public TCP or UDP port through a requests; a separate `NetworkLoadBalancerRoute` connects a listener to the service. Consumers choose distinct IPs in each location, one IP shared across locations, or both on the same listener. Status shows the allocated IPs and -where each is announced. TCP listeners can pass encrypted traffic to the -backend, and active health checks keep failed backends out of rotation. +where each is announced. Active health checks keep failed backends out of +rotation. TLS listeners can route by SNI and either pass encryption to the +backend or terminate it with a Datum-managed certificate. ## Motivation @@ -44,10 +45,9 @@ location-specific address or a shared address reachable from several locations. - Publish a TCP or UDP `NetworkService` through a named listener and separate route. -- Pass TLS through a TCP listener without requiring a certificate on the load - balancer, and actively check backend health before serving traffic. -- Define managed TLS termination as a future listener mode for non-HTTP - applications. +- Actively check backend health before serving traffic. +- Support opaque TLS over TCP, SNI-based TLS passthrough, and managed TLS + termination for non-HTTP applications. - Support per-location unicast and multi-location anycast address requests, including a mix of both on one load balancer. - Show allocated addresses and serving locations without releasing an address @@ -58,19 +58,25 @@ location-specific address or a shared address reachable from several locations. ### Non-Goals -- HTTP routing, managed TLS termination, SNI routing, or changing the current - `HTTPProxy` API in the first release. +- HTTP routing or changing the current `HTTPProxy` API. - Weighted backends, source-based matches, or cross-location backend fallback in the first release. - Requiring consumers to create Gateway API or Galactic resources. ## Proposal -A `NetworkLoadBalancer` names a network, requests addresses, and defines TCP -or UDP listeners. Each listener names the address requests on which it serves. A -`NetworkLoadBalancerRoute` attaches to a named listener and points to a named -port on a `NetworkService`. The first release accepts one catch-all route per -listener; a conflicting route reports a condition. +A `NetworkLoadBalancer` names a network, requests addresses, and defines TCP, +UDP, or TLS listeners. Each listener names the address requests on which it +serves. A `NetworkLoadBalancerRoute` attaches to a named listener and points to +a named port on a `NetworkService`. The initial TCP/UDP slice accepts one +catch-all route per listener; a conflicting route reports a condition. + +A TLS listener adds `tls.mode: Passthrough` or `Terminate`. Routes attached to +it can match SNI hostnames so several TLS services share an address and port. +Terminate mode requests a Datum-managed certificate for verified hostnames. +Plain TCP listeners continue to carry TLS without inspecting it. The initial +delivery can ship TCP/UDP routing and health checks before TLS listeners; the +TLS API remains part of this enhancement. `Unicast` allocates one IP per selected location. `Anycast` allocates one IP announced from all selected locations. These names describe addressing *across @@ -89,6 +95,10 @@ location to `All` creates another unicast IP while the anycast IP stays stable. announcement is withdrawn. - An operator exposes a TLS application through a TCP listener; the backend retains its certificate and the load balancer forwards the encrypted stream. +- An operator routes `db.example.com` and `mq.example.com` on the same TLS + port by SNI while each backend terminates its own TLS. +- An operator uses a Datum-managed certificate on a TLS listener, so a + non-HTTP backend can receive decrypted TCP without managing that certificate. ### Risks and Mitigations @@ -107,6 +117,14 @@ location to `All` creates another unicast IP while the anycast IP stays stable. - **Probe failure can withdraw a healthy location.** Report probe errors separately from an unhealthy backend and avoid making one probe runner a single point of failure. +- **A TLS hostname could be served with the wrong certificate or backend.** + Verify domain ownership before issuing managed certificates, require route + hostnames to be covered by the listener's certificate in Terminate mode, and + reject ambiguous SNI matches. +- **One unready listener can affect other ports on its IP.** BGP announces an + address, not a port. Announce an address from a location only when all of its + listeners are ready there; use separate address requests for independent + failure domains. ## Design Details @@ -212,6 +230,8 @@ announcement state changes instead. Load balancer conditions distinguish allocation and programming. Route conditions report attachment, backend resolution, and programming. `Ready` means at least one selected location serves traffic; location status identifies any that do not. +An address is announced from a location only when every listener using it is +ready there. NSO resolves service membership, allocates IPs, selects eligible locations, and reports status. Cell controllers turn resolved intent into Galactic @@ -242,15 +262,67 @@ counts and probe failures by location. ### TLS -A TCP listener forwards TLS bytes unchanged. The application terminates TLS, -owns its certificate, and can use any TCP port. No TLS setting is needed on -the first-release listener. +A TCP listener forwards TLS bytes unchanged to a backend that owns its +certificate. A TLS listener adds two explicit modes: + +- **Passthrough:** Inspect the ClientHello SNI only to choose a route, then + forward the encrypted stream. The backend owns its certificate. +- **Terminate:** Select a certificate and end client TLS at the load balancer. + `certificate.mode: Managed` uses Datum's domain verification and certificate + lifecycle. The route must explicitly choose plain TCP or re-encrypted TLS + for the backend connection. + +The proposed TLS-specific fields are: + +```yaml +# NetworkLoadBalancer spec fragment +spec: + listeners: + - name: secure + protocol: TLS + port: 443 + tls: + mode: Terminate + certificate: + mode: Managed + hostnames: [db.example.com, mq.example.com] +--- +# NetworkLoadBalancerRoute spec fragment +spec: + parentRef: + name: services + listener: secure + sniHostnames: [db.example.com] + backends: + - networkService: + name: database + port: database + transport: + protocol: TCP +``` -A future `TLS` listener may terminate TLS with a certificate reference for -non-HTTP applications. SNI-based dispatch would use a distinct TLS route -capability. Both need a proxy-capable data plane and certificate lifecycle; -Galactic's current DSR rule cannot terminate or inspect TLS. HTTP termination -remains with `HTTPProxy`. +`sniHostnames` is valid only on a TLS listener. Initial matching uses exact +hostnames. Each hostname belongs to one route on that listener; an omitted SNI +match is the single default for clients without a matching SNI. Without a +default, the listener rejects those connections. Terminate mode requires +verified hostnames covered by a ready certificate before serving them. Status +reports hostname verification and certificate readiness per listener and +location, separately from VIP allocation. A certificate failure withdraws +that listener's address from the affected location until it is ready; +unrelated addresses remain available. A no-SNI client needs an explicit +`certificate.defaultHostname` and a default route, or the connection is rejected. + +The terminated route chooses backend transport explicitly: plain `TCP` as +shown, or `TLS` with a server name and certificate validation. Passthrough +uses `tls.mode: Passthrough`, omits the certificate, and keeps end-to-end TLS; +its routes do not set backend transport. Only visible SNI can be matched; +connections without a matching visible name use the default or are rejected. + +SNI inspection and TLS termination need a proxy-capable data plane; Galactic's +current DSR rule only forwards TCP and cannot provide either. HTTP routing and +termination remain with `HTTPProxy`. A terminating proxy opens a new backend +connection, so its source-IP behavior and optional PROXY protocol delivery +must be explicit to consumers. ### Route Controls @@ -290,17 +362,19 @@ advertisements are gone. ### Rollout, Upgrade and Rollback Planning Start with single-location TCP and UDP with active probes, then test -multi-location unicast and anycast. Exercise route updates, UDP flow stability, -probe failures, loss of the last backend, location removal, and withdrawal -before IP release. Existing `NetworkService` and `HTTPProxy` behavior should -be unaffected. +multi-location unicast and anycast. Add SNI passthrough before managed TLS +termination. Exercise route updates, UDP flow stability, probe failures, loss +of the last backend, location removal, TLS certificate issuance and rotation, +and withdrawal before IP release. Existing `NetworkService` and `HTTPProxy` +behavior should be unaffected. ### Monitoring Requirements Operators need counts of allocated VIPs, programmed rules, healthy backends, -and announced locations, plus failures to allocate, probe, program, or -withdraw. Consumers read load balancer and route conditions, backend health, -and per-address announcement status. +and announced locations, plus failures to allocate, probe, program, issue +certificates, or withdraw. Consumers read load balancer and route conditions, +backend health, TLS hostname and certificate status, and per-address +announcement status. ### Dependencies @@ -310,6 +384,8 @@ and backend port protocols to match; `HTTPProxy` must continue to reject UDP backends. Selected cells need a health probe runner, Galactic gateways, backend VIP bindings, and a way to report programming and announcement state. IPv4 VIPs additionally need Galactic IPv4 ingress support. +SNI routing and managed TLS need a proxy-capable edge path, hostname ownership +verification, and certificate issuance and distribution. ### Scalability @@ -346,4 +422,6 @@ withdrawal across cells. ## Infrastructure Needed Public VIP pools, cell-level programming and status propagation, and a -multi-location traffic test environment are required before rollout. +multi-location traffic test environment are required before rollout. TLS +listeners additionally need an edge proxy tier and managed certificate +distribution. From 4c96d68b4a3b73322389dbf67ec120e2cdffbe51 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sun, 11 Oct 2026 17:52:03 -0500 Subject: [PATCH 07/25] docs: define Gateway API L4 compatibility target --- docs/enhancements/network-load-balancers.md | 115 ++++++++++++++------ 1 file changed, 84 insertions(+), 31 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index 1b5a25a7..e6cf1336 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -16,6 +16,7 @@ latest-milestone: "v0.x" - [Design Details](#design-details) - [Backend Health](#backend-health) - [TLS](#tls) + - [Gateway API L4 Compatibility](#gateway-api-l4-compatibility) - [Route Controls](#route-controls) - [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) - [Implementation History](#implementation-history) @@ -53,14 +54,15 @@ location-specific address or a shared address reachable from several locations. - Show allocated addresses and serving locations without releasing an address when its backends are temporarily unavailable. - Use an address request shape that can later serve Application Load Balancers. +- Cover Gateway API's TCPRoute, UDPRoute, and TLSRoute behavior, including its + optional L4 features, before claiming full L4 compatibility. - Leave room for common L4 traffic controls without requiring them in the first release. ### Non-Goals - HTTP routing or changing the current `HTTPProxy` API. -- Weighted backends, source-based matches, or cross-location backend fallback - in the first release. +- Source-based matches or cross-location backend fallback in the first release. - Requiring consumers to create Gateway API or Galactic resources. ## Proposal @@ -68,12 +70,15 @@ location-specific address or a shared address reachable from several locations. A `NetworkLoadBalancer` names a network, requests addresses, and defines TCP, UDP, or TLS listeners. Each listener names the address requests on which it serves. A `NetworkLoadBalancerRoute` attaches to a named listener and points to -a named port on a `NetworkService`. The initial TCP/UDP slice accepts one -catch-all route per listener; a conflicting route reports a condition. +named ports on `NetworkService` backends. TCP and UDP listeners have one +effective catch-all route each. If several attach, the oldest wins, breaking +timestamp ties by namespace/name; all attachments remain Accepted, and native +status identifies the effective route. Each route can distribute new +connections or flows across weighted backends. A TLS listener adds `tls.mode: Passthrough` or `Terminate`. Routes attached to it can match SNI hostnames so several TLS services share an address and port. -Terminate mode requests a Datum-managed certificate for verified hostnames. +Terminate mode uses a Datum-managed certificate or a referenced TLS Secret. Plain TCP listeners continue to carry TLS without inspecting it. The initial delivery can ship TCP/UDP routing and health checks before TLS listeners; the TLS API remains part of this enhancement. @@ -111,16 +116,16 @@ location to `All` creates another unicast IP while the anycast IP stays stable. selected location. - **An `All` request can expand over time.** Expose each added location and IP in status, and bound the number of locations and addresses per load balancer. -- **Overlapping routes could select different backends for the same flow.** When - match-based routes are added, require one default and deterministic precedence - or reject ambiguous overlaps. +- **Overlapping routes could select different backends for the same flow.** Use + Gateway API precedence for overlapping route attachments and report the + effective route in native status. - **Probe failure can withdraw a healthy location.** Report probe errors separately from an unhealthy backend and avoid making one probe runner a single point of failure. - **A TLS hostname could be served with the wrong certificate or backend.** Verify domain ownership before issuing managed certificates, require route hostnames to be covered by the listener's certificate in Terminate mode, and - reject ambiguous SNI matches. + apply deterministic SNI precedence. - **One unready listener can affect other ports on its IP.** BGP announces an address, not a port. Announce an address from a location only when all of its listeners are ready there; use separate address requests for independent @@ -178,9 +183,9 @@ kind: NetworkLoadBalancerRoute metadata: name: game spec: - parentRef: - name: game - listener: game + parentRefs: + - name: game + listener: game backends: - networkService: name: game @@ -191,8 +196,12 @@ spec: path: /healthz ``` -`backends` is a list so a later weighted split or ordered failover does not -require replacing the field. The first release requires exactly one entry. +`backends` accepts multiple entries with an optional `weight` (default 1, +zero disables selection). Weights split new connections or flows, not packets +within a flow. An unresolved backend or one with no endpoints retains its +weight and its share is rejected, as Gateway API requires. Failed health probes +exclude individual endpoints while healthy endpoints remain. Status +distinguishes resolution, endpoint availability, and health. The load balancer reports one entry per allocated IP. Unicast produces an entry per location; anycast produces one entry with several announcements: @@ -289,9 +298,9 @@ spec: --- # NetworkLoadBalancerRoute spec fragment spec: - parentRef: - name: services - listener: secure + parentRefs: + - name: services + listener: secure sniHostnames: [db.example.com] backends: - networkService: @@ -301,9 +310,11 @@ spec: protocol: TCP ``` -`sniHostnames` is valid only on a TLS listener. Initial matching uses exact -hostnames. Each hostname belongs to one route on that listener; an omitted SNI -match is the single default for clients without a matching SNI. Without a +`sniHostnames` is valid only on a TLS listener. Exact and `*.` wildcard DNS +names use Gateway API hostname intersection and precedence rules with the +listener's optional `hostname`. Each effective hostname belongs to one route +on that listener; an omitted SNI match is the single default for clients +without a matching SNI. Without a default, the listener rejects those connections. Terminate mode requires verified hostnames covered by a ready certificate before serving them. Status reports hostname verification and certificate readiness per listener and @@ -312,8 +323,11 @@ that listener's address from the affected location until it is ready; unrelated addresses remain available. A no-SNI client needs an explicit `certificate.defaultHostname` and a default route, or the connection is rejected. -The terminated route chooses backend transport explicitly: plain `TCP` as -shown, or `TLS` with a server name and certificate validation. Passthrough +The terminating listener alternatively accepts +`certificate: {mode: SecretRef, secretRef: {name: ...}}`; cross-namespace +references require an explicit grant. The terminated route chooses backend +transport explicitly: plain `TCP` as shown, or `TLS` with a server name and +certificate validation (system roots or referenced CA bundle). Passthrough uses `tls.mode: Passthrough`, omits the certificate, and keeps end-to-end TLS; its routes do not set backend transport. Only visible SNI can be matched; connections without a matching visible name use the default or are rejected. @@ -324,15 +338,49 @@ termination remain with `HTTPProxy`. A terminating proxy opens a new backend connection, so its source-IP behavior and optional PROXY protocol delivery must be explicit to consumers. +### Gateway API L4 Compatibility + +The Datum API is the primary consumer experience. Its L4 behavior must cover +[TCPRoute](https://gateway-api.sigs.k8s.io/guides/user-guides/tcp/), +[UDPRoute](https://gateway-api.sigs.k8s.io/guides/user-guides/udp/), +[TLSRoute](https://gateway-api.sigs.k8s.io/reference/api-types/tlsroute/), +and their Core and Extended features. A Gateway API adapter must translate +Kubernetes objects into the same internal intent, so users can choose either +API. Full compatibility means passing the relevant Gateway API conformance +tests and documenting any implementation-specific backend mapping. + +| Gateway API capability | Datum API behavior | +| --- | --- | +| Gateway addresses, named TCP/UDP/TLS listeners, ports and TLS mode | `NetworkLoadBalancer.exposure.addresses` and `listeners`; Datum additionally selects unicast/anycast and locations. | +| TCPRoute and UDPRoute, listener attachment, backendRefs and weights | `NetworkLoadBalancerRoute.parentRefs` and weighted `backends`; one effective catch-all per TCP/UDP listener, with Gateway API precedence. | +| TLSRoute SNI hostnames, wildcard matching and precedence | TLS listener `hostname` and route `sniHostnames`; passthrough and termination, including distinct TLS listeners on one port when hostnames do not conflict. | +| TLS certificates and backend re-encryption | Listener managed certificate or Secret reference; route backend TLS with SNI and trust validation, corresponding to BackendTLSPolicy. | +| AllowedRoutes, parentRefs, cross-namespace ReferenceGrant and status | Explicit attachment policy and grants; report Accepted, ResolvedRefs and programming per attachment, plus listener conflicts and readiness. | +| ListenerSet and multiple parent attachments | Adapter expands these into listeners and route attachments while preserving ownership, conflict resolution and status for each source object. | + +The native route needs multiple `parentRefs` so one route can attach to several +listeners or load balancers, with optional listener name or port selection. +Listener attachment policy defaults to the same namespace; cross-namespace +attachment and backend or certificate references require grants. The adapter +maps Kubernetes Services and their numbered ports to resolved backends; +`NetworkService` remains the native backend type. It must preserve Gateway API +rules for invalid backend shares, conflicting listeners and routes, hostname +intersection, and per-parent status. TCP and UDP may share a port; TCP and TLS +may not. Datum-only source-CIDR and locality policies cannot +change the behavior of a Gateway API resource unless explicitly configured +through a Datum extension. + +The first delivery may expose a smaller native feature set, but it must not +claim Gateway API L4 compatibility until TCPRoute, UDPRoute, TLSRoute, their +L4 Extended features, and the adapter pass conformance. Track the target +against a pinned Gateway API release; the upstream API can evolve independently. + ### Route Controls The listener selects destination IP, protocol, and port. The route selects a backend for each TCP connection or UDP flow. The following controls are -candidates for later releases, after the single-backend catch-all route works: +candidates for later releases beyond Gateway API L4 parity: -- **Weighted backends:** Split new TCP connections or UDP flows among - `NetworkService` backends by weight. Changing weights must not interrupt - established TCP connections; UDP flow stability needs an explicit contract. - **Priority failover:** Use a standby backend when the preferred one lacks enough healthy members, with an explicit threshold. - **Backend locality:** Select `LocalOnly`, `PreferLocal`, or `Global` backends @@ -346,9 +394,9 @@ candidates for later releases, after the single-backend catch-all route works: Connection draining, source-IP preservation, and PROXY protocol are delivery settings rather than route matches; they need their own data plane contracts. -Galactic's current rule has a backend list but no weights, route -matches, or priority policy, so these controls need implementation before the -API accepts them. +Galactic's current rule has a backend list but no weights, route matches, or +priority policy. Weighted selection is required for Gateway API parity; +the other controls need implementation before the API accepts them. ## Production Readiness Review Questionnaire @@ -361,8 +409,8 @@ advertisements are gone. ### Rollout, Upgrade and Rollback Planning -Start with single-location TCP and UDP with active probes, then test -multi-location unicast and anycast. Add SNI passthrough before managed TLS +Start with single-location TCP and UDP with weighted backends and active probes, +then test multi-location unicast and anycast. Add SNI passthrough before managed TLS termination. Exercise route updates, UDP flow stability, probe failures, loss of the last backend, location removal, TLS certificate issuance and rotation, and withdrawal before IP release. Existing `NetworkService` and `HTTPProxy` @@ -386,6 +434,10 @@ backend VIP bindings, and a way to report programming and announcement state. IPv4 VIPs additionally need Galactic IPv4 ingress support. SNI routing and managed TLS need a proxy-capable edge path, hostname ownership verification, and certificate issuance and distribution. +Gateway API parity also needs an attachment/grant model, Service backend +translation, Secret certificate references, weighted selection, wildcard SNI, +mixed TLS modes, backend TLS validation, per-parent status, and conformance +tests against a pinned upstream release. ### Scalability @@ -418,6 +470,7 @@ withdrawal across cells. leaves no independent route lifecycle or place for future route policy. - **Expose Gateway API directly.** It adds concepts consumers do not need for this use case and does not express Datum's location and address policies. + Offer an adapter for portability while retaining the Datum API. ## Infrastructure Needed From e8370eaf43c0509e1fab4282528cd35645bf2fe2 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Mon, 12 Oct 2026 12:11:24 -0400 Subject: [PATCH 08/25] docs: scope NLB addresses to selected locations --- docs/enhancements/network-load-balancers.md | 115 ++++++++++---------- 1 file changed, 55 insertions(+), 60 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index e6cf1336..f9033e88 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -29,18 +29,18 @@ latest-milestone: "v0.x" A project can publish a `NetworkService` on a public TCP or UDP port through a `NetworkLoadBalancer`. The load balancer owns listeners and public address requests; a separate `NetworkLoadBalancerRoute` connects a listener to the -service. Consumers choose distinct IPs in each location, one IP shared across -locations, or both on the same listener. Status shows the allocated IPs and -where each is announced. Active health checks keep failed backends out of -rotation. TLS listeners can route by SNI and either pass encryption to the -backend or terminate it with a Datum-managed certificate. +service. Each selected location receives a distinct public IP. Status shows +the allocated IP and announcement state for each location. Active health +checks keep failed backends out of rotation. TLS listeners can route by SNI and +either pass encryption to the backend or terminate it with a Datum-managed +certificate. ## Motivation `NetworkService` identifies workloads and their ports, but does not give them a public Layer 4 entry point. Consumers need a stable address and port without -managing Galactic resources. They also need to choose whether clients use a -location-specific address or a shared address reachable from several locations. +managing Galactic resources. They also need stable addresses in the locations +where their service runs. ### Goals @@ -49,8 +49,8 @@ location-specific address or a shared address reachable from several locations. - Actively check backend health before serving traffic. - Support opaque TLS over TCP, SNI-based TLS passthrough, and managed TLS termination for non-HTTP applications. -- Support per-location unicast and multi-location anycast address requests, - including a mix of both on one load balancer. +- Allocate a distinct public address in each selected location, including + locations added after an `All` request is created. - Show allocated addresses and serving locations without releasing an address when its backends are temporarily unavailable. - Use an address request shape that can later serve Application Load Balancers. @@ -63,6 +63,9 @@ location-specific address or a shared address reachable from several locations. - HTTP routing or changing the current `HTTPProxy` API. - Source-based matches or cross-location backend fallback in the first release. +- Anycast or another single public IP shared across locations. A future Global + Services Load Balancer will own global entry points and cross-location + service behavior. - Requiring consumers to create Gateway API or Galactic resources. ## Proposal @@ -83,19 +86,18 @@ Plain TCP listeners continue to carry TLS without inspecting it. The initial delivery can ship TCP/UDP routing and health checks before TLS listeners; the TLS API remains part of this enhancement. -`Unicast` allocates one IP per selected location. `Anycast` allocates one IP -announced from all selected locations. These names describe addressing *across -locations*: multiple edge nodes may announce an IP within one location. -`locations.from: All` follows current and future eligible locations. An -explicit `Names` selection limits the request to listed locations. Adding a -location to `All` creates another unicast IP while the anycast IP stays stable. +Each selected location receives its own IP. Multiple edge nodes may announce +that IP within the location. `locations.from: All` follows current and future +eligible locations; an explicit `Names` selection limits the request to listed +locations. Adding a location to `All` creates another IP without changing the +existing location addresses. ### User Stories - A game operator publishes the same UDP port on a different IP in each region so clients can choose a region. -- The operator also publishes one global IP on that listener so clients can use - a single endpoint; the status shows which regions currently announce it. +- The operator adds a new region while existing regional client endpoints keep + their addresses. - If a region loses its serving backends, its IP remains allocated while its announcement is withdrawn. - An operator exposes a TLS application through a TCP listener; the backend @@ -111,9 +113,8 @@ location to `All` creates another unicast IP while the anycast IP stays stable. location only after its rule is programmed with healthy backends; withdraw it when the last healthy backend disappears. Workload readiness alone does not prove the application answers on its port. -- **Reusing an anycast IP for another load balancer misroutes traffic.** Hold - one globally unique allocation and propagate that allocation to every - selected location. +- **Reusing a public IP for another load balancer misroutes traffic.** Allocate + every location address from a globally unique public pool. - **An `All` request can expand over time.** Expose each added location and IP in status, and bound the number of locations and addresses per load balancer. - **Overlapping routes could select different backends for the same flow.** Use @@ -163,12 +164,6 @@ spec: exposure: addresses: - name: regional - mode: Unicast - family: IPv6 - locations: - from: All - - name: global - mode: Anycast family: IPv6 locations: from: All @@ -176,7 +171,7 @@ spec: - name: game protocol: UDP port: 27015 - addressRefs: [regional, global] + addressRefs: [regional] --- apiVersion: networking.datumapis.com/v1alpha kind: NetworkLoadBalancerRoute @@ -203,32 +198,29 @@ weight and its share is rejected, as Gateway API requires. Failed health probes exclude individual endpoints while healthy endpoints remain. Status distinguishes resolution, endpoint availability, and health. -The load balancer reports one entry per allocated IP. Unicast produces an entry -per location; anycast produces one entry with several announcements: +The load balancer reports one entry per requested location and allocated IP: ```yaml status: addresses: - requestName: regional + location: us-central-1 family: IPv6 value: 2001:db8:100::10 - announcements: - - location: us-central-1 - state: Announced + conditions: + - type: Allocated + status: "True" + - type: Announced + status: "True" - requestName: regional + location: eu-west-1 family: IPv6 value: 2001:db8:200::11 - announcements: - - location: eu-west-1 - state: Announced - - requestName: global - family: IPv6 - value: 2001:db8:300::20 - announcements: - - location: us-central-1 - state: Announced - - location: eu-west-1 - state: Announced + conditions: + - type: Allocated + status: "True" + - type: Announced + status: "True" ``` These IPs are illustrative. IPv6 is shown because Galactic's current public @@ -245,8 +237,8 @@ ready there. NSO resolves service membership, allocates IPs, selects eligible locations, and reports status. Cell controllers turn resolved intent into Galactic `NetworkRule` and `ServiceVIPBinding` objects. The initial route uses healthy -backends in the ingress location. The same anycast IP is installed in every -selected cell; the control plane must not allocate one IP per cell for it. +backends in the ingress location. The control plane allocates one IP per +selected location and shares that location's IP among its serving cells. For UDP, packets from the same client address and port select the same backend while the backend set is unchanged. A backend change may move that flow; the @@ -264,7 +256,7 @@ and can be set on the health check. An endpoint serves only when its workload reports `HolderAvailable` and its active probe passes. Probes run from each serving location. The cell publishes -one healthy backend set for all of its gateway nodes so anycast nodes make +one healthy backend set for all gateway nodes in that location so they make consistent selections. A location with no healthy backends withdraws its announcement but keeps its IP allocation. Status reports healthy backend counts and probe failures by location. @@ -351,7 +343,7 @@ tests and documenting any implementation-specific backend mapping. | Gateway API capability | Datum API behavior | | --- | --- | -| Gateway addresses, named TCP/UDP/TLS listeners, ports and TLS mode | `NetworkLoadBalancer.exposure.addresses` and `listeners`; Datum additionally selects unicast/anycast and locations. | +| Gateway addresses, named TCP/UDP/TLS listeners, ports and TLS mode | `NetworkLoadBalancer.exposure.addresses` and `listeners`; Datum additionally selects locations and allocates one address in each. | | TCPRoute and UDPRoute, listener attachment, backendRefs and weights | `NetworkLoadBalancerRoute.parentRefs` and weighted `backends`; one effective catch-all per TCP/UDP listener, with Gateway API precedence. | | TLSRoute SNI hostnames, wildcard matching and precedence | TLS listener `hostname` and route `sniHostnames`; passthrough and termination, including distinct TLS listeners on one port when hostnames do not conflict. | | TLS certificates and backend re-encryption | Listener managed certificate or Secret reference; route backend TLS with SNI and trust validation, corresponding to BackendTLSPolicy. | @@ -410,11 +402,11 @@ advertisements are gone. ### Rollout, Upgrade and Rollback Planning Start with single-location TCP and UDP with weighted backends and active probes, -then test multi-location unicast and anycast. Add SNI passthrough before managed TLS -termination. Exercise route updates, UDP flow stability, probe failures, loss -of the last backend, location removal, TLS certificate issuance and rotation, -and withdrawal before IP release. Existing `NetworkService` and `HTTPProxy` -behavior should be unaffected. +then test multi-location operation with distinct addresses. Add SNI passthrough +before managed TLS termination. Exercise route updates, UDP flow stability, +probe failures, loss of the last backend, location removal, TLS certificate +issuance and rotation, and withdrawal before IP release. Existing +`NetworkService` and `HTTPProxy` behavior should be unaffected. ### Monitoring Requirements @@ -426,11 +418,12 @@ announcement status. ### Dependencies -IPAM must provide globally unique anycast VIPs and location-specific unicast -VIPs. NSO must allow UDP `NetworkService` ports and require the route listener -and backend port protocols to match; `HTTPProxy` must continue to reject UDP -backends. Selected cells need a health probe runner, Galactic gateways, -backend VIP bindings, and a way to report programming and announcement state. +IPAM must provide globally unique public IPv6 VIPs, one for each selected +location. NSO must allow UDP `NetworkService` ports and require the route +listener and backend port protocols to match; `HTTPProxy` must continue to +reject UDP backends. Selected cells need a health probe runner, Galactic +gateways, backend VIP bindings, and a way to report programming and +announcement state. IPv4 VIPs additionally need Galactic IPv4 ingress support. SNI routing and managed TLS need a proxy-capable edge path, hostname ownership verification, and certificate issuance and distribution. @@ -455,12 +448,14 @@ address that could not be allocated. ## Implementation History - 2026-10-11: Initial proposal. +- 2026-10-12: Scoped NLB addresses to one IP per selected location; deferred + global entry points to a separate Global Services Load Balancer concept. ## Drawbacks -Separate routes add an object to a simple single-backend setup. Unicast across -many locations consumes an IP in each one, and anycast requires coordinated -withdrawal across cells. +Separate routes add an object to a simple single-backend setup. Serving many +locations consumes one public IP in each, and each IP requires coordinated +withdrawal across the cells serving its location. ## Alternatives From 7e9f68d347b23c0f6beccc6e2dcc05c7ab72e576 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Mon, 12 Oct 2026 12:22:46 -0400 Subject: [PATCH 09/25] Plan IPv4 ingress for NLB data plane --- docs/enhancements/network-load-balancers.md | 50 ++++++++++++++++++--- 1 file changed, 44 insertions(+), 6 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index f9033e88..436eab33 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -14,6 +14,7 @@ latest-milestone: "v0.x" - [User Stories](#user-stories) - [Risks and Mitigations](#risks-and-mitigations) - [Design Details](#design-details) + - [IPv4 Ingress Data Plane](#ipv4-ingress-data-plane) - [Backend Health](#backend-health) - [TLS](#tls) - [Gateway API L4 Compatibility](#gateway-api-l4-compatibility) @@ -223,11 +224,13 @@ status: status: "True" ``` -These IPs are illustrative. IPv6 is shown because Galactic's current public -ingress packet path handles IPv6. IPv4 address requests need IPv4 ingress -support before they can be served and must be rejected where unavailable. An -allocation remains in status when a route or backend is unavailable; its -announcement state changes instead. Load balancer conditions distinguish +These IPs are illustrative. The first NLB release accepts IPv6 address +requests because Galactic's current public ingress packet path handles IPv6. +The cell intent carries the public VIP family and each backend address family +separately. IPv4 requests remain rejected until the IPv4 ingress path and its +capability checks are ready. An allocation remains in status when a route or +backend is unavailable; its announcement state changes instead. Load balancer +conditions distinguish allocation and programming. Route conditions report attachment, backend resolution, and programming. `Ready` means at least one selected location serves traffic; location status identifies any that do not. @@ -244,6 +247,39 @@ For UDP, packets from the same client address and port select the same backend while the backend set is unchanged. A backend change may move that flow; the first release offers no UDP session or draining guarantee. +### IPv4 Ingress Data Plane + +An IPv4 public VIP can front a service whose workload address is IPv6. The +ingress family and backend family are separate properties of cell intent. The +cell validates the VIP, every backend address, and their declared families +before programming traffic. An unsupported family pair remains unprogrammed +and unannounced, with a specific reason in status. IPv4 address requests stay +disabled in the consumer API until serving cells advertise this capability. + +The IPv4 gateway path needs an IPv4 listener and flow table for TCP and UDP. +Its flow identity includes the original destination VIP, protocol, port, and +client tuple, so two VIPs that target the same backend cannot merge into one +flow. For an IPv4 client and IPv6 backend, use a stateful L4 proxy: it accepts +the IPv4 connection or UDP flow, opens an IPv6 backend flow, and sends replies +through the same state. The current IPv6 DSR forwarding path cannot supply +this cross-family return path. The proxy contract must specify what source +address the backend sees; client address delivery, if offered, needs an +explicit mechanism and cannot be assumed from DSR behavior. + +The IPv4 path must handle IPv4 and IPv6 checksums, MTU and fragmentation, +ICMP errors, TCP resets and timeouts, and UDP flow expiry in both directions. +IPv4 VIPs receive distinct per-location /32 announcements, gated by healthy +backends and successful programming. Withdrawal completes before IPAM releases +an address. A location reports IPv4 and IPv6 programming and announcement +independently, so failure of one family does not hide the state of the other. + +Rollout requires a public IPv4 IPAM class, family-aware cell and Galactic +programming, and a capability signal from every serving gateway in a location. +CI must exercise IPv4 TCP and UDP ingress to IPv6 backends and replies, two +VIPs targeting the same backend port, backend health changes, and announcement +withdrawal. Only then should admission accept IPv4 and allow a listener to +reference both IPv4 and IPv6 address requests. + ### Backend Health Health checks are configured per route backend. A TCP backend defaults to a @@ -424,7 +460,9 @@ listener and backend port protocols to match; `HTTPProxy` must continue to reject UDP backends. Selected cells need a health probe runner, Galactic gateways, backend VIP bindings, and a way to report programming and announcement state. -IPv4 VIPs additionally need Galactic IPv4 ingress support. +IPv4 VIPs additionally need a public IPv4 pool, family-aware cell intent, +Galactic IPv4 ingress and cross-family proxy support, and a serving capability +gate before API admission is enabled. SNI routing and managed TLS need a proxy-capable edge path, hostname ownership verification, and certificate issuance and distribution. Gateway API parity also needs an attachment/grant model, Service backend From 5406d7db11f10db8238d71d31cef989394b2d1ec Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 16 Oct 2026 15:00:28 -0500 Subject: [PATCH 10/25] docs: default NLB locations to Backends --- docs/enhancements/network-load-balancers.md | 65 ++++++++++++++------- 1 file changed, 45 insertions(+), 20 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index 436eab33..91b32adb 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -31,10 +31,11 @@ A project can publish a `NetworkService` on a public TCP or UDP port through a `NetworkLoadBalancer`. The load balancer owns listeners and public address requests; a separate `NetworkLoadBalancerRoute` connects a listener to the service. Each selected location receives a distinct public IP. Status shows -the allocated IP and announcement state for each location. Active health -checks keep failed backends out of rotation. TLS listeners can route by SNI and -either pass encryption to the backend or terminate it with a Datum-managed -certificate. +the allocated IP and announcement state for each location. By default, those +locations follow the network interfaces selected by the backend services. +Active health checks keep failed backends out of rotation. TLS listeners can +route by SNI and either pass encryption to the backend or terminate it with a +Datum-managed certificate. ## Motivation @@ -50,8 +51,8 @@ where their service runs. - Actively check backend health before serving traffic. - Support opaque TLS over TCP, SNI-based TLS passthrough, and managed TLS termination for non-HTTP applications. -- Allocate a distinct public address in each selected location, including - locations added after an `All` request is created. +- Allocate a distinct public address in every backend location by default, + including locations added after the address request is created. - Show allocated addresses and serving locations without releasing an address when its backends are temporarily unavailable. - Use an address request shape that can later serve Application Load Balancers. @@ -88,15 +89,26 @@ delivery can ship TCP/UDP routing and health checks before TLS listeners; the TLS API remains part of this enhancement. Each selected location receives its own IP. Multiple edge nodes may announce -that IP within the location. `locations.from: All` follows current and future -eligible locations; an explicit `Names` selection limits the request to listed -locations. Adding a location to `All` creates another IP without changing the +that IP within the location. Omitting `locations` defaults to +`locations.from: Backends`: an address request follows the locations of the +`NetworkService` backends on listeners using that address. `All` explicitly +selects every eligible platform location; `Names` selects only listed +locations. Adding a backend location creates another IP without changing existing location addresses. +For an address shared by several listeners or SNI routes, `Backends` selects a +location only when every positive-weight backend service on the effective +routes has a selected, bound network interface there. This keeps every route +on the IP usable in the same locations. Consumers can use separate address +requests when services have different footprints. Interface presence selects +locations; health and programming determine announcement. With no attached +route, placement is pending and no location IP is allocated. + ### User Stories - A game operator publishes the same UDP port on a different IP in each region - so clients can choose a region. + where the game service has members, so clients can choose a region without + maintaining a location list. - The operator adds a new region while existing regional client endpoints keep their addresses. - If a region loses its serving backends, its IP remains allocated while its @@ -116,8 +128,9 @@ existing location addresses. prove the application answers on its port. - **Reusing a public IP for another load balancer misroutes traffic.** Allocate every location address from a globally unique public pool. -- **An `All` request can expand over time.** Expose each added location and IP - in status, and bound the number of locations and addresses per load balancer. +- **A `Backends` or `All` request can expand over time.** Expose each added + location and IP in status, and bound the number of locations and addresses + per load balancer. - **Overlapping routes could select different backends for the same flow.** Use Gateway API precedence for overlapping route attachments and report the effective route in native status. @@ -166,8 +179,6 @@ spec: addresses: - name: regional family: IPv6 - locations: - from: All listeners: - name: game protocol: UDP @@ -192,6 +203,10 @@ spec: path: /healthz ``` +The address request omits `locations`, so it follows the `game` service's +selected interfaces. `locations: {from: Backends}` expresses the same choice +explicitly. + `backends` accepts multiple entries with an optional `weight` (default 1, zero disables selection). Weights split new connections or flows, not packets within a flow. An unresolved backend or one with no endpoints retains its @@ -203,6 +218,10 @@ The load balancer reports one entry per requested location and allocated IP: ```yaml status: + addressRequests: + - name: regional + locationSource: Backends + resolvedLocations: [us-central-1, eu-west-1] addresses: - requestName: regional location: us-central-1 @@ -235,13 +254,19 @@ allocation and programming. Route conditions report attachment, backend resolution, and programming. `Ready` means at least one selected location serves traffic; location status identifies any that do not. An address is announced from a location only when every listener using it is -ready there. +ready there. Status reports the resolved locations for each address request +and shows `Backends` as the source when `locations` is omitted. Health loss +withdraws an announcement without releasing its IP; after the last selected +interface leaves a location, withdraw that location's announcement before +releasing its IP. NSO resolves service membership, allocates IPs, selects eligible locations, and reports status. Cell controllers turn resolved intent into Galactic `NetworkRule` and `ServiceVIPBinding` objects. The initial route uses healthy backends in the ingress location. The control plane allocates one IP per -selected location and shares that location's IP among its serving cells. +selected location and shares that location's IP among its serving cells. It +resolves `Backends` from actual membership, not the capped +`NetworkService.status.locations` display list. For UDP, packets from the same client address and port select the same backend while the backend set is unchanged. A backend change may move that flow; the @@ -472,10 +497,10 @@ tests against a pinned upstream release. ### Scalability -Bound listeners, routes, address requests, and selected locations. An `All` -request adds an allocation and cell configuration when a new location becomes -eligible; reconciliation must handle those additions without changing existing -IPs. +Bound listeners, routes, address requests, and selected locations. A dynamic +`Backends` or `All` request adds an allocation and cell configuration when a new +location qualifies; reconciliation must handle those additions without +changing existing IPs. ### Troubleshooting From d805facc4758bcf1fe96be699f0139bb8d1907e3 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 16 Oct 2026 15:08:13 -0500 Subject: [PATCH 11/25] docs: request NLB address families together --- docs/enhancements/network-load-balancers.md | 39 +++++++++++++-------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index 91b32adb..ac4d16d2 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -88,13 +88,18 @@ Plain TCP listeners continue to carry TLS without inspecting it. The initial delivery can ship TCP/UDP routing and health checks before TLS listeners; the TLS API remains part of this enhancement. -Each selected location receives its own IP. Multiple edge nodes may announce -that IP within the location. Omitting `locations` defaults to +Each selected location receives one IP per requested family. Multiple edge +nodes may announce each IP within the location. An address request sets +`families` to one or both of `IPv4` and `IPv6`. The list is required, unique, +and order-independent, so enabling IPv4 +later never changes an existing request implicitly. The listener references +the request once for both families. Omitting `locations` defaults to `locations.from: Backends`: an address request follows the locations of the `NetworkService` backends on listeners using that address. `All` explicitly selects every eligible platform location; `Names` selects only listed -locations. Adding a backend location creates another IP without changing -existing location addresses. +locations. Adding a backend location creates an IP for each family without +changing existing location addresses. Separate requests remain available when +families need different location selections or independent listener attachment. For an address shared by several listeners or SNI routes, `Backends` selects a location only when every positive-weight backend service on the effective @@ -178,7 +183,7 @@ spec: exposure: addresses: - name: regional - family: IPv6 + families: [IPv6] listeners: - name: game protocol: UDP @@ -205,7 +210,9 @@ spec: The address request omits `locations`, so it follows the `game` service's selected interfaces. `locations: {from: Backends}` expresses the same choice -explicitly. +explicitly. Once IPv4 ingress is available, changing the request to +`families: [IPv4, IPv6]` publishes both families on the same listener and +route. `backends` accepts multiple entries with an optional `weight` (default 1, zero disables selection). Weights split new connections or flows, not packets @@ -243,13 +250,17 @@ status: status: "True" ``` -These IPs are illustrative. The first NLB release accepts IPv6 address -requests because Galactic's current public ingress packet path handles IPv6. +These IPs are illustrative. Status keys each allocated address by request, +location, and family. Allocation and announcement are reported independently +for each family; a missing IPv4 allocation does not hide a working IPv6 +address, but the request reports partial readiness until both are available. +The first NLB release accepts only `families: [IPv6]` because Galactic's +current public ingress packet path handles IPv6. The cell intent carries the public VIP family and each backend address family -separately. IPv4 requests remain rejected until the IPv4 ingress path and its -capability checks are ready. An allocation remains in status when a route or -backend is unavailable; its announcement state changes instead. Load balancer -conditions distinguish +separately. A request containing IPv4 remains rejected until the IPv4 ingress +path and its capability checks are ready. An allocation remains in status when +a route or backend is unavailable; its announcement state changes instead. +Load balancer conditions distinguish allocation and programming. Route conditions report attachment, backend resolution, and programming. `Ready` means at least one selected location serves traffic; location status identifies any that do not. @@ -302,8 +313,8 @@ Rollout requires a public IPv4 IPAM class, family-aware cell and Galactic programming, and a capability signal from every serving gateway in a location. CI must exercise IPv4 TCP and UDP ingress to IPv6 backends and replies, two VIPs targeting the same backend port, backend health changes, and announcement -withdrawal. Only then should admission accept IPv4 and allow a listener to -reference both IPv4 and IPv6 address requests. +withdrawal. Only then should admission accept `families` containing IPv4 and +allow one address request to publish both families on a listener. ### Backend Health From 84053a8d3e7736053c9e4c7ca6daa15ff50d34ee Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 16 Oct 2026 15:28:26 -0500 Subject: [PATCH 12/25] docs: issue NLB hostname per location --- docs/enhancements/network-load-balancers.md | 131 +++++++++++++------- 1 file changed, 86 insertions(+), 45 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index ac4d16d2..30e42d47 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -30,8 +30,9 @@ latest-milestone: "v0.x" A project can publish a `NetworkService` on a public TCP or UDP port through a `NetworkLoadBalancer`. The load balancer owns listeners and public address requests; a separate `NetworkLoadBalancerRoute` connects a listener to the -service. Each selected location receives a distinct public IP. Status shows -the allocated IP and announcement state for each location. By default, those +service. Each selected location receives an IP per requested family and a +generated hostname users can target with a CNAME. Status shows the hostname, +allocated IPs, and announcement state for each location. By default, those locations follow the network interfaces selected by the backend services. Active health checks keep failed backends out of rotation. TLS listeners can route by SNI and either pass encryption to the backend or terminate it with a @@ -53,6 +54,8 @@ where their service runs. termination for non-HTTP applications. - Allocate a distinct public address in every backend location by default, including locations added after the address request is created. +- Issue a stable, platform-managed hostname in each selected location so users + can point a DNS CNAME at that location's NLB endpoint. - Show allocated addresses and serving locations without releasing an address when its backends are temporarily unavailable. - Use an address request shape that can later serve Application Load Balancers. @@ -91,8 +94,8 @@ TLS API remains part of this enhancement. Each selected location receives one IP per requested family. Multiple edge nodes may announce each IP within the location. An address request sets `families` to one or both of `IPv4` and `IPv6`. The list is required, unique, -and order-independent, so enabling IPv4 -later never changes an existing request implicitly. The listener references +and order-independent, so enabling IPv4 later never changes an existing +request implicitly. The listener references the request once for both families. Omitting `locations` defaults to `locations.from: Backends`: an address request follows the locations of the `NetworkService` backends on listeners using that address. `All` explicitly @@ -101,6 +104,17 @@ locations. Adding a backend location creates an IP for each family without changing existing location addresses. Separate requests remain available when families need different location selections or independent listener attachment. +Each address request produces one stable generated hostname per selected +location, reported as `status.endpoints[].canonicalHostname` and shared by +its IPv4 and IPv6 addresses. Its name is derived from the load balancer +identity, request name, and location under a Datum-managed zone; +it is independent of the current IPs. Consumers can CNAME a location-specific +name they own to it. Datum publishes an A or AAAA record only when that +family's address is allocated and announced in the location. The hostname +stays assigned during temporary health loss, even when it has no address +records. Different requests in the same location receive different hostnames. +DNS names do not carry a port; clients still connect to the listener's port. + For an address shared by several listeners or SNI routes, `Backends` selects a location only when every positive-weight backend service on the effective routes has a selected, bound network interface there. This keeps every route @@ -116,6 +130,8 @@ route, placement is pending and no location IP is allocated. maintaining a location list. - The operator adds a new region while existing regional client endpoints keep their addresses. +- The operator CNAMEs `us.game.example.com` to the generated hostname for the + US location and `eu.game.example.com` to the EU location. - If a region loses its serving backends, its IP remains allocated while its announcement is withdrawn. - An operator exposes a TLS application through a TCP listener; the backend @@ -134,8 +150,12 @@ route, placement is pending and no location IP is allocated. - **Reusing a public IP for another load balancer misroutes traffic.** Allocate every location address from a globally unique public pool. - **A `Backends` or `All` request can expand over time.** Expose each added - location and IP in status, and bound the number of locations and addresses - per load balancer. + location, hostname, and IP in status, and bound the number of locations and + addresses per load balancer. +- **DNS could direct users to an unannounced IP.** Publish A and AAAA records + only after the corresponding VIP is announced; withdraw the DNS record when + that announcement is withdrawn. Short TTLs limit stale answers; clients with + cached IPs can still fail after withdrawal. - **Overlapping routes could select different backends for the same flow.** Use Gateway API precedence for overlapping route attachments and report the effective route in native status. @@ -221,7 +241,8 @@ weight and its share is rejected, as Gateway API requires. Failed health probes exclude individual endpoints while healthy endpoints remain. Status distinguishes resolution, endpoint availability, and health. -The load balancer reports one entry per requested location and allocated IP: +The load balancer reports one endpoint per address request and location. Each +endpoint groups its generated hostname and addresses by family: ```yaml status: @@ -229,47 +250,56 @@ status: - name: regional locationSource: Backends resolvedLocations: [us-central-1, eu-west-1] - addresses: + endpoints: - requestName: regional location: us-central-1 - family: IPv6 - value: 2001:db8:100::10 - conditions: - - type: Allocated - status: "True" - - type: Announced - status: "True" + canonicalHostname: nlb-7f2.regional.us-central-1.datumproxy.net + addresses: + - family: IPv6 + value: 2001:db8:100::10 + conditions: + - type: Allocated + status: "True" + - type: Announced + status: "True" + - type: DNSPublished + status: "True" - requestName: regional location: eu-west-1 - family: IPv6 - value: 2001:db8:200::11 - conditions: - - type: Allocated - status: "True" - - type: Announced - status: "True" + canonicalHostname: nlb-7f2.regional.eu-west-1.datumproxy.net + addresses: + - family: IPv6 + value: 2001:db8:200::11 + conditions: + - type: Allocated + status: "True" + - type: Announced + status: "True" + - type: DNSPublished + status: "True" ``` -These IPs are illustrative. Status keys each allocated address by request, -location, and family. Allocation and announcement are reported independently -for each family; a missing IPv4 allocation does not hide a working IPv6 -address, but the request reports partial readiness until both are available. -The first NLB release accepts only `families: [IPv6]` because Galactic's -current public ingress packet path handles IPv6. -The cell intent carries the public VIP family and each backend address family -separately. A request containing IPv4 remains rejected until the IPv4 ingress +These names and IPs are illustrative. Endpoint identity is the address request +and location; each requested family has an address entry even while allocation +is pending. Allocation, announcement, and DNS publication are reported +independently for each family. A missing IPv4 allocation does not hide a +working IPv6 address, but the request reports partial readiness until both +are available. The first NLB release accepts only `families: [IPv6]` because +Galactic's current public ingress packet path handles IPv6. The cell intent +carries the public VIP family and each backend address family separately. A +request containing IPv4 remains rejected until the IPv4 ingress path and its capability checks are ready. An allocation remains in status when a route or backend is unavailable; its announcement state changes instead. -Load balancer conditions distinguish -allocation and programming. Route conditions report attachment, backend -resolution, and programming. `Ready` means at least one -selected location serves traffic; location status identifies any that do not. -An address is announced from a location only when every listener using it is -ready there. Status reports the resolved locations for each address request -and shows `Backends` as the source when `locations` is omitted. Health loss -withdraws an announcement without releasing its IP; after the last selected -interface leaves a location, withdraw that location's announcement before -releasing its IP. +Load balancer conditions distinguish allocation, programming, and DNS +publication. Route conditions report attachment, backend resolution, and +programming. `Ready` means at least one selected location has a serving IP +published under its generated hostname; location status identifies any that +do not. An address is announced from a location only when every listener using +it is ready there. Status reports the resolved locations for each address +request and shows `Backends` as the source when `locations` is omitted. Health loss +withdraws an announcement and its DNS record without releasing its IP; after +the last selected interface leaves a location, withdraw its DNS records and +announcement before releasing its IP. NSO resolves service membership, allocates IPs, selects eligible locations, and reports status. Cell controllers turn resolved intent into Galactic @@ -396,6 +426,13 @@ uses `tls.mode: Passthrough`, omits the certificate, and keeps end-to-end TLS; its routes do not set backend transport. Only visible SNI can be matched; connections without a matching visible name use the default or are rejected. +A CNAME to the generated location hostname changes DNS resolution, not the +client's TLS SNI. For TLS passthrough or termination on a customer's hostname, +that hostname must also appear in `sniHostnames`; termination needs a covering +certificate, while passthrough needs one on the backend. A generated hostname +does not automatically authorize or issue a certificate for the customer's +name. + SNI inspection and TLS termination need a proxy-capable data plane; Galactic's current DSR rule only forwards TCP and cannot provide either. HTTP routing and termination remain with `HTTPProxy`. A terminating proxy opens a new backend @@ -477,16 +514,17 @@ Start with single-location TCP and UDP with weighted backends and active probes, then test multi-location operation with distinct addresses. Add SNI passthrough before managed TLS termination. Exercise route updates, UDP flow stability, probe failures, loss of the last backend, location removal, TLS certificate -issuance and rotation, and withdrawal before IP release. Existing +issuance and rotation, CNAME resolution, A/AAAA publication, IP replacement +without hostname change, and withdrawal before IP release. Existing `NetworkService` and `HTTPProxy` behavior should be unaffected. ### Monitoring Requirements Operators need counts of allocated VIPs, programmed rules, healthy backends, -and announced locations, plus failures to allocate, probe, program, issue -certificates, or withdraw. Consumers read load balancer and route conditions, -backend health, TLS hostname and certificate status, and per-address -announcement status. +announced locations, and published DNS records, plus failures to allocate, +probe, program, publish DNS, issue certificates, or withdraw. Consumers read +load balancer and route conditions, backend health, generated location +hostnames, TLS certificate status, and per-address announcement and DNS status. ### Dependencies @@ -501,6 +539,9 @@ Galactic IPv4 ingress and cross-family proxy support, and a serving capability gate before API admission is enabled. SNI routing and managed TLS need a proxy-capable edge path, hostname ownership verification, and certificate issuance and distribution. +NSO needs a managed DNS zone and per-location A/AAAA record lifecycle tied to +VIP allocation and announcement; hostname assignment must remain stable across +IP replacement and temporary health loss. Gateway API parity also needs an attachment/grant model, Service backend translation, Secret certificate references, weighted selection, wildcard SNI, mixed TLS modes, backend TLS validation, per-parent status, and conformance From 4cf1bec0a4341f4a7c21bcb9901132752689ca92 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 16 Oct 2026 15:31:05 -0500 Subject: [PATCH 13/25] docs: shorten generated NLB hostnames --- docs/enhancements/network-load-balancers.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index 30e42d47..f0160413 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -106,10 +106,12 @@ families need different location selections or independent listener attachment. Each address request produces one stable generated hostname per selected location, reported as `status.endpoints[].canonicalHostname` and shared by -its IPv4 and IPv6 addresses. Its name is derived from the load balancer -identity, request name, and location under a Datum-managed zone; -it is independent of the current IPs. Consumers can CNAME a location-specific -name they own to it. Datum publishes an A or AAAA record only when that +its IPv4 and IPv6 addresses. Datum assigns each address request an opaque +endpoint ID unique across the platform. The hostname is +`..`, with a recognizable location and no +request name or IP embedded in it. The endpoint ID is reused across locations +and stays stable while the request exists. Consumers can CNAME a name they own +for that location to it. Datum publishes an A or AAAA record only when that family's address is allocated and announced in the location. The hostname stays assigned during temporary health loss, even when it has no address records. Different requests in the same location receive different hostnames. @@ -253,7 +255,7 @@ status: endpoints: - requestName: regional location: us-central-1 - canonicalHostname: nlb-7f2.regional.us-central-1.datumproxy.net + canonicalHostname: n-7f2k9q.us-central-1.datumproxy.net addresses: - family: IPv6 value: 2001:db8:100::10 @@ -266,7 +268,7 @@ status: status: "True" - requestName: regional location: eu-west-1 - canonicalHostname: nlb-7f2.regional.eu-west-1.datumproxy.net + canonicalHostname: n-7f2k9q.eu-west-1.datumproxy.net addresses: - family: IPv6 value: 2001:db8:200::11 From ab4b38c0061a7634ef913c429c9380a265582e2c Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 16 Oct 2026 15:57:24 -0500 Subject: [PATCH 14/25] docs: align NLB hostnames with proxy format --- docs/enhancements/network-load-balancers.md | 23 ++++++++++++--------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index f0160413..9017f716 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -107,14 +107,17 @@ families need different location selections or independent listener attachment. Each address request produces one stable generated hostname per selected location, reported as `status.endpoints[].canonicalHostname` and shared by its IPv4 and IPv6 addresses. Datum assigns each address request an opaque -endpoint ID unique across the platform. The hostname is -`..`, with a recognizable location and no -request name or IP embedded in it. The endpoint ID is reused across locations -and stays stable while the request exists. Consumers can CNAME a name they own -for that location to it. Datum publishes an A or AAAA record only when that -family's address is allocated and announced in the location. The hostname -stays assigned during temporary health loss, even when it has no address -records. Different requests in the same location receive different hostnames. +endpoint label using the proxy format: two words and five hash-derived +characters. The label is derived from the load balancer UID and request name; +the hostname is `..`. The label is +reused across locations, does not expose the request name or IP, and stays +stable while the request exists. NSO claims each generated hostname uniquely +and retains existing names if the generation algorithm changes. Consumers can +CNAME a name they own for that location to it. Datum publishes an A or AAAA +record only when that family's address is allocated and announced in the +location. The hostname stays assigned during temporary health loss, even when +it has no address records. Different requests in the same location receive +different hostnames. DNS names do not carry a port; clients still connect to the listener's port. For an address shared by several listeners or SNI routes, `Backends` selects a @@ -255,7 +258,7 @@ status: endpoints: - requestName: regional location: us-central-1 - canonicalHostname: n-7f2k9q.us-central-1.datumproxy.net + canonicalHostname: us-central-1.crown-raven-m7p6g.datumproxy.net addresses: - family: IPv6 value: 2001:db8:100::10 @@ -268,7 +271,7 @@ status: status: "True" - requestName: regional location: eu-west-1 - canonicalHostname: n-7f2k9q.eu-west-1.datumproxy.net + canonicalHostname: eu-west-1.crown-raven-m7p6g.datumproxy.net addresses: - family: IPv6 value: 2001:db8:200::11 From 7553b71c3881d84ac75c1c3cf1ef4506b908fb9f Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 16 Oct 2026 16:00:08 -0500 Subject: [PATCH 15/25] docs: derive NLB network from backends --- docs/enhancements/network-load-balancers.md | 33 +++++++++++++-------- 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index 9017f716..d3eacb60 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -75,15 +75,25 @@ where their service runs. ## Proposal -A `NetworkLoadBalancer` names a network, requests addresses, and defines TCP, -UDP, or TLS listeners. Each listener names the address requests on which it -serves. A `NetworkLoadBalancerRoute` attaches to a named listener and points to +A `NetworkLoadBalancer` requests addresses and defines TCP, UDP, or TLS +listeners. Each listener names the address requests on which it serves. A +`NetworkLoadBalancerRoute` attaches to a named listener and points to named ports on `NetworkService` backends. TCP and UDP listeners have one effective catch-all route each. If several attach, the oldest wins, breaking timestamp ties by namespace/name; all attachments remain Accepted, and native status identifies the effective route. Each route can distribute new connections or flows across weighted backends. +The NLB has no required VPC or `Network` reference. Its public endpoint lives +at the edge; backend interfaces belong to an NSO `Network`, which maps to the +fabric VPC in each location. NSO derives that network from the selected +interfaces of every positive-weight backend service on the NLB's effective +routes. All must resolve to the same network. Until they do, the NLB reports +`NetworkResolved=False` with `NoBackends` or `MultipleNetworks` and programs no +traffic. A network mismatch withdraws any existing announcements while +retaining allocated IPs. This lets users create the NLB before its workloads +without repeating network identity in the exposure API. + A TLS listener adds `tls.mode: Passthrough` or `Terminate`. Routes attached to it can match SNI hostnames so several TLS services share an address and port. Terminate mode uses a Datum-managed certificate or a referenced TLS Secret. @@ -203,8 +213,6 @@ kind: NetworkLoadBalancer metadata: name: game spec: - networkRef: - name: default exposure: addresses: - name: regional @@ -295,12 +303,12 @@ carries the public VIP family and each backend address family separately. A request containing IPv4 remains rejected until the IPv4 ingress path and its capability checks are ready. An allocation remains in status when a route or backend is unavailable; its announcement state changes instead. -Load balancer conditions distinguish allocation, programming, and DNS -publication. Route conditions report attachment, backend resolution, and -programming. `Ready` means at least one selected location has a serving IP -published under its generated hostname; location status identifies any that -do not. An address is announced from a location only when every listener using -it is ready there. Status reports the resolved locations for each address +Load balancer conditions distinguish network resolution, allocation, +programming, and DNS publication. Route conditions report attachment, backend +resolution, and programming. `Ready` means at least one selected location has +a serving IP published under its generated hostname; location status identifies +any that do not. An address is announced from a location only when every +listener using it is ready there. Status reports the resolved locations for each address request and shows `Backends` as the source when `locations` is omitted. Health loss withdraws an announcement and its DNS record without releasing its IP; after the last selected interface leaves a location, withdraw its DNS records and @@ -312,7 +320,8 @@ and reports status. Cell controllers turn resolved intent into Galactic backends in the ingress location. The control plane allocates one IP per selected location and shares that location's IP among its serving cells. It resolves `Backends` from actual membership, not the capped -`NetworkService.status.locations` display list. +`NetworkService.status.locations` display list. Network resolution likewise +uses actual interfaces rather than inferring identity from the display status. For UDP, packets from the same client address and port select the same backend while the backend set is unchanged. A backend change may move that flow; the From 309cbe70c9815bd2c9d4bb0ea53cee91679d21bd Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 16 Oct 2026 16:01:47 -0500 Subject: [PATCH 16/25] docs: scope NLBs to an explicit network --- docs/enhancements/network-load-balancers.md | 40 +++++++++++++-------- 1 file changed, 25 insertions(+), 15 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index d3eacb60..68fcdf84 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -75,24 +75,27 @@ where their service runs. ## Proposal -A `NetworkLoadBalancer` requests addresses and defines TCP, UDP, or TLS -listeners. Each listener names the address requests on which it serves. A -`NetworkLoadBalancerRoute` attaches to a named listener and points to +A `NetworkLoadBalancer` names an NSO `Network`, requests addresses, and defines +TCP, UDP, or TLS listeners. Each listener names the address requests on which +it serves. A `NetworkLoadBalancerRoute` attaches to a named listener and points to named ports on `NetworkService` backends. TCP and UDP listeners have one effective catch-all route each. If several attach, the oldest wins, breaking timestamp ties by namespace/name; all attachments remain Accepted, and native status identifies the effective route. Each route can distribute new connections or flows across weighted backends. -The NLB has no required VPC or `Network` reference. Its public endpoint lives -at the edge; backend interfaces belong to an NSO `Network`, which maps to the -fabric VPC in each location. NSO derives that network from the selected -interfaces of every positive-weight backend service on the NLB's effective -routes. All must resolve to the same network. Until they do, the NLB reports -`NetworkResolved=False` with `NoBackends` or `MultipleNetworks` and programs no -traffic. A network mismatch withdraws any existing announcements while -retaining allocated IPs. This lets users create the NLB before its workloads -without repeating network identity in the exposure API. +`spec.networkRef` is required and immutable. It names the NSO `Network` whose +backends this NLB exposes; the platform resolves that network to its fabric +VPC in each serving location. The public IP is an edge address, not a VPC +address. A stable network reference lets users create the NLB before any +backend exists and prevents changing service membership from silently moving +its traffic to another network. Every effective route backend must resolve to +interfaces on that network. A backend with no members cannot serve until +members appear. One resolving to a different network sets +`BackendResolved=False` with reason `NetworkMismatch` and invalidates its +route. Existing public IP allocations are retained; listener readiness still +controls announcements. Cross-network backends would need an explicit future +forwarding and authorization design. A TLS listener adds `tls.mode: Passthrough` or `Terminate`. Routes attached to it can match SNI hostnames so several TLS services share an address and port. @@ -213,6 +216,8 @@ kind: NetworkLoadBalancer metadata: name: game spec: + networkRef: + name: default exposure: addresses: - name: regional @@ -303,7 +308,7 @@ carries the public VIP family and each backend address family separately. A request containing IPv4 remains rejected until the IPv4 ingress path and its capability checks are ready. An allocation remains in status when a route or backend is unavailable; its announcement state changes instead. -Load balancer conditions distinguish network resolution, allocation, +Load balancer conditions distinguish network reference resolution, allocation, programming, and DNS publication. Route conditions report attachment, backend resolution, and programming. `Ready` means at least one selected location has a serving IP published under its generated hostname; location status identifies @@ -320,8 +325,10 @@ and reports status. Cell controllers turn resolved intent into Galactic backends in the ingress location. The control plane allocates one IP per selected location and shares that location's IP among its serving cells. It resolves `Backends` from actual membership, not the capped -`NetworkService.status.locations` display list. Network resolution likewise -uses actual interfaces rather than inferring identity from the display status. +`NetworkService.status.locations` display list. Route backend validation also +uses actual interfaces rather than inferring network identity from display +status. NSO creates a `NetworkBinding` for the referenced network in each +selected location before programming its cell resources. For UDP, packets from the same client address and port select the same backend while the backend set is unchanged. A backend change may move that flow; the @@ -592,6 +599,9 @@ withdrawal across the cells serving its location. membership to public address and listener lifecycle. - **Put the backend on each listener.** This is compact for one backend but leaves no independent route lifecycle or place for future route policy. +- **Infer the NLB network from backend membership.** This avoids a reference on + the NLB but makes its network identity depend on changing workload membership + and leaves it undefined before backends exist. - **Expose Gateway API directly.** It adds concepts consumers do not need for this use case and does not express Datum's location and address policies. Offer an adapter for portability while retaining the Datum API. From f3b40fce40df0fa1615608c222540549e6f70fa7 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 16 Oct 2026 16:08:26 -0500 Subject: [PATCH 17/25] docs: defer managed NLB certificates --- docs/enhancements/network-load-balancers.md | 63 ++++++++++----------- 1 file changed, 30 insertions(+), 33 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index 68fcdf84..edd47fcd 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -36,7 +36,7 @@ allocated IPs, and announcement state for each location. By default, those locations follow the network interfaces selected by the backend services. Active health checks keep failed backends out of rotation. TLS listeners can route by SNI and either pass encryption to the backend or terminate it with a -Datum-managed certificate. +referenced TLS Secret. ## Motivation @@ -50,8 +50,8 @@ where their service runs. - Publish a TCP or UDP `NetworkService` through a named listener and separate route. - Actively check backend health before serving traffic. -- Support opaque TLS over TCP, SNI-based TLS passthrough, and managed TLS - termination for non-HTTP applications. +- Support opaque TLS over TCP, SNI-based TLS passthrough, and TLS termination + with a referenced certificate for non-HTTP applications. - Allocate a distinct public address in every backend location by default, including locations added after the address request is created. - Issue a stable, platform-managed hostname in each selected location so users @@ -67,6 +67,7 @@ where their service runs. ### Non-Goals - HTTP routing or changing the current `HTTPProxy` API. +- Datum-managed certificate issuance for NLB listeners in the first release. - Source-based matches or cross-location backend fallback in the first release. - Anycast or another single public IP shared across locations. A future Global Services Load Balancer will own global entry points and cross-location @@ -99,7 +100,7 @@ forwarding and authorization design. A TLS listener adds `tls.mode: Passthrough` or `Terminate`. Routes attached to it can match SNI hostnames so several TLS services share an address and port. -Terminate mode uses a Datum-managed certificate or a referenced TLS Secret. +Terminate mode uses a referenced TLS Secret. Plain TCP listeners continue to carry TLS without inspecting it. The initial delivery can ship TCP/UDP routing and health checks before TLS listeners; the TLS API remains part of this enhancement. @@ -156,8 +157,8 @@ route, placement is pending and no location IP is allocated. retains its certificate and the load balancer forwards the encrypted stream. - An operator routes `db.example.com` and `mq.example.com` on the same TLS port by SNI while each backend terminates its own TLS. -- An operator uses a Datum-managed certificate on a TLS listener, so a - non-HTTP backend can receive decrypted TCP without managing that certificate. +- An operator references a TLS Secret on a listener, so a non-HTTP backend can + receive decrypted TCP. ### Risks and Mitigations @@ -181,9 +182,9 @@ route, placement is pending and no location IP is allocated. separately from an unhealthy backend and avoid making one probe runner a single point of failure. - **A TLS hostname could be served with the wrong certificate or backend.** - Verify domain ownership before issuing managed certificates, require route - hostnames to be covered by the listener's certificate in Terminate mode, and - apply deterministic SNI precedence. + Require route hostnames to be covered by the referenced certificate in + Terminate mode, enforce Secret reference permissions, and apply deterministic + SNI precedence. - **One unready listener can affect other ports on its IP.** BGP announces an address, not a port. Announce an address from a location only when all of its listeners are ready there; use separate address requests for independent @@ -392,9 +393,8 @@ certificate. A TLS listener adds two explicit modes: - **Passthrough:** Inspect the ClientHello SNI only to choose a route, then forward the encrypted stream. The backend owns its certificate. - **Terminate:** Select a certificate and end client TLS at the load balancer. - `certificate.mode: Managed` uses Datum's domain verification and certificate - lifecycle. The route must explicitly choose plain TCP or re-encrypted TLS - for the backend connection. + Reference a TLS Secret holding the certificate. The route must explicitly + choose plain TCP or re-encrypted TLS for the backend connection. The proposed TLS-specific fields are: @@ -408,8 +408,9 @@ spec: tls: mode: Terminate certificate: - mode: Managed - hostnames: [db.example.com, mq.example.com] + mode: SecretRef + secretRef: + name: services-tls --- # NetworkLoadBalancerRoute spec fragment spec: @@ -429,17 +430,14 @@ spec: names use Gateway API hostname intersection and precedence rules with the listener's optional `hostname`. Each effective hostname belongs to one route on that listener; an omitted SNI match is the single default for clients -without a matching SNI. Without a -default, the listener rejects those connections. Terminate mode requires -verified hostnames covered by a ready certificate before serving them. Status -reports hostname verification and certificate readiness per listener and -location, separately from VIP allocation. A certificate failure withdraws -that listener's address from the affected location until it is ready; -unrelated addresses remain available. A no-SNI client needs an explicit -`certificate.defaultHostname` and a default route, or the connection is rejected. - -The terminating listener alternatively accepts -`certificate: {mode: SecretRef, secretRef: {name: ...}}`; cross-namespace +without a matching SNI. Without a default, the listener rejects those +connections. Terminate mode requires a +valid referenced certificate covering each served hostname. Status reports +certificate readiness per listener and location, separately from VIP +allocation. A certificate failure prevents that listener from serving and +withdraws its address where required by shared-address readiness; unrelated +addresses remain available. A no-SNI client needs a default route and uses the +referenced certificate, or the connection is rejected. Cross-namespace Secret references require an explicit grant. The terminated route chooses backend transport explicitly: plain `TCP` as shown, or `TLS` with a server name and certificate validation (system roots or referenced CA bundle). Passthrough @@ -476,7 +474,7 @@ tests and documenting any implementation-specific backend mapping. | Gateway addresses, named TCP/UDP/TLS listeners, ports and TLS mode | `NetworkLoadBalancer.exposure.addresses` and `listeners`; Datum additionally selects locations and allocates one address in each. | | TCPRoute and UDPRoute, listener attachment, backendRefs and weights | `NetworkLoadBalancerRoute.parentRefs` and weighted `backends`; one effective catch-all per TCP/UDP listener, with Gateway API precedence. | | TLSRoute SNI hostnames, wildcard matching and precedence | TLS listener `hostname` and route `sniHostnames`; passthrough and termination, including distinct TLS listeners on one port when hostnames do not conflict. | -| TLS certificates and backend re-encryption | Listener managed certificate or Secret reference; route backend TLS with SNI and trust validation, corresponding to BackendTLSPolicy. | +| TLS certificates and backend re-encryption | Listener Secret reference; route backend TLS with SNI and trust validation, corresponding to BackendTLSPolicy. | | AllowedRoutes, parentRefs, cross-namespace ReferenceGrant and status | Explicit attachment policy and grants; report Accepted, ResolvedRefs and programming per attachment, plus listener conflicts and readiness. | | ListenerSet and multiple parent attachments | Adapter expands these into listeners and route attachments while preserving ownership, conflict resolution and status for each source object. | @@ -533,9 +531,9 @@ advertisements are gone. Start with single-location TCP and UDP with weighted backends and active probes, then test multi-location operation with distinct addresses. Add SNI passthrough -before managed TLS termination. Exercise route updates, UDP flow stability, +before Secret-backed TLS termination. Exercise route updates, UDP flow stability, probe failures, loss of the last backend, location removal, TLS certificate -issuance and rotation, CNAME resolution, A/AAAA publication, IP replacement +loading and rotation, CNAME resolution, A/AAAA publication, IP replacement without hostname change, and withdrawal before IP release. Existing `NetworkService` and `HTTPProxy` behavior should be unaffected. @@ -543,7 +541,7 @@ without hostname change, and withdrawal before IP release. Existing Operators need counts of allocated VIPs, programmed rules, healthy backends, announced locations, and published DNS records, plus failures to allocate, -probe, program, publish DNS, issue certificates, or withdraw. Consumers read +probe, program, publish DNS, load certificates, or withdraw. Consumers read load balancer and route conditions, backend health, generated location hostnames, TLS certificate status, and per-address announcement and DNS status. @@ -558,8 +556,8 @@ announcement state. IPv4 VIPs additionally need a public IPv4 pool, family-aware cell intent, Galactic IPv4 ingress and cross-family proxy support, and a serving capability gate before API admission is enabled. -SNI routing and managed TLS need a proxy-capable edge path, hostname ownership -verification, and certificate issuance and distribution. +SNI routing and TLS termination need a proxy-capable edge path and secure +Secret distribution with certificate validation. NSO needs a managed DNS zone and per-location A/AAAA record lifecycle tied to VIP allocation and announcement; hostname assignment must remain stable across IP replacement and temporary health loss. @@ -610,5 +608,4 @@ withdrawal across the cells serving its location. Public VIP pools, cell-level programming and status propagation, and a multi-location traffic test environment are required before rollout. TLS -listeners additionally need an edge proxy tier and managed certificate -distribution. +listeners additionally need an edge proxy tier and secure Secret distribution. From 2fffb6713aa7b8ae5d1dea32dfaff6ce620cbfaa Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 16 Oct 2026 16:09:51 -0500 Subject: [PATCH 18/25] docs: simplify NLB address status --- docs/enhancements/network-load-balancers.md | 36 +++++++++++---------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index edd47fcd..a2f8b1b7 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -261,7 +261,8 @@ exclude individual endpoints while healthy endpoints remain. Status distinguishes resolution, endpoint availability, and health. The load balancer reports one endpoint per address request and location. Each -endpoint groups its generated hostname and addresses by family: +endpoint groups its generated hostname and addresses by family. This example +shows the status shape once dual-stack requests are enabled: ```yaml status: @@ -274,34 +275,35 @@ status: location: us-central-1 canonicalHostname: us-central-1.crown-raven-m7p6g.datumproxy.net addresses: + - family: IPv4 + value: 192.0.2.10 + announced: true + dnsPublished: true - family: IPv6 value: 2001:db8:100::10 - conditions: - - type: Allocated - status: "True" - - type: Announced - status: "True" - - type: DNSPublished - status: "True" + announced: true + dnsPublished: true - requestName: regional location: eu-west-1 canonicalHostname: eu-west-1.crown-raven-m7p6g.datumproxy.net addresses: + - family: IPv4 + value: 198.51.100.11 + announced: true + dnsPublished: true - family: IPv6 value: 2001:db8:200::11 - conditions: - - type: Allocated - status: "True" - - type: Announced - status: "True" - - type: DNSPublished - status: "True" + announced: true + dnsPublished: true ``` These names and IPs are illustrative. Endpoint identity is the address request and location; each requested family has an address entry even while allocation -is pending. Allocation, announcement, and DNS publication are reported -independently for each family. A missing IPv4 allocation does not hide a +is pending. An absent `value` means allocation is pending; `announced` and +`dnsPublished` report the two later steps separately. A non-ready family can +include a short `reason` and `message` for its unmet step. Full Kubernetes +conditions remain on the load balancer and routes, not on every family address. +A missing IPv4 allocation does not hide a working IPv6 address, but the request reports partial readiness until both are available. The first NLB release accepts only `families: [IPv6]` because Galactic's current public ingress packet path handles IPv6. The cell intent From d0563278a0df82c10900853fe3ef95438b38bf0c Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 16 Oct 2026 16:11:31 -0500 Subject: [PATCH 19/25] docs: remove managed certificate references --- docs/enhancements/network-load-balancers.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index a2f8b1b7..2506e6b7 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -67,7 +67,6 @@ where their service runs. ### Non-Goals - HTTP routing or changing the current `HTTPProxy` API. -- Datum-managed certificate issuance for NLB listeners in the first release. - Source-based matches or cross-location backend fallback in the first release. - Anycast or another single public IP shared across locations. A future Global Services Load Balancer will own global entry points and cross-location @@ -409,10 +408,8 @@ spec: port: 443 tls: mode: Terminate - certificate: - mode: SecretRef - secretRef: - name: services-tls + certificateRef: + name: services-tls --- # NetworkLoadBalancerRoute spec fragment spec: @@ -450,9 +447,8 @@ connections without a matching visible name use the default or are rejected. A CNAME to the generated location hostname changes DNS resolution, not the client's TLS SNI. For TLS passthrough or termination on a customer's hostname, that hostname must also appear in `sniHostnames`; termination needs a covering -certificate, while passthrough needs one on the backend. A generated hostname -does not automatically authorize or issue a certificate for the customer's -name. +certificate in the referenced Secret, while passthrough needs one on the +backend. SNI inspection and TLS termination need a proxy-capable data plane; Galactic's current DSR rule only forwards TCP and cannot provide either. HTTP routing and From 3b3aab4251f5cb4f7c9b9758da610ad036aeed2e Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Wed, 21 Oct 2026 14:51:18 -0500 Subject: [PATCH 20/25] docs: use local round robin selection for NLB backends --- docs/enhancements/network-load-balancers.md | 118 +++++++++++--------- 1 file changed, 67 insertions(+), 51 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index 2506e6b7..d6003640 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -49,7 +49,8 @@ where their service runs. - Publish a TCP or UDP `NetworkService` through a named listener and separate route. -- Actively check backend health before serving traffic. +- Actively check backend health and select healthy endpoints round robin within + the ingress location. - Support opaque TLS over TCP, SNI-based TLS passthrough, and TLS termination with a referenced certificate for non-HTTP applications. - Allocate a distinct public address in every backend location by default, @@ -59,8 +60,8 @@ where their service runs. - Show allocated addresses and serving locations without releasing an address when its backends are temporarily unavailable. - Use an address request shape that can later serve Application Load Balancers. -- Cover Gateway API's TCPRoute, UDPRoute, and TLSRoute behavior, including its - optional L4 features, before claiming full L4 compatibility. +- Support unweighted TCPRoute, UDPRoute, and TLSRoute behavior through a + Gateway API adapter, and document unsupported optional features. - Leave room for common L4 traffic controls without requiring them in the first release. @@ -72,6 +73,7 @@ where their service runs. Services Load Balancer will own global entry points and cross-location service behavior. - Requiring consumers to create Gateway API or Galactic resources. +- Per-backend traffic weights. ## Proposal @@ -81,8 +83,9 @@ it serves. A `NetworkLoadBalancerRoute` attaches to a named listener and points named ports on `NetworkService` backends. TCP and UDP listeners have one effective catch-all route each. If several attach, the oldest wins, breaking timestamp ties by namespace/name; all attachments remain Accepted, and native -status identifies the effective route. Each route can distribute new -connections or flows across weighted backends. +status identifies the effective route. Each route selects a healthy endpoint +in the ingress location for each new TCP connection or UDP flow using round +robin. `spec.networkRef` is required and immutable. It names the NSO `Network` whose backends this NLB exposes; the platform resolves that network to its fabric @@ -134,9 +137,9 @@ different hostnames. DNS names do not carry a port; clients still connect to the listener's port. For an address shared by several listeners or SNI routes, `Backends` selects a -location only when every positive-weight backend service on the effective -routes has a selected, bound network interface there. This keeps every route -on the IP usable in the same locations. Consumers can use separate address +location only when every backend service on the effective routes has a +selected, bound network interface there. This keeps every route on the IP +usable in the same locations. Consumers can use separate address requests when services have different footprints. Interface presence selects locations; health and programming determine announcement. With no attached route, placement is pending and no location IP is allocated. @@ -252,12 +255,17 @@ explicitly. Once IPv4 ingress is available, changing the request to `families: [IPv4, IPv6]` publishes both families on the same listener and route. -`backends` accepts multiple entries with an optional `weight` (default 1, -zero disables selection). Weights split new connections or flows, not packets -within a flow. An unresolved backend or one with no endpoints retains its -weight and its share is rejected, as Gateway API requires. Failed health probes -exclude individual endpoints while healthy endpoints remain. Status -distinguishes resolution, endpoint availability, and health. +`backends` accepts multiple entries without traffic weights. In each serving +location, the route combines their healthy endpoints into one set and assigns +new TCP connections or UDP flows to those endpoints round robin. Each healthy +endpoint receives an equal turn, so a service with more healthy endpoints +receives more connections or flows. A backend with no healthy endpoints +contributes none; the route is unready in that location only when no healthy +endpoint remains across all its backends. Failed health probes remove +individual endpoints from the rotation. Status distinguishes resolution, +endpoint availability, and health. +An existing connection or flow must keep its chosen endpoint rather than +rotate on every packet. The load balancer reports one endpoint per address request and location. Each endpoint groups its generated hostname and addresses by family. This example @@ -332,9 +340,10 @@ uses actual interfaces rather than inferring network identity from display status. NSO creates a `NetworkBinding` for the referenced network in each selected location before programming its cell resources. -For UDP, packets from the same client address and port select the same backend -while the backend set is unchanged. A backend change may move that flow; the -first release offers no UDP session or draining guarantee. +For UDP, packets in the same client flow keep the endpoint chosen when that +flow began while the gateway retains its flow assignment. A backend change or +gateway failover may move that flow; the first release offers no UDP session +or draining guarantee. ### IPv4 Ingress Data Plane @@ -381,8 +390,8 @@ and can be set on the health check. An endpoint serves only when its workload reports `HolderAvailable` and its active probe passes. Probes run from each serving location. The cell publishes -one healthy backend set for all gateway nodes in that location so they make -consistent selections. A location with no healthy backends withdraws its +one healthy endpoint set for all gateway nodes in that location so they share +the same endpoint eligibility. A location with no healthy backends withdraws its announcement but keeps its IP allocation. Status reports healthy backend counts and probe failures by location. @@ -458,19 +467,19 @@ must be explicit to consumers. ### Gateway API L4 Compatibility -The Datum API is the primary consumer experience. Its L4 behavior must cover -[TCPRoute](https://gateway-api.sigs.k8s.io/guides/user-guides/tcp/), -[UDPRoute](https://gateway-api.sigs.k8s.io/guides/user-guides/udp/), -[TLSRoute](https://gateway-api.sigs.k8s.io/reference/api-types/tlsroute/), -and their Core and Extended features. A Gateway API adapter must translate -Kubernetes objects into the same internal intent, so users can choose either -API. Full compatibility means passing the relevant Gateway API conformance -tests and documenting any implementation-specific backend mapping. +The Datum API is the primary consumer experience. A Gateway API adapter can +translate unweighted [TCPRoute](https://gateway-api.sigs.k8s.io/guides/user-guides/tcp/), +[UDPRoute](https://gateway-api.sigs.k8s.io/guides/user-guides/udp/), and +[TLSRoute](https://gateway-api.sigs.k8s.io/reference/api-types/tlsroute/) +objects into the same internal intent. The adapter must report unsupported +optional features instead of silently changing their behavior. Claim only the +Gateway API features that pass the relevant conformance tests, and document +any implementation-specific backend mapping. | Gateway API capability | Datum API behavior | | --- | --- | | Gateway addresses, named TCP/UDP/TLS listeners, ports and TLS mode | `NetworkLoadBalancer.exposure.addresses` and `listeners`; Datum additionally selects locations and allocates one address in each. | -| TCPRoute and UDPRoute, listener attachment, backendRefs and weights | `NetworkLoadBalancerRoute.parentRefs` and weighted `backends`; one effective catch-all per TCP/UDP listener, with Gateway API precedence. | +| TCPRoute and UDPRoute, listener attachment and backendRefs | `NetworkLoadBalancerRoute.parentRefs` and unweighted `backends`; one effective catch-all per TCP/UDP listener, with Gateway API precedence. Weighted backendRefs are unsupported. | | TLSRoute SNI hostnames, wildcard matching and precedence | TLS listener `hostname` and route `sniHostnames`; passthrough and termination, including distinct TLS listeners on one port when hostnames do not conflict. | | TLS certificates and backend re-encryption | Listener Secret reference; route backend TLS with SNI and trust validation, corresponding to BackendTLSPolicy. | | AllowedRoutes, parentRefs, cross-namespace ReferenceGrant and status | Explicit attachment policy and grants; report Accepted, ResolvedRefs and programming per attachment, plus listener conflicts and readiness. | @@ -481,23 +490,25 @@ listeners or load balancers, with optional listener name or port selection. Listener attachment policy defaults to the same namespace; cross-namespace attachment and backend or certificate references require grants. The adapter maps Kubernetes Services and their numbered ports to resolved backends; -`NetworkService` remains the native backend type. It must preserve Gateway API -rules for invalid backend shares, conflicting listeners and routes, hostname -intersection, and per-parent status. TCP and UDP may share a port; TCP and TLS -may not. Datum-only source-CIDR and locality policies cannot -change the behavior of a Gateway API resource unless explicitly configured -through a Datum extension. - -The first delivery may expose a smaller native feature set, but it must not -claim Gateway API L4 compatibility until TCPRoute, UDPRoute, TLSRoute, their -L4 Extended features, and the adapter pass conformance. Track the target -against a pinned Gateway API release; the upstream API can evolve independently. +`NetworkService` remains the native backend type. The adapter reports +unsupported weighted backendRefs; it must preserve Gateway API rules for +conflicting listeners and routes, hostname intersection, and per-parent +status. TCP and UDP may share a port; TCP and TLS may not. Datum-only +source-CIDR and locality policies cannot change the behavior of a Gateway API +resource unless explicitly configured through a Datum extension. + +The first delivery may expose a smaller native feature set. It must not +claim support for any Gateway API L4 feature until the adapter passes that +feature's conformance tests. Full Extended-feature compatibility, including +weighted backendRefs, is outside this proposal. Track the supported feature +set against a pinned Gateway API release; the upstream API can evolve +independently. ### Route Controls The listener selects destination IP, protocol, and port. The route selects a backend for each TCP connection or UDP flow. The following controls are -candidates for later releases beyond Gateway API L4 parity: +candidates for later releases beyond the initial NLB scope: - **Priority failover:** Use a standby backend when the preferred one lacks enough healthy members, with an explicit threshold. @@ -512,9 +523,12 @@ candidates for later releases beyond Gateway API L4 parity: Connection draining, source-IP preservation, and PROXY protocol are delivery settings rather than route matches; they need their own data plane contracts. -Galactic's current rule has a backend list but no weights, route matches, or -priority policy. Weighted selection is required for Gateway API parity; -the other controls need implementation before the API accepts them. +Galactic currently uses a deterministic Maglev hash across backend endpoints. +Strict round-robin assignment of new connections or flows needs a different +selection mechanism plus affinity for subsequent packets. The design must +specify what happens when an anycast flow reaches a different gateway node; +per-gateway counters alone cannot preserve its chosen endpoint. Route matches +and priority policy need implementation before the API accepts them. ## Production Readiness Review Questionnaire @@ -527,9 +541,10 @@ advertisements are gone. ### Rollout, Upgrade and Rollback Planning -Start with single-location TCP and UDP with weighted backends and active probes, -then test multi-location operation with distinct addresses. Add SNI passthrough -before Secret-backed TLS termination. Exercise route updates, UDP flow stability, +Start with single-location TCP and UDP using round robin across healthy +endpoints and active probes, then test multi-location operation with distinct +addresses. Add SNI passthrough before Secret-backed TLS termination. Exercise +route updates, round-robin selection, affinity during a flow, gateway failover, probe failures, loss of the last backend, location removal, TLS certificate loading and rotation, CNAME resolution, A/AAAA publication, IP replacement without hostname change, and withdrawal before IP release. Existing @@ -549,7 +564,8 @@ IPAM must provide globally unique public IPv6 VIPs, one for each selected location. NSO must allow UDP `NetworkService` ports and require the route listener and backend port protocols to match; `HTTPProxy` must continue to reject UDP backends. Selected cells need a health probe runner, Galactic -gateways, backend VIP bindings, and a way to report programming and +gateways, backend VIP bindings, round-robin flow selection with an explicit +affinity and gateway-failover contract, and a way to report programming and announcement state. IPv4 VIPs additionally need a public IPv4 pool, family-aware cell intent, Galactic IPv4 ingress and cross-family proxy support, and a serving capability @@ -559,10 +575,10 @@ Secret distribution with certificate validation. NSO needs a managed DNS zone and per-location A/AAAA record lifecycle tied to VIP allocation and announcement; hostname assignment must remain stable across IP replacement and temporary health loss. -Gateway API parity also needs an attachment/grant model, Service backend -translation, Secret certificate references, weighted selection, wildcard SNI, -mixed TLS modes, backend TLS validation, per-parent status, and conformance -tests against a pinned upstream release. +Gateway API support also needs an attachment/grant model, Service backend +translation, Secret certificate references, wildcard SNI, mixed TLS modes, +backend TLS validation, per-parent status, and conformance tests against a +pinned upstream release. Weighted backendRefs remain unsupported. ### Scalability From 32a4956f8379f45b74690707741f9ee840e3aa2a Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Wed, 21 Oct 2026 15:19:39 -0500 Subject: [PATCH 21/25] docs: scope NLB TLS to opaque TCP forwarding --- docs/enhancements/network-load-balancers.md | 269 ++++++++------------ 1 file changed, 102 insertions(+), 167 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index d6003640..ac219319 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -16,7 +16,7 @@ latest-milestone: "v0.x" - [Design Details](#design-details) - [IPv4 Ingress Data Plane](#ipv4-ingress-data-plane) - [Backend Health](#backend-health) - - [TLS](#tls) + - [TLS over TCP](#tls-over-tcp) - [Gateway API L4 Compatibility](#gateway-api-l4-compatibility) - [Route Controls](#route-controls) - [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) @@ -34,9 +34,9 @@ service. Each selected location receives an IP per requested family and a generated hostname users can target with a CNAME. Status shows the hostname, allocated IPs, and announcement state for each location. By default, those locations follow the network interfaces selected by the backend services. -Active health checks keep failed backends out of rotation. TLS listeners can -route by SNI and either pass encryption to the backend or terminate it with a -referenced TLS Secret. +Active health checks keep failed backends out of rotation. A TCP listener can +carry opaque TLS to a backend that owns its certificate; the NLB does not +inspect SNI or terminate TLS. ## Motivation @@ -51,8 +51,7 @@ where their service runs. route. - Actively check backend health and select healthy endpoints round robin within the ingress location. -- Support opaque TLS over TCP, SNI-based TLS passthrough, and TLS termination - with a referenced certificate for non-HTTP applications. +- Carry opaque TLS over a TCP listener to a backend that owns its certificate. - Allocate a distinct public address in every backend location by default, including locations added after the address request is created. - Issue a stable, platform-managed hostname in each selected location so users @@ -60,8 +59,8 @@ where their service runs. - Show allocated addresses and serving locations without releasing an address when its backends are temporarily unavailable. - Use an address request shape that can later serve Application Load Balancers. -- Support unweighted TCPRoute, UDPRoute, and TLSRoute behavior through a - Gateway API adapter, and document unsupported optional features. +- Allow a future Gateway API adapter to translate unweighted TCPRoute and + UDPRoute behavior, and document unsupported optional features. - Leave room for common L4 traffic controls without requiring them in the first release. @@ -74,11 +73,13 @@ where their service runs. service behavior. - Requiring consumers to create Gateway API or Galactic resources. - Per-backend traffic weights. +- SNI-based routing, TLS termination, backend TLS re-encryption, and TLSRoute. + These require a separate proxy-capable data plane and API proposal. ## Proposal A `NetworkLoadBalancer` names an NSO `Network`, requests addresses, and defines -TCP, UDP, or TLS listeners. Each listener names the address requests on which +TCP or UDP listeners. Each listener names the address requests on which it serves. A `NetworkLoadBalancerRoute` attaches to a named listener and points to named ports on `NetworkService` backends. TCP and UDP listeners have one effective catch-all route each. If several attach, the oldest wins, breaking @@ -100,12 +101,11 @@ route. Existing public IP allocations are retained; listener readiness still controls announcements. Cross-network backends would need an explicit future forwarding and authorization design. -A TLS listener adds `tls.mode: Passthrough` or `Terminate`. Routes attached to -it can match SNI hostnames so several TLS services share an address and port. -Terminate mode uses a referenced TLS Secret. -Plain TCP listeners continue to carry TLS without inspecting it. The initial -delivery can ship TCP/UDP routing and health checks before TLS listeners; the -TLS API remains part of this enhancement. +A TCP listener can forward TLS bytes unchanged to its selected backend, which +owns the certificate and TLS handshake. This does not add a TLS listener mode, +SNI route matching, certificate references, or backend transport settings to +the NLB API. Admission rejects those fields if supplied. Multiple TLS names +cannot select different backends on one TCP listener in this proposal. Each selected location receives one IP per requested family. Multiple edge nodes may announce each IP within the location. An address request sets @@ -136,7 +136,7 @@ it has no address records. Different requests in the same location receive different hostnames. DNS names do not carry a port; clients still connect to the listener's port. -For an address shared by several listeners or SNI routes, `Backends` selects a +For an address shared by several listeners, `Backends` selects a location only when every backend service on the effective routes has a selected, bound network interface there. This keeps every route on the IP usable in the same locations. Consumers can use separate address @@ -157,10 +157,6 @@ route, placement is pending and no location IP is allocated. announcement is withdrawn. - An operator exposes a TLS application through a TCP listener; the backend retains its certificate and the load balancer forwards the encrypted stream. -- An operator routes `db.example.com` and `mq.example.com` on the same TLS - port by SNI while each backend terminates its own TLS. -- An operator references a TLS Secret on a listener, so a non-HTTP backend can - receive decrypted TCP. ### Risks and Mitigations @@ -183,10 +179,6 @@ route, placement is pending and no location IP is allocated. - **Probe failure can withdraw a healthy location.** Report probe errors separately from an unhealthy backend and avoid making one probe runner a single point of failure. -- **A TLS hostname could be served with the wrong certificate or backend.** - Require route hostnames to be covered by the referenced certificate in - Terminate mode, enforce Secret reference permissions, and apply deterministic - SNI precedence. - **One unready listener can affect other ports on its IP.** BGP announces an address, not a port. Announce an address from a location only when all of its listeners are ready there; use separate address requests for independent @@ -312,12 +304,13 @@ include a short `reason` and `message` for its unmet step. Full Kubernetes conditions remain on the load balancer and routes, not on every family address. A missing IPv4 allocation does not hide a working IPv6 address, but the request reports partial readiness until both -are available. The first NLB release accepts only `families: [IPv6]` because -Galactic's current public ingress packet path handles IPv6. The cell intent -carries the public VIP family and each backend address family separately. A -request containing IPv4 remains rejected until the IPv4 ingress -path and its capability checks are ready. An allocation remains in status when -a route or backend is unavailable; its announcement state changes instead. +are available. Galactic can serve IPv4 VIPs over IPv6 backends when its IPv4 +VIP feature is enabled. NSO may still begin with `families: [IPv6]`; a request +containing IPv4 remains rejected until the serving cells have the required +Galactic version, feature gate, bindings, and public IPv4 pool. The cell intent +carries the public VIP family and each backend address family separately. An +allocation remains in status when a route or backend is unavailable; its +announcement state changes instead. Load balancer conditions distinguish network reference resolution, allocation, programming, and DNS publication. Route conditions report attachment, backend resolution, and programming. `Ready` means at least one selected location has @@ -330,10 +323,11 @@ the last selected interface leaves a location, withdraw its DNS records and announcement before releasing its IP. NSO resolves service membership, allocates IPs, selects eligible locations, -and reports status. Cell controllers turn resolved intent into Galactic -`NetworkRule` and `ServiceVIPBinding` objects. The initial route uses healthy -backends in the ingress location. The control plane allocates one IP per -selected location and shares that location's IP among its serving cells. It +and reports status. Cell controllers translate resolved intent into Galactic +`NetworkService` objects and their generated `ServiceVIPBinding` objects. The +initial route uses healthy backends in the ingress location. The control plane +allocates one IP per selected location and shares that location's IP among its +serving cells. It resolves `Backends` from actual membership, not the capped `NetworkService.status.locations` display list. Route backend validation also uses actual interfaces rather than inferring network identity from display @@ -347,34 +341,28 @@ or draining guarantee. ### IPv4 Ingress Data Plane -An IPv4 public VIP can front a service whose workload address is IPv6. The -ingress family and backend family are separate properties of cell intent. The -cell validates the VIP, every backend address, and their declared families -before programming traffic. An unsupported family pair remains unprogrammed -and unannounced, with a specific reason in status. IPv4 address requests stay -disabled in the consumer API until serving cells advertise this capability. - -The IPv4 gateway path needs an IPv4 listener and flow table for TCP and UDP. -Its flow identity includes the original destination VIP, protocol, port, and -client tuple, so two VIPs that target the same backend cannot merge into one -flow. For an IPv4 client and IPv6 backend, use a stateful L4 proxy: it accepts -the IPv4 connection or UDP flow, opens an IPv6 backend flow, and sends replies -through the same state. The current IPv6 DSR forwarding path cannot supply -this cross-family return path. The proxy contract must specify what source -address the backend sees; client address delivery, if offered, needs an -explicit mechanism and cannot be assumed from DSR behavior. - -The IPv4 path must handle IPv4 and IPv6 checksums, MTU and fragmentation, -ICMP errors, TCP resets and timeouts, and UDP flow expiry in both directions. -IPv4 VIPs receive distinct per-location /32 announcements, gated by healthy -backends and successful programming. Withdrawal completes before IPAM releases -an address. A location reports IPv4 and IPv6 programming and announcement -independently, so failure of one family does not hide the state of the other. - -Rollout requires a public IPv4 IPAM class, family-aware cell and Galactic -programming, and a capability signal from every serving gateway in a location. -CI must exercise IPv4 TCP and UDP ingress to IPv6 backends and replies, two -VIPs targeting the same backend port, backend health changes, and announcement +An IPv4 public VIP can front IPv6 workload addresses through Galactic's +existing direct-server-return path. With `GALACTIC_GATEWAY_IPV4_VIPS` enabled, +the gateway accepts IPv4 TCP/UDP, encapsulates the original packet toward an +IPv6 backend, and advertises the IPv4 VIP only after a matching backend +`ServiceVIPBinding` is Bound. The backend node translates the client's IPv4 +address into `64:ff9b::/96` for the workload and translates the reply back to +IPv4 from the VIP. This path does not require a stateful L4 proxy. The ingress +and backend address families remain separate properties of cell intent. + +Galactic currently limits an IPv4 VIP to one per service and each backend +socket to one VIP per address family. If two NLBs reuse the same backend socket +behind different VIPs, Galactic must first support those independent reply +translations; otherwise NSO rejects the conflicting placement. IPv4 VIPs +receive distinct per-location /32 announcements, gated by healthy backends +and successful programming. +Withdrawal completes before IPAM releases an address. A location reports +IPv4 and IPv6 programming and announcement independently. + +NSO still needs a public IPv4 pool, family-aware cell intent, rollout of the +Galactic IPv4 feature gate and backend bindings, and a capability signal from +every serving gateway in a location. CI must exercise IPv4 TCP and UDP ingress +to IPv6 backends and replies, backend health changes, and announcement withdrawal. Only then should admission accept `families` containing IPv4 and allow one address request to publish both families on a listener. @@ -395,114 +383,62 @@ the same endpoint eligibility. A location with no healthy backends withdraws its announcement but keeps its IP allocation. Status reports healthy backend counts and probe failures by location. -### TLS +### TLS over TCP -A TCP listener forwards TLS bytes unchanged to a backend that owns its -certificate. A TLS listener adds two explicit modes: +Galactic's current gateway forwards TCP packets without inspecting the payload. +The NLB can therefore carry an encrypted TLS connection as ordinary TCP to a +backend that presents its own certificate. The NLB neither validates that +certificate nor selects a backend by SNI. A route remains the one effective +catch-all route for its TCP listener. Clients use a hostname covered by the +backend certificate; a CNAME to a generated location hostname changes DNS +resolution but does not change the client's TLS SNI. The NLB does not manage +the backend certificate or report TLS-specific readiness. -- **Passthrough:** Inspect the ClientHello SNI only to choose a route, then - forward the encrypted stream. The backend owns its certificate. -- **Terminate:** Select a certificate and end client TLS at the load balancer. - Reference a TLS Secret holding the certificate. The route must explicitly - choose plain TCP or re-encrypted TLS for the backend connection. - -The proposed TLS-specific fields are: - -```yaml -# NetworkLoadBalancer spec fragment -spec: - listeners: - - name: secure - protocol: TLS - port: 443 - tls: - mode: Terminate - certificateRef: - name: services-tls ---- -# NetworkLoadBalancerRoute spec fragment -spec: - parentRefs: - - name: services - listener: secure - sniHostnames: [db.example.com] - backends: - - networkService: - name: database - port: database - transport: - protocol: TCP -``` - -`sniHostnames` is valid only on a TLS listener. Exact and `*.` wildcard DNS -names use Gateway API hostname intersection and precedence rules with the -listener's optional `hostname`. Each effective hostname belongs to one route -on that listener; an omitted SNI match is the single default for clients -without a matching SNI. Without a default, the listener rejects those -connections. Terminate mode requires a -valid referenced certificate covering each served hostname. Status reports -certificate readiness per listener and location, separately from VIP -allocation. A certificate failure prevents that listener from serving and -withdraws its address where required by shared-address readiness; unrelated -addresses remain available. A no-SNI client needs a default route and uses the -referenced certificate, or the connection is rejected. Cross-namespace Secret -references require an explicit grant. The terminated route chooses backend -transport explicitly: plain `TCP` as shown, or `TLS` with a server name and -certificate validation (system roots or referenced CA bundle). Passthrough -uses `tls.mode: Passthrough`, omits the certificate, and keeps end-to-end TLS; -its routes do not set backend transport. Only visible SNI can be matched; -connections without a matching visible name use the default or are rejected. - -A CNAME to the generated location hostname changes DNS resolution, not the -client's TLS SNI. For TLS passthrough or termination on a customer's hostname, -that hostname must also appear in `sniHostnames`; termination needs a covering -certificate in the referenced Secret, while passthrough needs one on the -backend. - -SNI inspection and TLS termination need a proxy-capable data plane; Galactic's -current DSR rule only forwards TCP and cannot provide either. HTTP routing and -termination remain with `HTTPProxy`. A terminating proxy opens a new backend -connection, so its source-IP behavior and optional PROXY protocol delivery -must be explicit to consumers. +Galactic's current XDP direct-server-return gateway has no TLS listener, +ClientHello inspection, certificate handling, or TLS proxy. SNI passthrough, +TLS termination, and proxy-originated TLS to the backend are outside this +proposal. A future design must choose a proxy-capable edge path, define +certificate distribution and validation, and specify source-IP and return-path +behavior. HTTP routing and termination remain with `HTTPProxy`. ### Gateway API L4 Compatibility -The Datum API is the primary consumer experience. A Gateway API adapter can -translate unweighted [TCPRoute](https://gateway-api.sigs.k8s.io/guides/user-guides/tcp/), -[UDPRoute](https://gateway-api.sigs.k8s.io/guides/user-guides/udp/), and -[TLSRoute](https://gateway-api.sigs.k8s.io/reference/api-types/tlsroute/) -objects into the same internal intent. The adapter must report unsupported -optional features instead of silently changing their behavior. Claim only the -Gateway API features that pass the relevant conformance tests, and document -any implementation-specific backend mapping. +The Datum API is the primary consumer experience. A future Gateway API adapter +can translate unweighted +[TCPRoute](https://gateway-api.sigs.k8s.io/guides/user-guides/tcp/) and +[UDPRoute](https://gateway-api.sigs.k8s.io/guides/user-guides/udp/) objects into +the same internal intent. It must report unsupported optional features instead +of silently changing their behavior. TLSRoute is unsupported, including when +the backend could terminate TLS itself: TLSRoute requires a TLS listener and +SNI routing semantics that this API does not provide. Claim only Gateway API +features that pass the relevant conformance tests, and document any +implementation-specific backend mapping. | Gateway API capability | Datum API behavior | | --- | --- | -| Gateway addresses, named TCP/UDP/TLS listeners, ports and TLS mode | `NetworkLoadBalancer.exposure.addresses` and `listeners`; Datum additionally selects locations and allocates one address in each. | +| Gateway addresses, named TCP/UDP listeners and ports | `NetworkLoadBalancer.exposure.addresses` and `listeners`; Datum additionally selects locations and allocates one address in each. TLS listeners are unsupported. | | TCPRoute and UDPRoute, listener attachment and backendRefs | `NetworkLoadBalancerRoute.parentRefs` and unweighted `backends`; one effective catch-all per TCP/UDP listener, with Gateway API precedence. Weighted backendRefs are unsupported. | -| TLSRoute SNI hostnames, wildcard matching and precedence | TLS listener `hostname` and route `sniHostnames`; passthrough and termination, including distinct TLS listeners on one port when hostnames do not conflict. | -| TLS certificates and backend re-encryption | Listener Secret reference; route backend TLS with SNI and trust validation, corresponding to BackendTLSPolicy. | -| AllowedRoutes, parentRefs, cross-namespace ReferenceGrant and status | Explicit attachment policy and grants; report Accepted, ResolvedRefs and programming per attachment, plus listener conflicts and readiness. | +| AllowedRoutes, parentRefs, cross-namespace ReferenceGrant and status | Explicit attachment policy and backend grants; report Accepted, ResolvedRefs and programming per attachment, plus listener conflicts and readiness. | | ListenerSet and multiple parent attachments | Adapter expands these into listeners and route attachments while preserving ownership, conflict resolution and status for each source object. | The native route needs multiple `parentRefs` so one route can attach to several listeners or load balancers, with optional listener name or port selection. Listener attachment policy defaults to the same namespace; cross-namespace -attachment and backend or certificate references require grants. The adapter -maps Kubernetes Services and their numbered ports to resolved backends; +attachment and backend references require grants. The adapter maps Kubernetes +Services and their numbered ports to resolved backends; `NetworkService` remains the native backend type. The adapter reports unsupported weighted backendRefs; it must preserve Gateway API rules for -conflicting listeners and routes, hostname intersection, and per-parent -status. TCP and UDP may share a port; TCP and TLS may not. Datum-only -source-CIDR and locality policies cannot change the behavior of a Gateway API +conflicting listeners and routes, and per-parent status. TCP and UDP may share +a port. Datum-only source-CIDR and locality policies cannot change the behavior +of a Gateway API resource unless explicitly configured through a Datum extension. The first delivery may expose a smaller native feature set. It must not claim support for any Gateway API L4 feature until the adapter passes that -feature's conformance tests. Full Extended-feature compatibility, including -weighted backendRefs, is outside this proposal. Track the supported feature -set against a pinned Gateway API release; the upstream API can evolve -independently. +feature's conformance tests. TLSRoute, BackendTLSPolicy, and full +Extended-feature compatibility, including weighted backendRefs, are outside +this proposal. Track the supported feature set against a pinned Gateway API +release; the upstream API can evolve independently. ### Route Controls @@ -543,20 +479,19 @@ advertisements are gone. Start with single-location TCP and UDP using round robin across healthy endpoints and active probes, then test multi-location operation with distinct -addresses. Add SNI passthrough before Secret-backed TLS termination. Exercise -route updates, round-robin selection, affinity during a flow, gateway failover, -probe failures, loss of the last backend, location removal, TLS certificate -loading and rotation, CNAME resolution, A/AAAA publication, IP replacement -without hostname change, and withdrawal before IP release. Existing -`NetworkService` and `HTTPProxy` behavior should be unaffected. +addresses. Exercise route updates, round-robin selection, affinity during a +flow, gateway failover, probe failures, loss of the last backend, location +removal, opaque TLS over a TCP listener, CNAME resolution, A/AAAA publication, +IP replacement without hostname change, and withdrawal before IP release. +Existing `NetworkService` and `HTTPProxy` behavior should be unaffected. ### Monitoring Requirements Operators need counts of allocated VIPs, programmed rules, healthy backends, announced locations, and published DNS records, plus failures to allocate, -probe, program, publish DNS, load certificates, or withdraw. Consumers read -load balancer and route conditions, backend health, generated location -hostnames, TLS certificate status, and per-address announcement and DNS status. +probe, program, publish DNS, or withdraw. Consumers read load balancer and route +conditions, backend health, generated location +hostnames, and per-address announcement and DNS status. ### Dependencies @@ -568,16 +503,13 @@ gateways, backend VIP bindings, round-robin flow selection with an explicit affinity and gateway-failover contract, and a way to report programming and announcement state. IPv4 VIPs additionally need a public IPv4 pool, family-aware cell intent, -Galactic IPv4 ingress and cross-family proxy support, and a serving capability -gate before API admission is enabled. -SNI routing and TLS termination need a proxy-capable edge path and secure -Secret distribution with certificate validation. +Galactic's existing IPv4 feature enabled with backend bindings, and a serving +capability gate before API admission is enabled. NSO needs a managed DNS zone and per-location A/AAAA record lifecycle tied to VIP allocation and announcement; hostname assignment must remain stable across IP replacement and temporary health loss. Gateway API support also needs an attachment/grant model, Service backend -translation, Secret certificate references, wildcard SNI, mixed TLS modes, -backend TLS validation, per-parent status, and conformance tests against a +translation, per-parent status, and conformance tests against a pinned upstream release. Weighted backendRefs remain unsupported. ### Scalability @@ -598,6 +530,10 @@ address that could not be allocated. - 2026-10-11: Initial proposal. - 2026-10-12: Scoped NLB addresses to one IP per selected location; deferred global entry points to a separate Global Services Load Balancer concept. +- 2026-10-21: Scoped TLS to opaque forwarding through a TCP listener; deferred + SNI routing, TLS termination, and backend re-encryption to a separate proxy + design; aligned IPv4 dependencies with Galactic's existing gated IPv4 VIP + path. ## Drawbacks @@ -621,5 +557,4 @@ withdrawal across the cells serving its location. ## Infrastructure Needed Public VIP pools, cell-level programming and status propagation, and a -multi-location traffic test environment are required before rollout. TLS -listeners additionally need an edge proxy tier and secure Secret distribution. +multi-location traffic test environment are required before rollout. From a8ce03a04975fca5d54fe39e9ce533adc5e1020a Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Wed, 21 Oct 2026 15:44:44 -0500 Subject: [PATCH 22/25] docs: preserve future NLB API behind capability gates --- docs/enhancements/network-load-balancers.md | 193 ++++++++++++++++---- 1 file changed, 154 insertions(+), 39 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index ac219319..e4f5bcac 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -16,7 +16,8 @@ latest-milestone: "v0.x" - [Design Details](#design-details) - [IPv4 Ingress Data Plane](#ipv4-ingress-data-plane) - [Backend Health](#backend-health) - - [TLS over TCP](#tls-over-tcp) + - [TLS](#tls) + - [Initial Delivery and Capability Gates](#initial-delivery-and-capability-gates) - [Gateway API L4 Compatibility](#gateway-api-l4-compatibility) - [Route Controls](#route-controls) - [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) @@ -34,9 +35,10 @@ service. Each selected location receives an IP per requested family and a generated hostname users can target with a CNAME. Status shows the hostname, allocated IPs, and announcement state for each location. By default, those locations follow the network interfaces selected by the backend services. -Active health checks keep failed backends out of rotation. A TCP listener can -carry opaque TLS to a backend that owns its certificate; the NLB does not -inspect SNI or terminate TLS. +Active health checks keep failed backends out of rotation. The API also models +TLS passthrough and termination so consumers need not replace their NLBs when +a proxy-capable data plane becomes available. The first delivery carries only +opaque TLS over TCP; it rejects TLS listeners until that data plane is ready. ## Motivation @@ -51,7 +53,11 @@ where their service runs. route. - Actively check backend health and select healthy endpoints round robin within the ingress location. -- Carry opaque TLS over a TCP listener to a backend that owns its certificate. +- Support opaque TLS over TCP initially, with an API shape for later SNI-based + passthrough, TLS termination, and optional backend TLS. +- Let a route refer to multiple `NetworkService` backends by named port, + including backends that use different port numbers. Allow one backend socket + to be reused behind distinct NLBs once Galactic supports it. - Allocate a distinct public address in every backend location by default, including locations added after the address request is created. - Issue a stable, platform-managed hostname in each selected location so users @@ -60,7 +66,8 @@ where their service runs. when its backends are temporarily unavailable. - Use an address request shape that can later serve Application Load Balancers. - Allow a future Gateway API adapter to translate unweighted TCPRoute and - UDPRoute behavior, and document unsupported optional features. + UDPRoute behavior, and later TLSRoute when TLS listeners are served. + Document unsupported optional features. - Leave room for common L4 traffic controls without requiring them in the first release. @@ -73,13 +80,13 @@ where their service runs. service behavior. - Requiring consumers to create Gateway API or Galactic resources. - Per-backend traffic weights. -- SNI-based routing, TLS termination, backend TLS re-encryption, and TLSRoute. - These require a separate proxy-capable data plane and API proposal. +- Serving SNI-based routing, TLS termination, backend TLS re-encryption, or + TLSRoute in the first delivery. These need a proxy-capable data plane. ## Proposal A `NetworkLoadBalancer` names an NSO `Network`, requests addresses, and defines -TCP or UDP listeners. Each listener names the address requests on which +TCP, UDP, or TLS listeners. Each listener names the address requests on which it serves. A `NetworkLoadBalancerRoute` attaches to a named listener and points to named ports on `NetworkService` backends. TCP and UDP listeners have one effective catch-all route each. If several attach, the oldest wins, breaking @@ -102,10 +109,11 @@ controls announcements. Cross-network backends would need an explicit future forwarding and authorization design. A TCP listener can forward TLS bytes unchanged to its selected backend, which -owns the certificate and TLS handshake. This does not add a TLS listener mode, -SNI route matching, certificate references, or backend transport settings to -the NLB API. Admission rejects those fields if supplied. Multiple TLS names -cannot select different backends on one TCP listener in this proposal. +owns the certificate and TLS handshake. A TLS listener adds `tls.mode` for +`Passthrough` or `Terminate`, with route `sniHostnames` and an explicit backend +transport for termination. These fields describe the intended long-term +consumer API; the first delivery rejects TLS listeners with a specific +unsupported-capability error rather than treating them as TCP. Each selected location receives one IP per requested family. Multiple edge nodes may announce each IP within the location. An address request sets @@ -157,6 +165,10 @@ route, placement is pending and no location IP is allocated. announcement is withdrawn. - An operator exposes a TLS application through a TCP listener; the backend retains its certificate and the load balancer forwards the encrypted stream. +- Once TLS listeners are enabled, an operator routes different SNI names on + one port to different services without replacing the NLB. +- Once termination is enabled, an operator references a TLS Secret so a + non-HTTP backend can receive plain TCP or validated TLS. ### Risks and Mitigations @@ -179,6 +191,10 @@ route, placement is pending and no location IP is allocated. - **Probe failure can withdraw a healthy location.** Report probe errors separately from an unhealthy backend and avoid making one probe runner a single point of failure. +- **A TLS hostname could be served with the wrong certificate or backend once + TLS listeners are enabled.** Require certificate coverage, enforce Secret + reference permissions, and apply deterministic SNI precedence before + lifting the TLS capability gate. - **One unready listener can affect other ports on its IP.** BGP announces an address, not a port. Announce an address from a location only when all of its listeners are ready there; use separate address requests for independent @@ -259,6 +275,16 @@ endpoint availability, and health. An existing connection or flow must keep its chosen endpoint rather than rotate on every packet. +Each backend entry resolves its named `NetworkService` port independently. +The consumer API does not require all entries to resolve to the same numeric +port: future Galactic intent must allow an independently configured destination +port for each endpoint. Likewise, one `NetworkService` may back several +listeners or NLBs; consumers do not duplicate it to fit Galactic's current +one-port rule. NSO materializes cell-local forwarding intent for each effective +listener, address request, and location. A single Galactic `NetworkService` can +carry the IPv4 and IPv6 VIPs for one such intent when both families are enabled. +The number of internal objects is not a consumer API constraint. + The load balancer reports one endpoint per address request and location. Each endpoint groups its generated hostname and addresses by family. This example shows the status shape once dual-stack requests are enabled: @@ -383,23 +409,100 @@ the same endpoint eligibility. A location with no healthy backends withdraws its announcement but keeps its IP allocation. Status reports healthy backend counts and probe failures by location. -### TLS over TCP +### TLS + +A TCP listener forwards TLS bytes unchanged to a backend that owns its +certificate. The NLB does not inspect SNI, validate that certificate, or +report TLS-specific readiness for this mode. A CNAME to the generated location +hostname changes DNS resolution, not the client's TLS SNI; the backend must +present a certificate covering the name the client uses. + +The consumer API also has a TLS listener with two modes: + +- **Passthrough:** Inspect the ClientHello SNI to choose a route, then forward + the encrypted stream. The backend owns its certificate. +- **Terminate:** Select a certificate from a referenced TLS Secret and end + client TLS at the load balancer. The route explicitly selects plain TCP or + re-encrypted TLS for the backend connection. -Galactic's current gateway forwards TCP packets without inspecting the payload. -The NLB can therefore carry an encrypted TLS connection as ordinary TCP to a -backend that presents its own certificate. The NLB neither validates that -certificate nor selects a backend by SNI. A route remains the one effective -catch-all route for its TCP listener. Clients use a hostname covered by the -backend certificate; a CNAME to a generated location hostname changes DNS -resolution but does not change the client's TLS SNI. The NLB does not manage -the backend certificate or report TLS-specific readiness. +The proposed TLS-specific fields are: + +```yaml +# NetworkLoadBalancer spec fragment +spec: + listeners: + - name: secure + protocol: TLS + port: 443 + tls: + mode: Terminate + certificateRef: + name: services-tls +--- +# NetworkLoadBalancerRoute spec fragment +spec: + parentRefs: + - name: services + listener: secure + sniHostnames: [db.example.com] + backends: + - networkService: + name: database + port: database + transport: + protocol: TCP +``` -Galactic's current XDP direct-server-return gateway has no TLS listener, -ClientHello inspection, certificate handling, or TLS proxy. SNI passthrough, -TLS termination, and proxy-originated TLS to the backend are outside this -proposal. A future design must choose a proxy-capable edge path, define -certificate distribution and validation, and specify source-IP and return-path -behavior. HTTP routing and termination remain with `HTTPProxy`. +`sniHostnames` is valid only on a TLS listener. Exact and `*.` wildcard DNS +names use Gateway API hostname intersection and precedence rules with the +listener's optional `hostname`. A route without an SNI match is the default +for clients without a matching visible name; without a default, the listener +rejects those connections. Terminate mode requires a certificate covering +each served hostname, and status reports certificate readiness separately +from VIP allocation. Cross-namespace Secret references require a grant. +Backend `transport: TLS` requires a server name and certificate validation +using system roots or a referenced CA bundle. Passthrough omits the listener +certificate and backend transport; the backend performs the TLS handshake. + +Galactic's current XDP direct-server-return gateway has no ClientHello +inspection, certificate handling, or TLS proxy. TLS listeners are therefore +rejected until a proxy-capable edge path is deployed with certificate +distribution, validation, source-IP, and return-path contracts. Opaque TLS +through a TCP listener is the only TLS behavior in the first delivery. HTTP +routing and termination remain with `HTTPProxy`. + +### Initial Delivery and Capability Gates + +The API fields above describe the long-term consumer contract. Galactic's +current one-protocol, one-VIP-port, one-backend-port `NetworkService` is +an internal programming unit, not a limit on the public `NetworkService` or +`NetworkLoadBalancerRoute` schema. NSO may create several Galactic resources +for one NLB. It must not silently alter route selection or drop a backend to +fit a current datapath limit. The published CRD schema keeps `TLS` in the +listener protocol enum and keeps `backends` as a list of independently named +ports; it does not encode temporary one-backend or one-port limits. Admission +and programming checks enforce the capabilities available in serving cells. + +The first rollout enforces these temporary restrictions in validation and +programming until the required capabilities are available, while retaining +the broader field shapes: + +| Consumer request | Initial behavior | Required to lift the restriction | +| --- | --- | --- | +| TLS listener, SNI match, or backend TLS transport | Reject with `UnsupportedCapability`; opaque TLS on a TCP listener remains valid. | Proxy-capable edge path, certificate lifecycle, and TLS route programming. | +| More than one backend entry on a route, including entries with different resolved port numbers | Reject until the cell can program the complete set. Keep `backends` as a list of named-port references. | Galactic intent that represents every selected endpoint and its destination port, plus health and selection behavior. | +| Reuse one backend address, port, and protocol in multiple forwarding rules of the same address family | Reject a conflicting attachment or report `BackendClaimConflict` if changing membership creates it later. | Galactic backend return-path support for multiple rules and VIPs on one socket. | +| One address request shared by multiple listeners | Reject until announcements can be gated on every listener's readiness in the location. | Aggregate VIP readiness and coordinated advertisement. | +| IPv4 in `families` | Reject where the public pool, backend bindings, or Galactic IPv4 feature gate is unavailable. | Capability reported by every serving cell in the location. | + +Validation returns the unsupported field and reason when a consumer submits +an unsupported spec. A previously valid route that becomes unprogrammable +after membership or location changes reports a condition, keeps its allocated +address, and withdraws affected announcements. An unsupported route is not +partially programmed; health-driven exclusion of individual endpoints remains +valid. Round robin and flow affinity are required before any NLB listener +serves traffic; the first delivery must not substitute Galactic's current +Maglev selection silently. ### Gateway API L4 Compatibility @@ -408,16 +511,16 @@ can translate unweighted [TCPRoute](https://gateway-api.sigs.k8s.io/guides/user-guides/tcp/) and [UDPRoute](https://gateway-api.sigs.k8s.io/guides/user-guides/udp/) objects into the same internal intent. It must report unsupported optional features instead -of silently changing their behavior. TLSRoute is unsupported, including when -the backend could terminate TLS itself: TLSRoute requires a TLS listener and -SNI routing semantics that this API does not provide. Claim only Gateway API -features that pass the relevant conformance tests, and document any +of silently changing their behavior. TLSRoute translation is deferred until +the TLS listener and SNI behavior above are actually served. Claim only Gateway +API features that pass the relevant conformance tests, and document any implementation-specific backend mapping. | Gateway API capability | Datum API behavior | | --- | --- | -| Gateway addresses, named TCP/UDP listeners and ports | `NetworkLoadBalancer.exposure.addresses` and `listeners`; Datum additionally selects locations and allocates one address in each. TLS listeners are unsupported. | +| Gateway addresses, named TCP/UDP/TLS listeners and ports | `NetworkLoadBalancer.exposure.addresses` and `listeners`; Datum additionally selects locations and allocates one address in each. TLS listeners are gated initially. | | TCPRoute and UDPRoute, listener attachment and backendRefs | `NetworkLoadBalancerRoute.parentRefs` and unweighted `backends`; one effective catch-all per TCP/UDP listener, with Gateway API precedence. Weighted backendRefs are unsupported. | +| TLSRoute SNI hostnames and TLS certificates | TLS listener `hostname`, `tls.mode`, and certificate reference, plus route `sniHostnames`; adapter support waits for the proxy-capable data plane. | | AllowedRoutes, parentRefs, cross-namespace ReferenceGrant and status | Explicit attachment policy and backend grants; report Accepted, ResolvedRefs and programming per attachment, plus listener conflicts and readiness. | | ListenerSet and multiple parent attachments | Adapter expands these into listeners and route attachments while preserving ownership, conflict resolution and status for each source object. | @@ -435,9 +538,10 @@ resource unless explicitly configured through a Datum extension. The first delivery may expose a smaller native feature set. It must not claim support for any Gateway API L4 feature until the adapter passes that -feature's conformance tests. TLSRoute, BackendTLSPolicy, and full -Extended-feature compatibility, including weighted backendRefs, are outside -this proposal. Track the supported feature set against a pinned Gateway API +feature's conformance tests. TLSRoute and BackendTLSPolicy are deferred until +their corresponding TLS behavior is served. Full Extended-feature +compatibility, including weighted backendRefs, is outside this proposal. +Track the supported feature set against a pinned Gateway API release; the upstream API can evolve independently. ### Route Controls @@ -484,19 +588,25 @@ flow, gateway failover, probe failures, loss of the last backend, location removal, opaque TLS over a TCP listener, CNAME resolution, A/AAAA publication, IP replacement without hostname change, and withdrawal before IP release. Existing `NetworkService` and `HTTPProxy` behavior should be unaffected. +Expand the rollout to multiple backend entries and shared backend sockets only +after their Galactic contracts are verified. Enable SNI passthrough and then +Secret-backed TLS termination as separate capability-gated steps. ### Monitoring Requirements Operators need counts of allocated VIPs, programmed rules, healthy backends, announced locations, and published DNS records, plus failures to allocate, probe, program, publish DNS, or withdraw. Consumers read load balancer and route -conditions, backend health, generated location -hostnames, and per-address announcement and DNS status. +conditions, backend health, generated location hostnames, and per-address +announcement and DNS status. +When TLS listeners are enabled, add certificate readiness and rotation failure +signals per listener and location. ### Dependencies IPAM must provide globally unique public IPv6 VIPs, one for each selected -location. NSO must allow UDP `NetworkService` ports and require the route +location. NSO must allow UDP `NetworkService` ports (tracked in +[PR #683](https://github.com/datum-cloud/network-services-operator/pull/683)) and require the route listener and backend port protocols to match; `HTTPProxy` must continue to reject UDP backends. Selected cells need a health probe runner, Galactic gateways, backend VIP bindings, round-robin flow selection with an explicit @@ -511,6 +621,8 @@ IP replacement and temporary health loss. Gateway API support also needs an attachment/grant model, Service backend translation, per-parent status, and conformance tests against a pinned upstream release. Weighted backendRefs remain unsupported. +Serving TLS listeners additionally needs an edge proxy tier, secure Secret +distribution, SNI routing, and backend TLS validation. ### Scalability @@ -534,6 +646,9 @@ address that could not be allocated. SNI routing, TLS termination, and backend re-encryption to a separate proxy design; aligned IPv4 dependencies with Galactic's existing gated IPv4 VIP path. +- 2026-10-21: Kept long-term TLS and multi-backend fields in the consumer API + proposal while defining temporary validation gates for unsupported traffic + configurations. ## Drawbacks From 1fe4eab73180be2d548390e6c54653d20202f98d Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Wed, 21 Oct 2026 20:38:44 -0500 Subject: [PATCH 23/25] Clarify NLB placement, socket ownership, and health capacity --- docs/enhancements/network-load-balancers.md | 45 ++++++++++++++++----- 1 file changed, 35 insertions(+), 10 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index e4f5bcac..70e098ee 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -123,7 +123,8 @@ request implicitly. The listener references the request once for both families. Omitting `locations` defaults to `locations.from: Backends`: an address request follows the locations of the `NetworkService` backends on listeners using that address. `All` explicitly -selects every eligible platform location; `Names` selects only listed +selects every platform location whose `Ready` condition is `True` for its +current generation; `Names` selects only listed locations. Adding a backend location creates an IP for each family without changing existing location addresses. Separate requests remain available when families need different location selections or independent listener attachment. @@ -144,13 +145,26 @@ it has no address records. Different requests in the same location receive different hostnames. DNS names do not carry a port; clients still connect to the listener's port. -For an address shared by several listeners, `Backends` selects a -location only when every backend service on the effective routes has a -selected, bound network interface there. This keeps every route on the IP -usable in the same locations. Consumers can use separate address -requests when services have different footprints. Interface presence selects -locations; health and programming determine announcement. With no attached -route, placement is pending and no location IP is allocated. +For one effective listener, `Backends` selects the union of locations with a +selected, bound interface in any of its backend services. A backend absent +from one selected location contributes no endpoints there; that location can +still serve through another backend. Interface presence selects locations; +health and programming determine announcement. With no attached route, +placement is pending and no location IP is allocated. + +For a future address shared by several listeners, placement must be decided +across all listeners using that address, and announcement requires every +listener to be ready in the location. Sharing an address across listeners and +selecting multiple address requests on one listener are capability-gated in +the first release, while the API retains both shapes for later support. + +A route without an allocated IPv6 VIP in a location owns no Galactic backend +socket there. Once a VIP is allocated, an effective route with resolved +backends may claim each endpoint socket in that location. Claims are assigned +in route creation order, with a stable name tie-break; a route blocked by an +older owner claims nothing and cannot block a later route. Route status and +cell intent use the same ownership decision. A withdrawn or unavailable VIP +releases its socket claim after serving intent is withdrawn. ### User Stories @@ -399,8 +413,19 @@ TCP connect probe on its serving port; it may instead name a TCP, HTTP, or HTTPS probe on another `NetworkService` port. A UDP backend must name a TCP, HTTP, or HTTPS health endpoint that reports whether the UDP application can serve. There is no generic UDP probe with reliable success semantics. Probe -interval, timeout, and healthy and unhealthy thresholds have platform defaults -and can be set on the health check. +interval, timeout, and healthy and unhealthy thresholds have platform defaults. +The fields remain in the consumer API, but explicit values are rejected until +the cell contract and runner can honor them. + +The first cell runner admits at most 64 targets per health object, evaluates +at most eight probes concurrently, and gives each probe a two-second timeout +after it gets a slot. A full 64-target sweep can require eight waves, or 16 +seconds. The next sweep begins five seconds after completion; five seconds is +not a per-target interval at this capacity. Health status is published after +each full sweep. To keep an early successful result eligible until the next +bounded sweep publishes, its freshness window covers two maximum sweeps plus +the pause and one probe timeout of scheduling slack (39 seconds). A delayed +runner beyond that bound expires health closed. An endpoint serves only when its workload reports `HolderAvailable` and its active probe passes. Probes run from each serving location. The cell publishes From eba53782a7d92057c4cf614cb227086f2c30a4e1 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 23 Oct 2026 16:59:01 +0000 Subject: [PATCH 24/25] docs: Address NLB proposal review feedback Align the proposal with the API and the cell contract after review. Key changes: - use ResolvedRefs with reason NetworkMismatch, matching the route API - add a Backend Trust section on how serving cells verify backends - replace strict round robin with consistent per-flow selection, which keeps a flow on its endpoint across gateway nodes --- docs/enhancements/network-load-balancers.md | 70 ++++++++++++++------- 1 file changed, 46 insertions(+), 24 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index 70e098ee..ead8acf4 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -16,6 +16,7 @@ latest-milestone: "v0.x" - [Design Details](#design-details) - [IPv4 Ingress Data Plane](#ipv4-ingress-data-plane) - [Backend Health](#backend-health) + - [Backend Trust](#backend-trust) - [TLS](#tls) - [Initial Delivery and Capability Gates](#initial-delivery-and-capability-gates) - [Gateway API L4 Compatibility](#gateway-api-l4-compatibility) @@ -51,8 +52,9 @@ where their service runs. - Publish a TCP or UDP `NetworkService` through a named listener and separate route. -- Actively check backend health and select healthy endpoints round robin within - the ingress location. +- Actively check backend health and spread new connections and flows across + healthy endpoints within the ingress location, keeping each flow on one + endpoint. - Support opaque TLS over TCP initially, with an API shape for later SNI-based passthrough, TLS termination, and optional backend TLS. - Let a route refer to multiple `NetworkService` backends by named port, @@ -103,7 +105,7 @@ backend exists and prevents changing service membership from silently moving its traffic to another network. Every effective route backend must resolve to interfaces on that network. A backend with no members cannot serve until members appear. One resolving to a different network sets -`BackendResolved=False` with reason `NetworkMismatch` and invalidates its +`ResolvedRefs=False` with reason `NetworkMismatch` and invalidates its route. Existing public IP allocations are retained; listener readiness still controls announcements. Cross-network backends would need an explicit future forwarding and authorization design. @@ -202,6 +204,9 @@ releases its socket claim after serving intent is withdrawn. - **Overlapping routes could select different backends for the same flow.** Use Gateway API precedence for overlapping route attachments and report the effective route in native status. +- **A forged or stale backend record could send traffic to the wrong + workload.** The serving location re-verifies every backend against live + state before programming it; see [Backend Trust](#backend-trust). - **Probe failure can withdraw a healthy location.** Report probe errors separately from an unhealthy backend and avoid making one probe runner a single point of failure. @@ -279,9 +284,12 @@ route. `backends` accepts multiple entries without traffic weights. In each serving location, the route combines their healthy endpoints into one set and assigns -new TCP connections or UDP flows to those endpoints round robin. Each healthy -endpoint receives an equal turn, so a service with more healthy endpoints -receives more connections or flows. A backend with no healthy endpoints +new TCP connections or UDP flows to those endpoints with a consistent per-flow +hash (Galactic's Maglev selection). Each healthy endpoint receives an equal +share, so a service with more healthy endpoints receives more connections or +flows. Every gateway node in a location computes the same choice, so a flow +keeps its endpoint when it lands on a different node, and adding or removing an +endpoint moves only a small share of flows. A backend with no healthy endpoints contributes none; the route is unready in that location only when no healthy endpoint remains across all its backends. Failed health probes remove individual endpoints from the rotation. Status distinguishes resolution, @@ -434,6 +442,23 @@ the same endpoint eligibility. A location with no healthy backends withdraws its announcement but keeps its IP allocation. Status reports healthy backend counts and probe failures by location. +### Backend Trust + +The project control plane resolves route backends from copies of cell network +interfaces, but those copies are hints, not authorization. Each copy carries +the UID of the cell interface it came from; the route counts only copies that +carry it, so a deleted and recreated interface with the same name and address +is never mistaken for the old one. + +Before programming an endpoint, the serving cell reads the live network +interface and requires its UID, network, location, address and availability to +match the intent. It then reads that interface's VPC attachment and requires +the interface, owner, VPC and address to match, and checks that the source +namespace belongs to the project in the intent. Matching an address alone is +not enough, because different VPCs can reuse the same address. Any missing or +changed field removes that endpoint from the set; the cell reports which +endpoints it verified. + ### TLS A TCP listener forwards TLS bytes unchanged to a backend that owns its @@ -525,9 +550,8 @@ an unsupported spec. A previously valid route that becomes unprogrammable after membership or location changes reports a condition, keeps its allocated address, and withdraws affected announcements. An unsupported route is not partially programmed; health-driven exclusion of individual endpoints remains -valid. Round robin and flow affinity are required before any NLB listener -serves traffic; the first delivery must not substitute Galactic's current -Maglev selection silently. +valid. Consistent per-flow selection and flow affinity across gateway nodes are +required before any NLB listener serves traffic. ### Gateway API L4 Compatibility @@ -588,12 +612,11 @@ candidates for later releases beyond the initial NLB scope: Connection draining, source-IP preservation, and PROXY protocol are delivery settings rather than route matches; they need their own data plane contracts. -Galactic currently uses a deterministic Maglev hash across backend endpoints. -Strict round-robin assignment of new connections or flows needs a different -selection mechanism plus affinity for subsequent packets. The design must -specify what happens when an anycast flow reaches a different gateway node; -per-gateway counters alone cannot preserve its chosen endpoint. Route matches -and priority policy need implementation before the API accepts them. +Endpoint selection uses Galactic's deterministic Maglev hash, which gives an +even spread and keeps a flow on its endpoint even when anycast moves it to +another gateway node. Strict round robin is not offered: per-node counters +cannot preserve a flow's endpoint across nodes. Route matches and priority +policy need implementation before the API accepts them. ## Production Readiness Review Questionnaire @@ -606,12 +629,12 @@ advertisements are gone. ### Rollout, Upgrade and Rollback Planning -Start with single-location TCP and UDP using round robin across healthy -endpoints and active probes, then test multi-location operation with distinct -addresses. Exercise route updates, round-robin selection, affinity during a -flow, gateway failover, probe failures, loss of the last backend, location -removal, opaque TLS over a TCP listener, CNAME resolution, A/AAAA publication, -IP replacement without hostname change, and withdrawal before IP release. +Start with single-location TCP and UDP using consistent per-flow selection +across healthy endpoints and active probes, then test multi-location operation +with distinct addresses. Exercise route updates, even flow spread, affinity +during a flow, gateway failover, probe failures, loss of the last backend, +location removal, opaque TLS over a TCP listener, CNAME resolution, A/AAAA +publication, IP replacement without hostname change, and withdrawal before IP release. Existing `NetworkService` and `HTTPProxy` behavior should be unaffected. Expand the rollout to multiple backend entries and shared backend sockets only after their Galactic contracts are verified. Enable SNI passthrough and then @@ -634,9 +657,8 @@ location. NSO must allow UDP `NetworkService` ports (tracked in [PR #683](https://github.com/datum-cloud/network-services-operator/pull/683)) and require the route listener and backend port protocols to match; `HTTPProxy` must continue to reject UDP backends. Selected cells need a health probe runner, Galactic -gateways, backend VIP bindings, round-robin flow selection with an explicit -affinity and gateway-failover contract, and a way to report programming and -announcement state. +gateways, backend VIP bindings, consistent per-flow selection that survives +gateway failover, and a way to report programming and announcement state. IPv4 VIPs additionally need a public IPv4 pool, family-aware cell intent, Galactic's existing IPv4 feature enabled with backend bindings, and a serving capability gate before API admission is enabled. From 8234cb39e4d1a346b9e65e272170fc5574dd9bf0 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 23 Oct 2026 17:44:31 +0000 Subject: [PATCH 25/25] docs: Clarify flow affinity, hostnames and quotas --- docs/enhancements/network-load-balancers.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/docs/enhancements/network-load-balancers.md b/docs/enhancements/network-load-balancers.md index ead8acf4..a5577c01 100644 --- a/docs/enhancements/network-load-balancers.md +++ b/docs/enhancements/network-load-balancers.md @@ -53,8 +53,8 @@ where their service runs. - Publish a TCP or UDP `NetworkService` through a named listener and separate route. - Actively check backend health and spread new connections and flows across - healthy endpoints within the ingress location, keeping each flow on one - endpoint. + healthy endpoints within the ingress location, keeping each flow on its + endpoint while the healthy set is unchanged. - Support opaque TLS over TCP initially, with an API shape for later SNI-based passthrough, TLS termination, and optional backend TLS. - Let a route refer to multiple `NetworkService` backends by named port, @@ -138,8 +138,9 @@ endpoint label using the proxy format: two words and five hash-derived characters. The label is derived from the load balancer UID and request name; the hostname is `..`. The label is reused across locations, does not expose the request name or IP, and stays -stable while the request exists. NSO claims each generated hostname uniquely -and retains existing names if the generation algorithm changes. Consumers can +stable while the request exists. NSO will claim each generated hostname +uniquely and retain existing names if the generation algorithm changes; the +first release derives the name into status but does not yet claim it. Consumers can CNAME a name they own for that location to it. Datum publishes an A or AAAA record only when that family's address is allocated and announced in the location. The hostname stays assigned during temporary health loss, even when @@ -197,6 +198,9 @@ releases its socket claim after serving intent is withdrawn. - **A `Backends` or `All` request can expand over time.** Expose each added location, hostname, and IP in status, and bound the number of locations and addresses per load balancer. +- **A project could create unbounded load balancers.** Per-project quotas cap + the number of load balancers and routes; projects receive an allowance only + once a platform grant for those quotas is in place. - **DNS could direct users to an unannounced IP.** Publish A and AAAA records only after the corresponding VIP is announced; withdraw the DNS record when that announcement is withdrawn. Short TTLs limit stale answers; clients with @@ -696,6 +700,9 @@ address that could not be allocated. - 2026-10-21: Kept long-term TLS and multi-backend fields in the consumer API proposal while defining temporary validation gates for unsupported traffic configurations. +- 2026-10-23: Addressed review: used `ResolvedRefs` for network mismatches, + added backend trust checks, replaced round robin with consistent per-flow + selection, marked hostname claims as planned, and noted per-project quotas. ## Drawbacks