Public API for network load balancer #696

Task

Define a public API for the per-region network load balancer proposed in datum-cloud/enhancements#884, so it can be created and managed programmatically and consumed by clients such as datumctl (#694) and the web portal (#695). Its controller turns each load balancer into the internal NetworkRules that galactic serves.

Scope

• Load balancer resource. Region, addresses, protocol and port, backend set, session affinity, and status. Per fix(vrf): Tear down routes left behind by a restarted sidecar #884:
  • The region is an explicit field. Every backend must be in that region (Keep a load balancer's VIPs and backends in its own region #736). Keeping region a field leaves room for fix(vrf): Tear down routes left behind by a restarted sidecar #884's second phase, global anycast, without a breaking change.
  • An IPv6 address or subnet is allocated automatically on create (Allocate a default IPv6 address and reserved static IPv4 addresses for load balancers #735).
  • One or more reserved static IPv4 addresses can be bound (Allocate a default IPv6 address and reserved static IPv4 addresses for load balancers #735).
  • Protocols are TCP and UDP for now. The protocol field follows the extensible shape from Make NetworkRule's protocol field extensible to protocols without ports #734, so ESP, GRE, ICMP and SCTP can be added later without a breaking change.
  • Session affinity is None (default), ClientIP or ClientIPAndProtocol (Configurable session affinity for load balancer rules #733).
  • A backend listens on the load balancer's port. Port remapping isn't supported (NetworkRule backend port is accepted but ignored #737).
• IPv4 reservation resource. A tenant reserves a static IPv4 address in a region, independent of any load balancer, and binds it to one (Allocate a default IPv6 address and reserved static IPv4 addresses for load balancers #735).
• Status. Whether the load balancer is provisioned, advertised and reachable, and backend health (Health-check load balancer backends and drop unhealthy ones from the Maglev table #704, Define tenant-facing status for the network load balancer #708).
• Controller. Allocates the VIPs (Allocate load balancer VIPs from a platform-managed pool #702, Allocate a default IPv6 address and reserved static IPv4 addresses for load balancers #735) and writes NetworkRules into every cluster in the region that holds a backend (Keep NetworkRules consistent across clusters so backend replies have a return path #706).
• Placement. The resource's place in the existing Datum Cloud API surface (grouping, versioning, ownership within a VPC).

Open questions

• Does this resource need its own API group and version, or does it extend an existing networking API?
• What's the right granularity for backend health in status: per backend, or aggregate only?

Phase 3 of #698. The protocol and affinity fields depend on #734 and #733. Must not reach tenants before Phase 1 of #698 is done.