Add service-pool / load-balancing layer for multiple identical service instances

[chrono-kw]

Summary

Add a service-pool / load-balancing layer so a user or org can register several interchangeable instances of the same logical service and have the NyxID proxy spread traffic across them, with health-aware failover. Today a UserService binds to exactly one endpoint (+ optional one node), so running N identical backends (e.g. several GPU/compute workers, several upstream API instances, mirrored regions) means manually picking one slug per call — there is no way to say "balance across these."

This is the generic counterpart to what PR #969 (integrations/compute-pool-service) deliberately keeps out of core. That PR's own framing:

This is not a NyxID service-pool framework. Cross-service counting, quotas, metering, and load balancing should be handled by a future generic NyxID service-pool design rather than by a compute-specific core API. The compute service exposes /v1/status as a capacity signal that such a layer could use later.

So #969 ships the data-plane queue, and this issue tracks the NyxID-side control-plane feature it points at.

Motivation / use case

• Operator stands up 3 instances of the same backend (same auth, same API shape) for capacity or redundancy.
• Agents / org members keep calling one stable slug; NyxID decides which instance serves each request.
• When one instance is unhealthy or at capacity, traffic shifts to the others automatically instead of erroring.

Existing precedent to build on (not from scratch)

NyxID already does node-level failover, just not service-level load balancing:

• services/node_routing_service.rs → resolve_node_route() returns NodeRoute { fallback_node_ids: Vec<String> }, and the proxy already walks primary → fallback when a node is offline (test: resolve_node_route_fails_over_from_offline_node_to_online_fallback).
• models/user_service.rs binds a single endpoint_id + optional node_id.
• services/proxy_service.rs → resolve_proxy_target_from_user_service() resolves one target.

The ask is to generalize that failover into capacity-aware balancing across multiple service instances, not just node fallback within one service.

Proposed scope (for architect review — not final)

1. Pool model — a ServicePool grouping N member UserServices (or N endpoints under one service) that share a stable slug, owned by the same person/org user (reuse org_service::resolve_owner_access for ACL, consistent with Node / UserService).
2. Balancing strategies — at minimum round-robin and least-in-flight; ideally weighted and capacity-aware using a pluggable health/status signal (a member can expose something like Add standalone compute pool service integration #969's GET /v1/status → {queued, dispatched, active_workers}).
3. Health checks + failover — periodic or proxy-time health probe; skip/deprioritize unhealthy members; generalize fallback_node_ids to fallback members.
4. Proxy integration — pool resolution slots into proxy_service target resolution; a slug can resolve to a pool, then to a concrete member per-request.
5. Sticky routing (optional) — affinity by session/client_ref for multi-turn/stateful backends.
6. Observability — per-member request counts / errors for audit + the metering story below.

Explicitly out of scope here (track separately if wanted)

• Org/agent quotas, usage counting, and metering across pool members. Add standalone compute pool service integration #969 notes these belong to the same future layer; they're a distinct, larger workstream and shouldn't block basic balancing.

Open questions for architecture

• Pool as a new model vs. extending UserService with member_service_ids / a pool_id?
• Health signal: standardized contract (a /status-style endpoint NyxID polls) vs. passive (infer health from proxy error rates)? Add standalone compute pool service integration #969's /v1/status is a candidate shape.
• Does balancing run only over node-routed members, direct-HTTP members, or both?
• Interaction with existing fallback_node_ids — does the node fallback collapse into the pool layer, or stay underneath it?

References

• PR Add standalone compute pool service integration #969 — integrations/compute-pool-service (the data-plane service that defers to this)
• Closed prototype PR Add compute pool scheduler MVP #967 (proved the worker-pull UX / safety boundary)

[chronoai-bot]

github-devloop intake decision: enable
Service class: standard

Reason:
Implementation request for a standard service-discovery/load-balancing/health-check pattern; no human-gate condition, not merely an umbrella, and recurrence escalation does not apply because this is the class carrier rather than an instance with two closed siblings.

[chronoai-bot]

github-devloop thinking: consensus started

[chronoai-bot]

github-devloop convergence round 0 - no three-angle consensus; narrowing

Narrowed question: Does NyxID need an application-level ServicePool at all, or should the identical-backend case be handled by existing endpoint_id, upstream L7/service-discovery load balancing, fallback_node_ids, or #969's data-plane queue before narrowing to one slug over N existing UserService members with round-robin and unhealthy-member skip?

Angle stances:

• minimal: abstain — Established practice for this class is a standard load-balancer/service-discovery pool: a stable logical service resolves to health-checked interchangeable backends with one simple routing policy first; the current proposal is broader than the minimal path because ServicePool, least-in-flight, weighted/capacity-aware status, sticky routing, and observability are bundled while the data model is still an open question. Narrow this to “one slug over N existing UserService members, round-robin plus unhealthy-member skip/failover, proxy integration tests” or provide evidence that the extr
• structural: abstain — Established practice for this class is a separate service-discovery/load-balancing control plane with one authoritative pool-membership/health source and a narrow data-plane routing contract; this proposal is directionally sound, but structurally not ready because it leaves the decisive contracts open: whether ServicePool is its own model or embedded in UserService, whether health is a standardized typed contract or passive inference, and how pool member fallback composes with existing fallback_node_ids.
• delete: abstain — Established practice for this problem class is to put health-aware balancing behind a stable endpoint in infrastructure/service discovery/L7 proxy and keep application identity-control surfaces deterministic unless per-member authorization, audit, or metering requires otherwise; Add service-pool / load-balancing layer for multiple identical service instances #974 does not prove why existing endpoint_id, an upstream load balancer, current fallback_node_ids, or Add standalone compute pool service integration #969's data-plane queue cannot solve the identical-backend case, while it proposes new ServicePool, balancing, health, sticky-routing, and observability surface with quotas/metering explicitly out of scope, so t

⟦AI:FKST⟧

[chronoai-bot]

github-devloop convergence round 1 - no three-angle consensus; narrowing

Narrowed question: Reject the broad generic ServicePool framing for now; next round must either prove a NyxID-owned tenant/auth/source-of-truth reason that endpoint_id, upstream L7/service discovery, fallback_node_ids, and #969's data-plane queue cannot satisfy, or narrow to a minimal one-slug selector over existing UserService members with only round-robin and unhealthy-member skip.

Angle stances:

• minimal: abstain — Established service-discovery/L7 load-balancing practice favors reusing upstream balancers, service discovery, or existing endpoint/failover abstractions before adding a bespoke application-level balancer; under the minimal angle, this proposal has not proven that endpoint_id, upstream L7/service discovery, fallback_node_ids, or Add standalone compute pool service integration #969's data-plane queue cannot satisfy the identical-backend case, and its proposed ServicePool scope adds strategies, health probing, sticky routing, and observability beyond the smallest coherent path. I would approve after evidence of that gap plus a narrowed
• structural: abstain — Established practice for identical backend instances is to keep load balancing and health checking in L7/service-discovery/service-mesh infrastructure, or in the owning data plane, unless the application has a tenant/auth/source-of-truth reason to choose members itself; this proposal has not proven that existing endpoint_id, upstream balancing, fallback_node_ids, or Add standalone compute pool service integration #969's queue cannot handle the case. Structurally, adding ServicePool now risks duplicating routing authority across UserService, node_routing_service, proxy_service, and the compute data plane without a settled ownersh
• delete: abstain — Established practice puts identical-backend load balancing and health checks in L7/service-discovery/service-mesh infrastructure or behind one stable upstream, with application routing only for domain-specific selection; Add service-pool / load-balancing layer for multiple identical service instances #974 proposes a generic ServicePool with round-robin, least-in-flight, health checks, sticky routing, and observability, but does not prove why existing endpoint_id, upstream balancing, fallback_node_ids, or Add standalone compute pool service integration #969's data-plane queue cannot cover the stated "one stable slug" need, so the new NyxID surface should be narrowed or deleted until necessity is demonstrated.

⟦AI:FKST⟧

[chronoai-bot]

github-devloop decision: approve
Three-angle verdicts: minimal=approve structural=approve delete=approve

minimal:
Established practice for load balancing is a stable logical endpoint selecting among health-checked interchangeable backends, normally via L7/service discovery unless the app owns identity, authorization, tenancy, or audit semantics; from the minimal angle, the broad ServicePool scope remains unproved, so the smallest coherent approvable path is exactly this narrowing: either justify why existing endpoint_id, upstream L7/service discovery, fallback_node_ids, and #969's data-plane queue do not apply, or reduce the issue to one stable slug over existing UserService members with round-robin and unhealthy-member skip.

structural:
Established practice for this class is to keep load balancing/service discovery behind one authoritative membership/health source with a narrow routing contract; this narrowing is structurally acceptable because it rejects an unproven generic "ServicePool" model and requires either proof of a NyxID-owned tenant/auth/source-of-truth gap beyond "endpoint_id", upstream L7/service discovery, "fallback_node_ids", and #969's data-plane queue, or a minimal one-slug selector over existing "UserService" members with only round-robin and unhealthy-member skip.

delete:
Established practice for identical backend capacity/redundancy is infrastructure or service-discovery/L7 balancing behind one stable endpoint, while an identity/proxy app should keep routing deterministic unless it owns a tenant/auth/audit/source-of-truth boundary; the current narrowed proposal correctly rejects the broad generic ServicePool surface because the issue still does not prove why endpoint_id, upstream L7/service discovery, fallback_node_ids, or #969's data-plane queue are insufficient, and only leaves room for a much smaller one-slug selector over existing UserService members if that need is proven.

⟦AI:FKST⟧