Skip to content

feat(cloud): tenant provider registry and model-group fallback #410

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
seonghobae wants to merge 1 commit into from
Closed

feat(cloud): tenant provider registry and model-group fallback #410

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 97 additions & 0 deletions docs/superpowers/plans/2026-08-13-tenant-cloud-routing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Tenant Cloud Routing Implementation Plan

> Execute with strict red-green-refactor TDD on the stacked PR based on PR #96.

**Goal:** Add a tenant-scoped, PostgreSQL-backed provider/model registry, a direct OpenAI-compatible model-group endpoint with deterministic fallback, and Cloud Native deployment evidence.

**Architecture:** Provider secret values remain in the existing pgcrypto KV. New normalized tenant-routing metadata references tenant-qualified KV names. A request-scoped group executor creates existing `ModelAgent` values and calls the existing `ModelClient`; the Cloud Gateway is a separate importable/runtime module so the current standalone server remains compatible.

**Stack:** Python standard library, optional psycopg DB extra, PostgreSQL 18/pgcrypto, Docker Compose, Kubernetes manifests, GitHub Actions.

---

## Task 1 — Lock the contracts with failing tests

**Files:**
- Create `tests/test_tenant_registry.py`
- Create `tests/test_model_group_fallback.py`
- Create `tests/test_cloud_gateway_http.py`
- Create `tests/test_cloud_native_contract.py`

1. Write tests for secret non-disclosure, rotation, tenant isolation, endpoint ownership, ordering, disablement, and normalized SQL names.
2. Write tests for first-failure/second-success, empty-result fallback, no out-of-group attempt, deterministic evidence, and all-failed behavior.
3. Write HTTP tests for auth, tenant header, admin CRUD, OpenAI completion shape, minimal liveness, database readiness, and web UI secret handling.
4. Write deployment-contract tests for two Compose gateways, two Kubernetes replicas, probes, provider-key isolation, and live-workflow secret names.
5. Run the exact test files and retain the expected import failures as RED evidence.

## Task 2 — Implement the tenant registry

**Files:**
- Create `contextual_orchestrator/tenant_registry.py`

1. Add immutable domain records and stable domain exceptions.
2. Add an in-memory backend with injectable shared state for deterministic tests.
3. Add PostgreSQL schema and CRUD using parameter binding and pgcrypto.
4. Namespace every KV credential by tenant and label.
5. Resolve only enabled, same-tenant group members in deterministic order.
6. Add beginner-readable public docstrings.

## Task 3 — Implement sequential model-group fallback

**Files:**
- Create `contextual_orchestrator/model_group.py`

1. Build request-scoped `ModelAgent` values from resolved endpoint metadata.
2. Call the existing `ModelClient` without copying provider transport logic.
3. Reject empty/non-string output and continue to the next member.
4. Return secret-free attempt evidence and usage.
5. Raise one stable redacted error after complete exhaustion.

## Task 4 — Add the Cloud Gateway and web control plane

**Files:**
- Create `contextual_orchestrator/cloud_admin.py`
- Create `contextual_orchestrator/cloud_gateway.py`

1. Add KV-resolved admin/inference authentication.
2. Add `/livez`, `/readyz`, and authenticated detailed readiness.
3. Add tenant, credential, group, endpoint, and membership JSON routes.
4. Add OpenAI-compatible `/v1/chat/completions` using `model` as the group name.
5. Add a same-origin admin UI that stores no raw token or provider secret.
6. Add bounded bodies, stable errors, no-store headers, and strict validation.

## Task 5 — Add Cloud Native deployment and bootstrap

**Files:**
- Create `scripts/bootstrap_tenant_registry.py`
- Create `scripts/verify_live_provider_fallback.py`
- Create `deploy/docker-compose.cloud.yml`
- Create `deploy/kubernetes/namespace.yaml`
- Create `deploy/kubernetes/config-map.yaml`
- Create `deploy/kubernetes/deployment.yaml`
- Create `deploy/kubernetes/service.yaml`
- Create `deploy/kubernetes/network-policy.yaml`
- Create `deploy/kubernetes/pod-disruption-budget.yaml`
- Create `deploy/kubernetes/bootstrap-job.yaml`
- Create `.github/workflows/live-tenant-provider-fallback.yml`

1. Keep provider keys only in the one-shot bootstrap environment.
2. Prove two gateway processes share one PostgreSQL registry.
3. Add bounded live probes for OpenRouter, NVIDIA NIM, and Bytez.
4. Force the first group member to fail and verify the next valid provider wins.
5. Emit only secret-redacted summaries.

## Task 6 — Documentation and exact-head verification

**Files:**
- Create `docs/adr/0011-tenant-provider-registry.md`
- Create `docs/tenant-cloud-routing.md`
- Create `docs/doctoring/tenant-cloud-routing-references.md`
- Update `CHANGELOG.md`

1. Run the four focused test files.
2. Run the full branch-coverage suite and public-docstring gate.
3. Run deployment-contract and Postgres integration workflows.
4. Run Security, fuzz, SAST, SBOM, and package checks on the same contributor head.
5. Review every automated/human thread and fix every valid finding.
6. Keep the PR Draft until the security prerequisite, exact-head checks, and independent approval are all satisfied.
101 changes: 101 additions & 0 deletions docs/superpowers/specs/2026-08-13-tenant-cloud-routing-design.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Tenant Cloud Routing Design

**Date:** 2026-08-13
**Status:** Accepted for the bounded stacked implementation
**Base authority:** PR #96 exact head, not protected `main`

## Problem

The current gateway resolves one named provider credential from a process-global KV and keeps its agent pool in process memory. That is insufficient for a Cloud Native deployment where several Docker services or Kubernetes Pods must share one tenant's provider keys and model routing policy without treating one process as configuration authority.

## Approaches considered

### A. Keep process-local agent JSON and copy it into every Pod

Rejected. Rotation, endpoint disablement, and fallback order would drift between replicas, and a rollout would be required for every routing change.

### B. Add a new general-purpose secret-vault service

Rejected for this slice. It duplicates the existing pgcrypto credential boundary and would create another identity and availability dependency before a second proven implementation requires it.

### C. Reuse the encrypted KV for secret values and add normalized tenant routing metadata

Selected. Provider secrets remain in `provider_credentials`; tenant-qualified credential names prevent collisions. Separate normalized tables own tenant, credential metadata, groups, endpoints, and ordered membership. Every gateway replica resolves the group from PostgreSQL on each request and then uses the existing `ModelClient` provider trust boundary.

## Identity boundary

Keyverse/cwl-idp, or an equivalent verified identity proxy, remains the identity authority. The gateway does not create users, passwords, or identity tokens. The direct Cloud Gateway API requires an authenticated admin/inference bearer resolved from the shared KV. The browser UI stores no bearer in local or session storage and assumes same-origin identity-proxy injection in production.

## Data model

```mermaid
erDiagram
tenant_records ||--o{ tenant_provider_credentials : owns
tenant_records ||--o{ tenant_model_groups : owns
tenant_records ||--o{ tenant_model_endpoints : owns
tenant_model_groups ||--o{ tenant_group_memberships : orders
tenant_model_endpoints ||--o{ tenant_group_memberships : participates
tenant_provider_credentials ||--o{ tenant_model_endpoints : authenticates
```

The secret is not duplicated in tenant metadata. `tenant_provider_credentials.credential_key` points to the pgcrypto-encrypted `provider_credentials.credential_name` row.

## Request flow

```mermaid
sequenceDiagram
participant Caller
participant GatewayA
participant Postgres
participant Provider1
participant Provider2

Caller->>GatewayA: POST /v1/chat/completions\nmodel=general_chat\nX-Contextual-Tenant=acme_corporation
GatewayA->>Postgres: resolve enabled group members for tenant
Postgres-->>GatewayA: ordered endpoint + credential_key metadata
GatewayA->>Postgres: decrypt endpoint 1 credential through KV seam
GatewayA->>Provider1: strict OpenAI-compatible request
Provider1--xGatewayA: failure/invalid completion
GatewayA->>Postgres: decrypt endpoint 2 credential through KV seam
GatewayA->>Provider2: strict OpenAI-compatible request
Provider2-->>GatewayA: complete valid response
GatewayA-->>Caller: completion + secret-free routing evidence
```

## Fallback semantics

This slice implements only `sequential_failover`.

- Membership order is explicit and unique within a group.
- Fallback never leaves the requested tenant or group.
- Disabled tenant, credential, group, endpoint, or membership is excluded.
- A candidate wins only after returning a non-empty complete string through `ModelClient`.
- Attempt evidence contains endpoint/provider/model identifiers and stable outcome codes, never exception text, prompts, responses, or credentials.
- Immediate race and delayed hedge remain owned by issue #102.

## Cloud Native deployment

- Gateway replicas are stateless with respect to tenant routing configuration.
- PostgreSQL is the shared control-plane authority.
- `/livez` performs no dependency access.
- `/readyz` performs only a bounded database ping and never calls an LLM.
- Provider keys enter through a one-shot bootstrap Job; gateway Pods do not receive provider API-key environment variables.
- Two Compose gateway services and a Kubernetes Deployment with two replicas exercise the shared-state contract.

## Verification

Offline tests are authoritative for merge gating. A separate manually dispatched workflow may consume `OPENROUTER_API_KEY`, `NVIDIA_NIM_API_KEY`, and `BYTEZ_API_KEY` to seed the registry, prove cross-process visibility, probe each official OpenAI-compatible provider surface, and demonstrate fallback after an intentionally failing first endpoint.

## References — APA 7th

Dean, J., & Barroso, L. A. (2013). The tail at scale. *Communications of the ACM, 56*(2), 74–80. https://doi.org/10.1145/2408776.2408794

Kubernetes Authors. (2026). *Liveness, readiness, and startup probes*. Kubernetes Documentation. https://kubernetes.io/docs/concepts/workloads/pods/probes/

PostgreSQL Global Development Group. (2026). *pgcrypto—Cryptographic functions*. PostgreSQL 18 Documentation. https://www.postgresql.org/docs/current/pgcrypto.html

OpenRouter. (2026). *API reference*. https://openrouter.ai/docs/api_reference/overview

NVIDIA. (2026). *NIM for large language models API reference*. https://docs.nvidia.com/nim/large-language-models/latest/api-reference.html

Bytez. (2026). *OpenAI-compatible chat completions*. https://docs.bytez.com/http-reference/oaiCompliant/chatCompletions
Loading
Loading