diff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.md index 976e42a6bc..79f2ce73f5 100644 --- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md +++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md @@ -56,6 +56,18 @@ Before running anything, verify ALL of these: `pre-gate-sync` can restart every deployment; T2 then renews the hold once, before Health/Playwright, and records `PortForwards=` evidence. Without it an in-run sync leaves the hold on terminated pods. +- [ ] When the same owned profile is reused after a HEAD change, the + orchestrator runs `pre-gate-sync` for a targeted change and refreshes + forwards after any restarted services. For `targeted-sync`, also supply + a bounded `T2_HEALTHCHECK_COMMAND` that exercises the affected + user-facing path. +- [ ] Before a login-based health journey patches HCC or creates fault + fixtures, resolve its seeded test credential in-process. Source + `scripts/e2e/load-dotenv.sh` and `scripts/e2e/admin-credentials.sh`; + use `dotenv_load_canonical_root` and `e2e_resolve_admin_password`. + An explicit `E2E_USER_PASSWORD` may override the journey credential. + Never inspect, print, log, or persist `.env` contents, even for a + key-presence check. Fail before cluster mutation if resolution fails. Shell contract-test rule: fixtures that exercise Git/lease state must use a temporary repository via `scripts/tests/lib/minikube-fixture-repo.sh`. Keep the @@ -113,6 +125,12 @@ Rules that override any shortcut idea: - Never set `T2_RUN_T0=false` / `T2_RUN_T1=false` by hand to skip a lane that was not certified on this HEAD; the harness refuses it unless the plan is `already-synced`, and evidence must show the earlier green runs. +- Before repeating a completed full T0/T1/T2 certification, propose the + repeat with its reason and expected cost and ask the user to decide. A + recommendation is not consent. If the user declines after HEAD changes, + preserve the prior evidence but report the new HEAD as uncertified. A + same-HEAD `minikube-t2-runtime` retry finishes an authorized run without + repeating T0/T1. ## Step 3 — T1 specifics (Real PostgreSQL) diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.md index 3eacad1a34..239ce976ba 100644 --- a/.cursor/skills/minikube-t0-t1-t2/reference.md +++ b/.cursor/skills/minikube-t0-t1-t2/reference.md @@ -97,7 +97,7 @@ Companion to `SKILL.md`. Source of truth: `scripts/minikube/t2.sh`, | `ZERO_TESTS_EXECUTED` | A lane was configured off or executed nothing (including a required-but-missing Playwright journey). | Re-run with the lane enabled, or supply the required `T2_PLAYWRIGHT_COMMAND`. | | `REAL_PG_REQUIRED_BUT_UNAVAILABLE` | The explicit context, isolated PostgreSQL, DSN route, or exact-head database precondition is unavailable. | Re-enter `make minikube-t2` or `make minikube-t2-real-postgres` on the owned profile. Do not `docker run`, restart Docker Desktop, or probe published ports. | | `REAL_PG_REPORT_INCOMPLETE` | Vitest JSON did not prove the exact expected physical files. | Repair test selection/reporter output; do not accept suite counters as file identity. | -| `REAL_PG_SUITE_FAILED` | Reporter, test assertions, process exit, or parent result contract failed. | Iterate with standalone T1; once green, run full T2 for final exact-head lane evidence. | +| `REAL_PG_SUITE_FAILED` | Reporter, test assertions, process exit, or parent result contract failed. | Iterate with standalone T1; once green, run full T2 for final exact-head lane evidence. If Darwin 27 reports `ECONNRESET` in `routes.externalGfsRateLimitStress.realPostgres.integration.test.ts`, check the suite's shared `http.Agent`: it must cap `maxSockets` below macOS `kern.ipc.somaxconn` (the proven value is 64). An unbounded 481-request burst is reset before Express accepts it; this is a test-socket defect, not a PostgreSQL or Minikube failure. | | `POSTGRES_NOT_READY` | PostgreSQL precondition failed, or a PVC reset was requested without the exact expected UID. | Fix DB readiness; never guess a PVC UID. | | `PROFILE_UNHEALTHY` | A required deployment is unready in a fail-loud check (`T2_PLAN_MODE=false`: standalone preflight or the final exact-head T2), the mandatory targeted-sync health command is missing, or a supplied user-facing health command failed or timed out. The orchestrator planner never emits this merely for an unready deployment — it selects `full-reconcile` instead. | For a targeted sync, supply the affected service's profile-owned `T2_HEALTHCHECK_COMMAND`; otherwise repair the reported health failure. Re-run on the same HEAD and profile. Do not run manual repair scripts between runs. | | `NP08_HCC_AUTHORIZATION_FAILED` | The deployed Host-to-HCC authorization journey or cleanup failed after T0/T1 passed. | Repair the first NP08 failure, then use `minikube-t2-runtime` on the exact same profile/HEAD; never refresh/reissue in the test. | diff --git a/.github/workflows/ci-public.yml b/.github/workflows/ci-public.yml index 9f1b8da45a..8863727845 100644 --- a/.github/workflows/ci-public.yml +++ b/.github/workflows/ci-public.yml @@ -534,6 +534,9 @@ jobs: - name: Test HCC watch-recovery gate coordination run: bash scripts/tests/test-hcc-watch-recovery-gate.sh + - name: Test autoresearch gate wrappers + run: bash scripts/tests/test-autoresearch-gates.sh + - name: Test HCC watch recovery TLS proxy and configuration restoration run: | bash scripts/tests/test-hcc-watch-api-proxy.sh @@ -542,6 +545,9 @@ jobs: - name: Test HCC watch-churn readiness gate contract run: bash scripts/tests/test-hcc-watch-churn-readiness-gate.sh + - name: Test HCC stateless cache-containment gate contract + run: bash scripts/tests/test-hcc-stateless-cache-containment.sh + - name: HCC policy lifecycle and restoration contracts run: | bash scripts/tests/test-hcc-watch-churn-empty-fleet.sh diff --git a/AGENTS.md b/AGENTS.md index cd2c881c60..f09a3117f2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -172,6 +172,26 @@ T2 with `make minikube-t2-runtime` (`T2_RUN_T0=false T2_RUN_T1=false`). That path is valid only when the pre-gate marker already matches HEAD (`already-synced`). Do not set those flags to skip an uncertified lane. +Before repeating an already completed full T0/T1/T2 certification, propose +the repeat with its reason and expected cost, then ask the user whether to run +it. A recommendation alone does not authorize a repeat. This choice does not +turn an earlier HEAD's evidence into certification for a newer HEAD: if the +user declines, report the newer HEAD as uncertified. A same-HEAD +`minikube-t2-runtime` retry to finish an authorized run is the supported +recovery path, not a new full recertification. + +For branch-owned user-facing E2E health journeys, resolve the seeded test +credential locally before patching HCC or creates fault-injection fixtures. +Use `scripts/e2e/load-dotenv.sh` with `dotenv_load_canonical_root` and +`scripts/e2e/admin-credentials.sh` with `e2e_resolve_admin_password`; an +explicit `E2E_USER_PASSWORD` may override that journey's credential. Keep the +resolved value in process memory. Never inspect, print, log, or persist `.env` +contents, even to test whether a key exists. Missing credentials must fail +preflight before mutating the owned profile. Use the profile's persisted +ports and the exact `T2_PORT_FORWARD_COMMAND` hold when `pre-gate-sync` can +restart a service; `targeted-sync` also requires a bounded +`T2_HEALTHCHECK_COMMAND`. + The active profile mutation lock is `$T2_LOCK_ROOT/.lock`; stale-lock reclaim coordination uses the sibling directory `$T2_LOCK_ROOT/.reclaim`. A killed reclaimer may leave that sibling diff --git a/Makefile b/Makefile index ecddfb408f..644c872bcd 100644 --- a/Makefile +++ b/Makefile @@ -137,6 +137,10 @@ test-unit-all: test-service-matrix ## Run unit tests across all services test-service-matrix: ## Check local test coverage against the CI service matrix @node scripts/dev/check-test-services.cjs $(TEST_SERVICES) +.PHONY: autoresearch-checks +autoresearch-checks: ## Run the local checks used by AutoResearch hard gates + @bash scripts/dev/autoresearch-checks.sh + .PHONY: test-codex-subscription-t0 test-codex-subscription-t0: ## Run the Codex subscription T0 aggregator (counts, no skips) @bash scripts/tests/test-codex-subscription-t0.sh @@ -784,6 +788,19 @@ minikube-t2-hcc-watch-recovery: ## Certify PR A recovery omission, runtime repai @bash scripts/tests/test-hcc-watch-pr-a.sh @E2E_HCC_PR_A=1 $(MAKE) minikube-t2-hcc-networkpolicy-lifecycle +.PHONY: minikube-t2-hcc-stateless-cache-containment +minikube-t2-hcc-stateless-cache-containment: ## Opt-in branch-owned T2 cache-containment journey; requires matching profile/context and refreshed user-facing forwards + @bash scripts/tests/test-hcc-stateless-cache-containment.sh + @context='$(CONTROL_API_REAL_PG_CONTEXT)'; \ + [ -n "$$context" ] || { echo 'Set CONTROL_API_REAL_PG_CONTEXT to the verified branch-owned context' >&2; exit 1; }; \ + [ "$(MINIKUBE_PROFILE)" = "$$context" ] || { echo 'MINIKUBE_PROFILE and CONTROL_API_REAL_PG_CONTEXT must select the same branch-owned profile/context' >&2; exit 1; }; \ + printf '%s\n' "$$context" | grep -Eq '^clerum-[a-z0-9][a-z0-9-]*-[0-9a-f]{8}$$' || { echo 'The selected context must be generated and branch-scoped' >&2; exit 1; } + @E2E_HCC_WATCH_FAULT_INJECTION=1 \ + T2_GATE_ID=minikube-t2 \ + T2_HEALTHCHECK_COMMAND='KUBECONTEXT="$$T2_CONTEXT" MINIKUBE_PROFILE="$$T2_PROFILE" E2E_BRANCH_PROFILE_ENV="$$T2_PROFILE_ENV" E2E_PROFILE_PORTS_ENV="$$T2_PORTS_ENV" E2E_EXPECTED_PRE_GATE_GATE=minikube-t2 bash scripts/e2e/e2e-hcc-stateless-cache-containment.sh' \ + T2_HEALTHCHECK_TIMEOUT_SECONDS=900 T2_HEALTHCHECK_KILL_GRACE_SECONDS=300 \ + $(MAKE) minikube-t2 + .PHONY: minikube-t2-runtime minikube-t2-runtime: ## Exact-head T2 after T0 and T1 already passed on this HEAD and profile @T2_RUN_T0=false T2_RUN_T1=false \ @@ -1373,12 +1390,12 @@ test-e2e-stateless-durability: ## Run stateless host durability E2E gate (Phase .PHONY: test-e2e-stateless-suspend-wake test-e2e-stateless-suspend-wake: ## Run stateless suspend/wake E2E gate (Phase 1 API + Phase 2 Desktop Playwright; needs port-forwards + seeded chatllm-stateless) @echo "Running stateless suspend/wake E2E gate..." - KUBECONTEXT=$(E2E_KUBECONTEXT) bash scripts/e2e/e2e-stateless-suspend-wake.sh + E2E_ALLOW_STATELESS_CADENCE_ACCELERATION=1 KUBECONTEXT=$(E2E_KUBECONTEXT) bash scripts/e2e/e2e-stateless-suspend-wake.sh .PHONY: test-e2e-stateless-wake-recovery test-e2e-stateless-wake-recovery: ## Run stateless wake-recovery latency gate (R1 warm draining / R2 cold suspended / R3 drained-window p95 budgets; needs port-forwards + seeded chatllm-stateless) @echo "Running stateless wake-recovery latency gate..." - KUBECONTEXT=$(E2E_KUBECONTEXT) bash scripts/e2e/e2e-stateless-wake-recovery.sh + E2E_ALLOW_STATELESS_CADENCE_ACCELERATION=1 KUBECONTEXT=$(E2E_KUBECONTEXT) bash scripts/e2e/e2e-stateless-wake-recovery.sh .PHONY: test-e2e-hcc-communicationchannel-watch-recovery test-e2e-hcc-communicationchannel-watch-recovery: ## Run isolated minikube HCC watch-recovery fault-injection gate diff --git a/control-api/package-lock.json b/control-api/package-lock.json index 817e575a96..9c880649cf 100644 --- a/control-api/package-lock.json +++ b/control-api/package-lock.json @@ -1,12 +1,12 @@ { "name": "clerum-control-api", - "version": "0.1.680", + "version": "0.1.681", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "clerum-control-api", - "version": "0.1.680", + "version": "0.1.681", "dependencies": { "@clerum/display-field": "file:../packages/display-field", "@clerum/grok-provider-attempt-contract": "file:../packages/grok-provider-attempt-contract", diff --git a/control-api/package.json b/control-api/package.json index ab5a7cfd63..2e64e8220d 100644 --- a/control-api/package.json +++ b/control-api/package.json @@ -1,6 +1,6 @@ { "name": "clerum-control-api", - "version": "0.1.680", + "version": "0.1.681", "description": "Internal Clerum control plane REST API + UI for managing CRDs and secrets", "main": "dist/main.js", "engines": { diff --git a/control-api/src/config.ts b/control-api/src/config.ts index ff868cf706..1c713947d2 100644 --- a/control-api/src/config.ts +++ b/control-api/src/config.ts @@ -1145,11 +1145,11 @@ export const config: Config = { ), // Default derived from the wake mechanism's worst case, not picked ad hoc. // rpc-proxy's wake-and-hold loop re-triggers POST /rpc/hosts/:hostRef/wake - // every wakeRetriggerMs=15000 for up to wakeMaxHoldMs=90000 (defaults in + // every wakeRetriggerMs=15000 for up to wakeMaxHoldMs=48000 (defaults in // rpc-proxy/src/config.ts), so one held request costs - // 1 + floor(90000/15000) = 7 calls per rpc-proxy instance; rpc-proxy runs + // 1 + floor(48000/15000) = 4 calls per rpc-proxy instance; rpc-proxy runs // replicas: 2 (deploy/base/rpc-proxy/rpc-proxy.yaml) with per-instance - // dedup, so the mechanism alone can emit 14 calls / 90s (~9.4/min) per + // dedup, so the mechanism alone can emit 8 calls / 48s (10.0/min) per // host. Desktop prewarm adds a burst of up to 3 calls per device. // Middleware order in routes/rpc-access/hosts.ts is limiter BEFORE // handler, and the hostWakeCoalesceWindowMs coalescer runs INSIDE the diff --git a/control-api/src/routes/rpc-access/users.ts b/control-api/src/routes/rpc-access/users.ts index c6d5ea2364..5dad840a15 100644 --- a/control-api/src/routes/rpc-access/users.ts +++ b/control-api/src/routes/rpc-access/users.ts @@ -1,5 +1,5 @@ import express, { Router } from 'express' -import type { Request } from 'express' +import type { Request, Response } from 'express' import { config } from '../../config.js' import { pool } from '../../db.js' import { K8sGateway } from '../../k8s.js' @@ -48,14 +48,25 @@ const HOST_ACCESS_SCOPES = [ type RpcAuthedRequest = Request & { rpcAuth?: RpcAccessClaims } -function logHostAccessDenial( +/** + * Response header that tells rpc-proxy why control-api denied Host access. The + * 403 body stays exactly `{"error":"Forbidden"}` (a pinned 21-byte contract); + * the reason travels out of band so rpc-proxy can tell removed access from + * other denials. Scope denials from the auth middleware carry no such header. + */ +export const HOST_ACCESS_DENIAL_REASON_HEADER = 'x-host-access-denial-reason' + +function denyHostAccess( req: RpcAuthedRequest, + res: Response, reason: RpcHostAccessDenialReason | 'claims_missing' ): void { req.log?.warn( { event: 'rpc_host_access_denied', reason }, 'rpc host access denied by control-plane authority' ) + res.setHeader(HOST_ACCESS_DENIAL_REASON_HEADER, reason) + res.status(403).json({ error: 'Forbidden' }) } async function bindDirectRunWithinBudget( @@ -187,8 +198,7 @@ export function createRpcAccessUsersRouter( const hostRef = String(req.params.hostRef || '').trim() const claims = req.rpcAuth if (!claims) { - logHostAccessDenial(req, 'claims_missing') - res.status(403).json({ error: 'Forbidden' }) + denyHostAccess(req, res, 'claims_missing') return } const authorization = await authorizeRpcHostAccess( @@ -199,8 +209,7 @@ export function createRpcAccessUsersRouter( directory ) if (!authorization.authorized) { - logHostAccessDenial(req, authorization.reason) - res.status(403).json({ error: 'Forbidden' }) + denyHostAccess(req, res, authorization.reason) return } res.status(200).json(authorization.connection) @@ -221,8 +230,7 @@ export function createRpcAccessUsersRouter( const claims = req.rpcAuth const binding = parseDirectRunBindingRequest(req.body) if (!claims) { - logHostAccessDenial(req, 'claims_missing') - res.status(403).json({ error: 'Forbidden' }) + denyHostAccess(req, res, 'claims_missing') return } if (!binding) { @@ -238,8 +246,7 @@ export function createRpcAccessUsersRouter( directory ) if (!authorization.authorized) { - logHostAccessDenial(req, authorization.reason) - res.status(403).json({ error: 'Forbidden' }) + denyHostAccess(req, res, authorization.reason) return } diff --git a/control-api/test/config.hostWakeRateLimit.test.ts b/control-api/test/config.hostWakeRateLimit.test.ts index b4ed3c88b8..f46f7218b3 100644 --- a/control-api/test/config.hostWakeRateLimit.test.ts +++ b/control-api/test/config.hostWakeRateLimit.test.ts @@ -30,15 +30,15 @@ import { readFileSync } from 'node:fs' // Hard-coded mirrors (source files named above). const WAKE_RETRIGGER_MS = 15_000 -const WAKE_MAX_HOLD_MS = 90_000 +const WAKE_MAX_HOLD_MS = 48_000 const RPC_PROXY_REPLICAS = 2 const PREWARM_CALLS_PER_DEVICE = 3 // 1 initial + PREWARM_REEMIT_MAX_ATTEMPTS const PREWARM_CONCURRENT_DEVICES_BUDGET = 4 // Worst case of the wake-and-hold mechanism alone, in raw calls: -// per instance: 1 initial + floor(maxHold / retrigger) retriggers = 7 -// fleet-wide: x replicas (per-instance dedup, no cross-instance dedup) -// per minute: 14 calls / 90s = ~9.33/min +// per instance: 1 initial + floor(maxHold / retrigger) retriggers = 4 +// fleet-wide: x replicas (per-instance dedup, no cross-instance dedup) = 8 +// per minute: 8 calls / 48s = 10.0/min const HOLD_CALLS_PER_INSTANCE = 1 + Math.floor(WAKE_MAX_HOLD_MS / WAKE_RETRIGGER_MS) const HOLD_CALLS_FLEET = HOLD_CALLS_PER_INSTANCE * RPC_PROXY_REPLICAS const MECHANISM_CALLS_PER_MIN = HOLD_CALLS_FLEET * (60_000 / WAKE_MAX_HOLD_MS) @@ -122,9 +122,9 @@ describe('host-wake rate limit budget (cross-service derivation guard)', () => { const config = await loadConfigWith({}) // Sanity-pin the derivation arithmetic itself. - expect(HOLD_CALLS_PER_INSTANCE).toBe(7) - expect(HOLD_CALLS_FLEET).toBe(14) - expect(MECHANISM_CALLS_PER_MIN).toBeCloseTo(9.333, 3) + expect(HOLD_CALLS_PER_INSTANCE).toBe(4) + expect(HOLD_CALLS_FLEET).toBe(8) + expect(MECHANISM_CALLS_PER_MIN).toBeCloseTo(10.0, 3) // Invariant 1: >=3x the single-hold mechanism volume, so the backstop // alone can never come close to exhausting the bucket. diff --git a/control-api/test/routes.mcp-host.renewal.refresh.rate-limit.test.ts b/control-api/test/routes.mcp-host.renewal.refresh.rate-limit.test.ts index 5fdfcaa0d6..086f972f32 100644 --- a/control-api/test/routes.mcp-host.renewal.refresh.rate-limit.test.ts +++ b/control-api/test/routes.mcp-host.renewal.refresh.rate-limit.test.ts @@ -1,4 +1,4 @@ -import { beforeEach, describe, expect, it, vi } from 'vitest' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import jwt from 'jsonwebtoken' import request from 'supertest' import { createApp } from '../src/app.js' @@ -63,6 +63,8 @@ vi.mock('../src/db.js', () => ({ describe('Workflow approval refresh rate limit', () => { let app: ReturnType + afterEach(() => vi.restoreAllMocks()) + beforeEach(() => { vi.clearAllMocks() rateLimitCounts.clear() @@ -72,6 +74,9 @@ describe('Workflow approval refresh rate limit', () => { it('does not consume a refresh token when the request is rate limited', async () => { const limit = config.approvalRlRefreshPerMin + // Pin the fixed-window clock before warm-up, including near a minute boundary. + const baseNow = Math.floor(Date.now() / 60_000) * 60_000 + 10_000 + const nowSpy = vi.spyOn(Date, 'now').mockReturnValue(baseNow) // Warm up exactly `limit` calls with unique refresh tokens — all should 200. for (let i = 0; i < limit; i++) { const { token } = issueMcpHostRefreshJwt('ns', 'recipe') @@ -94,8 +99,6 @@ describe('Workflow approval refresh rate limit', () => { // Rollover the 60s window so the limiter grants a fresh slot. The // rateLimiterService uses `Date.now()` internally — spy on it so we don't // actually wait a minute in the test. - const baseNow = Date.now() - const nowSpy = vi.spyOn(Date, 'now') nowSpy.mockReturnValue(baseNow + 60_001) try { diff --git a/control-api/test/routes.rpcAccessUsersMcpHost.test.ts b/control-api/test/routes.rpcAccessUsersMcpHost.test.ts index 4456436404..b380c7a0ea 100644 --- a/control-api/test/routes.rpcAccessUsersMcpHost.test.ts +++ b/control-api/test/routes.rpcAccessUsersMcpHost.test.ts @@ -1,9 +1,10 @@ -import { beforeEach, describe, expect, it, vi } from 'vitest' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import express from 'express' import request from 'supertest' import type { K8sGateway } from '../src/k8s.js' import { createRpcAccessUsersRouter } from '../src/routes/rpc-access/users.js' import { DirectRunBindingConflictError } from '../src/services/tracing/directRunAttributionBindingService.js' +import { signRpcAccessToken } from '../src/utils/auth/rpcAuthToken.js' /** * HTTP-level tests for the authorization gate on: @@ -45,39 +46,56 @@ vi.mock('../src/services/directory/index.js', () => svc) // token — the R2 per-session model-selector flow — is accepted, and that a // `mcp:servers:list`-only token is still rejected) run against the REAL // middleware + REAL signed tokens in routes.profile.test.ts. -vi.mock('../src/middleware/rpcAccessAuth.js', () => ({ - requireValidRpcAccessToken: - () => - ( - req: { params: Record; rpcAuth?: Record }, - _res: unknown, - next: () => void - ) => { - req.rpcAuth = { - ...req.rpcAuth, - sub: req.params.userId, - hostRefs: [req.params.hostRef], - } - next() - }, - requireValidRpcAccessTokenAny: - () => - ( - req: { params: Record; rpcAuth?: Record }, - _res: unknown, - next: () => void - ) => { +// +// The stub is steerable per test through `authState`, always reset in +// afterEach: `claims` overrides fields the stub would otherwise derive from the +// URL (so `sub` / `hostRefs` can be made to disagree with it), `omitClaims` +// lets the request through with no `rpcAuth` at all, and `realMiddleware` +// delegates to the REAL middleware so a genuine scope denial can be observed. +const authState = vi.hoisted(() => ({ + claims: {} as Record, + omitClaims: false, + realMiddleware: false, +})) + +vi.mock('../src/middleware/rpcAccessAuth.js', async importOriginal => { + const actual = await importOriginal() + type StubReq = { params: Record; rpcAuth?: Record } + const stubAuth = (req: StubReq, next: () => void) => { + if (!authState.omitClaims) { req.rpcAuth = { ...req.rpcAuth, sub: req.params.userId, hostRefs: [req.params.hostRef], + ...authState.claims, } - next() - }, - requireRpcTokenUserMatch: () => (_req: unknown, _res: unknown, next: () => void) => next(), - requireRpcTokenTeamMatch: () => (_req: unknown, _res: unknown, next: () => void) => next(), - requireRpcTokenHostMatch: () => (_req: unknown, _res: unknown, next: () => void) => next(), -})) + } + next() + } + return { + requireValidRpcAccessToken: + (...args: Parameters) => + (req: StubReq, res: unknown, next: () => void) => + authState.realMiddleware + ? actual.requireValidRpcAccessToken(...args)(req as never, res as never, next) + : stubAuth(req, next), + requireValidRpcAccessTokenAny: + (...args: Parameters) => + (req: StubReq, res: unknown, next: () => void) => + authState.realMiddleware + ? actual.requireValidRpcAccessTokenAny(...args)(req as never, res as never, next) + : stubAuth(req, next), + requireRpcTokenUserMatch: () => (_req: unknown, _res: unknown, next: () => void) => next(), + requireRpcTokenTeamMatch: () => (_req: unknown, _res: unknown, next: () => void) => next(), + requireRpcTokenHostMatch: () => (_req: unknown, _res: unknown, next: () => void) => next(), + } +}) + +afterEach(() => { + authState.claims = {} + authState.omitClaims = false + authState.realMiddleware = false +}) type HostCRD = { metadata: { name: string }; spec?: { enabled?: boolean } } @@ -107,6 +125,10 @@ function buildApp(hosts: HostCRD[], rpcAuth?: { teamId?: string }, bindingBudget return { app, gatewayStub, bindingService } } +// Pinned literally on purpose: rpc-proxy reads this exact header name, so the +// test must fail if the exported constant is ever renamed. +const DENIAL_HEADER = 'x-host-access-denial-reason' + const DIRECT_BINDING_BODY = { runId: '00000000-0000-4000-8000-000000000123', sessionId: 'session-a', @@ -187,6 +209,7 @@ describe('POST /rpc/access/users/:userId/mcp-hosts/:hostRef — atomic access an .post(`/rpc/access/users/${userId}/mcp-hosts/host-a`) .send(DIRECT_BINDING_BODY) .expect(403) + .expect(DENIAL_HEADER, 'directory_grant_missing') expect(gatewayStub.listResource).not.toHaveBeenCalled() expect(bindingService.bind).not.toHaveBeenCalled() @@ -344,6 +367,9 @@ describe('GET /rpc/access/users/:userId/mcp-hosts/:hostRef — CHECK #1 (user_ag .get(`/rpc/access/users/${userId}/mcp-hosts/${hostRef}`) .expect(403) + // The reason travels out of band so the body below can stay byte-identical. + expect(res.headers[DENIAL_HEADER]).toBe('directory_grant_missing') + // CRITICAL contract: the response body MUST be exactly {"error":"Forbidden"}. // The Desktop App error "No permitted scopes/hostRefs for requested RPC token" // is a CLIENT-SIDE translation of this 21-byte backend response. If you @@ -366,6 +392,7 @@ describe('GET /rpc/access/users/:userId/mcp-hosts/:hostRef — CHECK #1 (user_ag .get(`/rpc/access/users/user-has-chatllm-only/mcp-hosts/product`) .expect(403) expect(res.body).toEqual({ error: 'Forbidden' }) + expect(res.headers[DENIAL_HEADER]).toBe('directory_grant_missing') }) it("is an exact-string match (no substring bypass) — user with 'allinone' cannot access 'allinone2'", async () => { @@ -423,6 +450,7 @@ describe('GET /rpc/access/users/:userId/mcp-hosts/:hostRef — CHECK #2 (Host CR .get(`/rpc/access/users/user-orphan/mcp-hosts/researcher`) .expect(403) expect(res.body).toEqual({ error: 'Forbidden' }) + expect(res.headers[DENIAL_HEADER]).toBe('host_missing') }) it('returns 403 when the gateway returns an empty list of hosts', async () => { @@ -445,6 +473,7 @@ describe('GET /rpc/access/users/:userId/mcp-hosts/:hostRef — CHECK #3 (host mu ]) const res = await request(app).get(`/rpc/access/users/u/mcp-hosts/product`).expect(403) expect(res.body).toEqual({ error: 'Forbidden' }) + expect(res.headers[DENIAL_HEADER]).toBe('host_disabled') }) it('returns 200 when spec.enabled is undefined (default permissive, the common case)', async () => { @@ -537,6 +566,7 @@ describe('GET /rpc/access/users/:userId/mcp-hosts/:hostRef — team-level access .get(`/rpc/access/users/${userId}/mcp-hosts/${hostRef}`) .expect(403) expect(res.body).toEqual({ error: 'Forbidden' }) + expect(res.headers[DENIAL_HEADER]).toBe('directory_grant_missing') }) it('returns 403 when user_agents is empty AND no teamId is in the token', async () => { @@ -547,6 +577,7 @@ describe('GET /rpc/access/users/:userId/mcp-hosts/:hostRef — team-level access const res = await request(app).get(`/rpc/access/users/u/mcp-hosts/development`).expect(403) expect(res.body).toEqual({ error: 'Forbidden' }) + expect(res.headers[DENIAL_HEADER]).toBe('directory_grant_missing') // getTeamAgents must NOT be called when there is no teamId to look up expect(svc.getTeamAgents).not.toHaveBeenCalled() }) @@ -581,3 +612,177 @@ describe('Sibling endpoint sanity: GET /rpc/access/users/:userId/agents', () => expect(res.body).toEqual({ userId: 'u', agentNames: ['chatllm', 'product'] }) }) }) + +describe('host access denial reason header (rpc-proxy contract)', () => { + beforeEach(() => { + svc.getUserAgents.mockReset() + svc.getCurrentTeam.mockReset() + svc.getTeamAgents.mockReset() + }) + + const HOST_URL = (userId: string, hostRef: string) => + `/rpc/access/users/${userId}/mcp-hosts/${hostRef}` + const KNOWN_HOSTS: HostCRD[] = [{ metadata: { name: 'host-a' }, spec: { enabled: true } }] + + type DenialCase = { + reason: string + arrange: (userId: string) => { hosts: HostCRD[]; rpcAuth?: { teamId?: string } } + } + + // One case per control-api denial reason that reaches the handler. `arrange` + // wires the directory / claims / Host CRs so exactly that reason fires. + const DENIAL_CASES: DenialCase[] = [ + { + reason: 'directory_grant_missing', + arrange: userId => { + svc.getUserAgents.mockResolvedValue({ userId, agentNames: [] }) + return { hosts: KNOWN_HOSTS } + }, + }, + { + reason: 'team_membership_missing', + arrange: userId => { + svc.getUserAgents.mockResolvedValue({ userId, agentNames: [] }) + svc.getCurrentTeam.mockResolvedValue(null) + return { hosts: KNOWN_HOSTS, rpcAuth: { teamId: 'team-gone' } } + }, + }, + { + reason: 'subject_mismatch', + arrange: userId => { + svc.getUserAgents.mockResolvedValue({ userId, agentNames: ['host-a'] }) + authState.claims = { sub: 'someone-else' } + return { hosts: KNOWN_HOSTS } + }, + }, + { + reason: 'host_claim_missing', + arrange: userId => { + svc.getUserAgents.mockResolvedValue({ userId, agentNames: ['host-a'] }) + authState.claims = { hostRefs: ['other-host'] } + return { hosts: KNOWN_HOSTS } + }, + }, + { + reason: 'host_missing', + arrange: userId => { + svc.getUserAgents.mockResolvedValue({ userId, agentNames: ['host-a'] }) + return { hosts: [] } + }, + }, + { + reason: 'host_disabled', + arrange: userId => { + svc.getUserAgents.mockResolvedValue({ userId, agentNames: ['host-a'] }) + return { hosts: [{ metadata: { name: 'host-a' }, spec: { enabled: false } }] } + }, + }, + ] + + it.each(DENIAL_CASES)( + 'GET 403 carries the reason $reason in the header and keeps the 21-byte body', + async ({ reason, arrange }) => { + const userId = 'user-1' + const { hosts, rpcAuth } = arrange(userId) + const { app } = buildApp(hosts, rpcAuth) + + const res = await request(app).get(HOST_URL(userId, 'host-a')).expect(403) + + expect(res.headers[DENIAL_HEADER]).toBe(reason) + expect(res.body).toEqual({ error: 'Forbidden' }) + expect(Buffer.byteLength(res.text, 'utf-8')).toBe(21) + } + ) + + it.each(DENIAL_CASES)( + 'POST 403 carries the reason $reason in the header, keeps the 21-byte body and records no binding', + async ({ reason, arrange }) => { + const userId = 'user-1' + const { hosts, rpcAuth } = arrange(userId) + const { app, bindingService } = buildApp(hosts, rpcAuth) + + const res = await request(app) + .post(HOST_URL(userId, 'host-a')) + .send(DIRECT_BINDING_BODY) + .expect(403) + + expect(res.headers[DENIAL_HEADER]).toBe(reason) + expect(res.body).toEqual({ error: 'Forbidden' }) + expect(Buffer.byteLength(res.text, 'utf-8')).toBe(21) + // Witness for the negative: the route was entered and denied by the + // handler (header present above), and it never reached the binding step. + expect(bindingService.bind).not.toHaveBeenCalled() + } + ) + + it('GET without verified claims is denied with claims_missing', async () => { + authState.omitClaims = true + const { app } = buildApp(KNOWN_HOSTS) + + const res = await request(app).get(HOST_URL('user-1', 'host-a')).expect(403) + + expect(res.headers[DENIAL_HEADER]).toBe('claims_missing') + expect(Buffer.byteLength(res.text, 'utf-8')).toBe(21) + expect(svc.getUserAgents).not.toHaveBeenCalled() + }) + + it('POST without verified claims is denied with claims_missing', async () => { + authState.omitClaims = true + const { app, bindingService } = buildApp(KNOWN_HOSTS) + + const res = await request(app) + .post(HOST_URL('user-1', 'host-a')) + .send(DIRECT_BINDING_BODY) + .expect(403) + + expect(res.headers[DENIAL_HEADER]).toBe('claims_missing') + expect(Buffer.byteLength(res.text, 'utf-8')).toBe(21) + expect(bindingService.bind).not.toHaveBeenCalled() + }) + + it('a successful authorization carries no denial header', async () => { + svc.getUserAgents.mockResolvedValue({ userId: 'user-1', agentNames: ['host-a'] }) + const { app } = buildApp(KNOWN_HOSTS) + + const res = await request(app).get(HOST_URL('user-1', 'host-a')).expect(200) + + expect(res.body.hostRef).toBe('host-a') + expect(res.headers[DENIAL_HEADER]).toBeUndefined() + }) + + it('a middleware scope denial carries no denial header (only handler denials do)', async () => { + authState.realMiddleware = true + svc.getUserAgents.mockResolvedValue({ userId: 'user-1', agentNames: [] }) + const { app } = buildApp(KNOWN_HOSTS) + const tokenWith = (scopes: Array<'mcp:servers:list' | 'host:status:read'>, jti: string) => + signRpcAccessToken({ + sub: 'user-1', + typ: 'user', + teamId: 'team-1', + role: 'member', + scopes, + hostRefs: ['host-a'], + jti, + }) + + // Scope not in the route's allow-list: the REAL middleware answers 403. + const scopeDenied = await request(app) + .get(HOST_URL('user-1', 'host-a')) + .set('x-rpc-access-token', tokenWith(['mcp:servers:list'], 'rpc-jti-scope-denied')) + .expect(403) + expect(scopeDenied.body).toEqual({ error: 'Forbidden' }) + expect(scopeDenied.headers[DENIAL_HEADER]).toBeUndefined() + // Witness: the handler never ran for the scope denial. + expect(svc.getUserAgents).not.toHaveBeenCalled() + + // Same real middleware, allowed scope: the handler runs and its denial DOES + // carry the header, proving the absence above is the middleware's doing. + const handlerDenied = await request(app) + .get(HOST_URL('user-1', 'host-a')) + .set('x-rpc-access-token', tokenWith(['host:status:read'], 'rpc-jti-handler-denied')) + .expect(403) + // The token carries a teamId and the directory stub knows no such team. + expect(handlerDenied.headers[DENIAL_HEADER]).toBe('team_membership_missing') + expect(svc.getUserAgents).toHaveBeenCalledTimes(1) + }) +}) diff --git a/control-api/test/services.llmProviderAttemptAuthorizer.depth.test.ts b/control-api/test/services.llmProviderAttemptAuthorizer.depth.test.ts index c48245a902..1b538325e6 100644 --- a/control-api/test/services.llmProviderAttemptAuthorizer.depth.test.ts +++ b/control-api/test/services.llmProviderAttemptAuthorizer.depth.test.ts @@ -59,6 +59,19 @@ function nestedArrays(depth: number): string { return `{"x":${'['.repeat(depth)}${']'.repeat(depth)}}` } +/** Iterative (stack-free) depth of a JSON tree: the root container is level 1. */ +function nestingDepth(root: unknown): number { + let deepest = 0 + const pending: Array<{ value: unknown; level: number }> = [{ value: root, level: 1 }] + for (let next = pending.pop(); next !== undefined; next = pending.pop()) { + if (typeof next.value !== 'object' || next.value === null) continue + deepest = Math.max(deepest, next.level) + const children = Array.isArray(next.value) ? next.value : Object.values(next.value) + for (const child of children) pending.push({ value: child, level: next.level + 1 }) + } + return deepest +} + function nestedObjects(depth: number): string { return `${'{"n":'.repeat(depth - 1)}{}${'}'.repeat(depth - 1)}` } @@ -87,8 +100,10 @@ describe('authorizeLlmProviderAttempt nesting depth guard', () => { for (const depth of [5000, 150000]) { it(`rejects a ${depth}-deep ${p.provider} request with invalid_request, not a RangeError`, async () => { const body = wireBody(p, nestedArrays(depth)) - // Proves the fixture really is deep enough to overflow serialization. - if (depth === 150000) expect(() => JSON.stringify(body)).toThrow(RangeError) + // Whether V8's JSON.stringify overflows the stack at this depth is + // platform-dependent, so the fixture is verified by measuring its + // depth instead; the guard below is the platform-independent contract. + expect(nestingDepth(body)).toBeGreaterThanOrEqual(depth) const current = deps() const attempt = authorizeLlmProviderAttempt(claims(p.scope), body, current) await expect(attempt).rejects.toBeInstanceOf(LlmProviderAttemptAuthorizeError) @@ -100,6 +115,7 @@ describe('authorizeLlmProviderAttempt nesting depth guard', () => { it(`rejects deep nesting under an unknown ${p.provider} body key before serializing`, async () => { const body = wireBody(p, '{"type":"object"}', `,"extra":${nestedArrays(150000)}`) + expect(nestingDepth(body)).toBeGreaterThanOrEqual(150000) const current = deps() const attempt = authorizeLlmProviderAttempt(claims(p.scope), body, current) await expect(attempt).rejects.toMatchObject({ code: 'invalid_request' }) diff --git a/control-ui/components/HostWizard/index.tsx b/control-ui/components/HostWizard/index.tsx index 6aa970ab38..04a01edeb7 100644 --- a/control-ui/components/HostWizard/index.tsx +++ b/control-ui/components/HostWizard/index.tsx @@ -67,10 +67,6 @@ import { } from './constants' import type { CreatedResource, HostWizardProps, HostWizardValidationState } from './types' -// Stateless lifecycle support remains intact in the API and existing-agent UI; -// only creation through this wizard is temporarily unavailable. -const SHOW_STATELESS_AGENT_SELECTOR = false - // Asymmetric save gate (spec Topic 1b): the PRIMARY provider must be usable — // its required credential slot(s) filled — before create is allowed. Fallbacks // are optional and only warn, so they never enter this gate. @@ -977,50 +973,48 @@ export function HostWizard({
Namespace: {HOST_NAMESPACE}
- {SHOW_STATELESS_AGENT_SELECTOR ? ( -
- Agent type - - Stateless agents suspend after the idle window and wake on demand. Communication - channels keep stateless agents always-on unless the cluster explicitly enables - wake-on-interaction; desktop still requires a stateful agent. - -
-
)} diff --git a/control-ui/components/__tests__/GovernedTraceSurface.test.tsx b/control-ui/components/__tests__/GovernedTraceSurface.test.tsx index 1a984cab39..c42569fb9f 100644 --- a/control-ui/components/__tests__/GovernedTraceSurface.test.tsx +++ b/control-ui/components/__tests__/GovernedTraceSurface.test.tsx @@ -701,7 +701,8 @@ describe('GovernedTraceSurface', () => { expect(screen.getByText('30-day projection')).toBeInTheDocument() expect(screen.getByText(/This extrapolates persisted daily patterns/)).toBeInTheDocument() expect(screen.getByText('Variance rate')).toBeInTheDocument() - expect(screen.getByText(/33\.33%/)).toBeInTheDocument() + // Locale-independent: accept both decimal separators (33.33% and 33,33%). + expect(screen.getByText(/33[.,]33%/)).toBeInTheDocument() expect(screen.getByText('1h')).toBeInTheDocument() expect(screen.getByText('kube:unallocated')).toBeInTheDocument() expect(screen.getByText('pricing-export:2026-07-10:cpu')).toBeInTheDocument() diff --git a/control-ui/components/__tests__/HostWizard.test.tsx b/control-ui/components/__tests__/HostWizard.test.tsx index f18eed5f17..1805ce9030 100644 --- a/control-ui/components/__tests__/HostWizard.test.tsx +++ b/control-ui/components/__tests__/HostWizard.test.tsx @@ -840,14 +840,14 @@ describe('HostWizard — baseline render', () => { }) describe('HostWizard — Agent type (stateless lifecycle)', () => { - it('hides the Agent type selector from the create flow', async () => { + it('shows the Agent type selector in the create flow', async () => { await renderWizard() - expect(screen.queryByText('Agent type')).not.toBeInTheDocument() - expect(screen.queryByRole('radio', { name: /Stateful \(always on\)/i })).not.toBeInTheDocument() + expect(screen.getByText('Agent type')).toBeInTheDocument() + expect(screen.getByRole('radio', { name: /Stateful \(always on\)/i })).toBeInTheDocument() expect( - screen.queryByRole('radio', { name: /Stateless \(suspends when idle\)/i }) - ).not.toBeInTheDocument() + screen.getByRole('radio', { name: /Stateless \(suspends when idle\)/i }) + ).toBeInTheDocument() }) it('omits spec.lifecycle from the created Host when Stateful is kept (absent = disabled)', async () => { @@ -868,6 +868,24 @@ describe('HostWizard — Agent type (stateless lifecycle)', () => { expect('lifecycle' in payload.spec).toBe(false) expect('workflowControl' in payload.spec).toBe(false) }) + + it('sends spec.lifecycle.stateless when Stateless is selected', async () => { + await renderWizard() + fireEvent.click(screen.getByRole('radio', { name: /Stateless \(suspends when idle\)/i })) + await walkToAccessStep({ agentName: 'stateless-agent' }) + continueToConnectorsStep() + submitFromConnectorsStep() + + await waitFor(() => { + expect(api.apiSend).toHaveBeenCalledWith('POST', '/api/v1/admin/hosts', expect.any(Object)) + }) + const hostCall = vi + .mocked(api.apiSend) + .mock.calls.find(call => call[0] === 'POST' && call[1] === '/api/v1/admin/hosts') + expect(hostCall).toBeDefined() + const payload = hostCall![2] as { spec: { lifecycle?: { stateless?: boolean } } } + expect(payload.spec.lifecycle).toEqual({ stateless: true }) + }) }) describe('HostWizard — create-only seam + compensation (R5-C1/R5-B1, V-1, V-7)', () => { diff --git a/control-ui/lib/__tests__/budgets.test.ts b/control-ui/lib/__tests__/budgets.test.ts index e3998b377b..226227934b 100644 --- a/control-ui/lib/__tests__/budgets.test.ts +++ b/control-ui/lib/__tests__/budgets.test.ts @@ -56,7 +56,8 @@ describe('formatBudgetScope', () => { describe('formatBudgetAmount', () => { it('formats cost amounts with the budget currency', () => { - expect(formatBudgetAmount(12.5, 'cost', 'USD')).toContain('12.5') + // Locale-independent: accept both decimal separators (12.5 and 12,5). + expect(formatBudgetAmount(12.5, 'cost', 'USD')).toMatch(/12[.,]5/) expect(formatBudgetAmount(12.5, 'cost', 'USD')).toMatch(/\$|USD/) }) diff --git a/control-ui/package-lock.json b/control-ui/package-lock.json index c1d49250d0..6cf07b947f 100644 --- a/control-ui/package-lock.json +++ b/control-ui/package-lock.json @@ -1,12 +1,12 @@ { "name": "clerum-control-ui", - "version": "0.1.662", + "version": "0.1.663", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "clerum-control-ui", - "version": "0.1.662", + "version": "0.1.663", "dependencies": { "@clerum/display-field": "file:../packages/display-field", "@clerum/frontend-components": "file:../packages/frontend-components", diff --git a/control-ui/package.json b/control-ui/package.json index e75746e40c..42dfc7727b 100644 --- a/control-ui/package.json +++ b/control-ui/package.json @@ -1,6 +1,6 @@ { "name": "clerum-control-ui", - "version": "0.1.662", + "version": "0.1.663", "private": true, "scripts": { "dev": "NEXT_IGNORE_INCORRECT_LOCKFILE=1 next dev", diff --git a/deploy/base/control-plane/host-context-controller.yaml b/deploy/base/control-plane/host-context-controller.yaml index d6c0fa1e54..ec7a3d9f7a 100644 --- a/deploy/base/control-plane/host-context-controller.yaml +++ b/deploy/base/control-plane/host-context-controller.yaml @@ -76,6 +76,16 @@ spec: value: "8081" - name: CONTEXT_MAPPER_NAMESPACE value: mcp-server + # Product baseline for stateless Hosts. HCC uses the effective idle window + # max(idleMinutes, idleFloorMinutes), then drains active work before scale-to-zero. + - name: CONTEXT_MAPPER_STATELESS_IDLE_MINUTES + value: "30" + - name: CONTEXT_MAPPER_STATELESS_IDLE_FLOOR_MINUTES + value: "15" + - name: CONTEXT_MAPPER_STATELESS_DRAIN_GRACE_MS + value: "60000" + - name: CONTEXT_MAPPER_HEARTBEAT_POLL_MS + value: "10000" - name: CONTEXT_MAPPER_HOST_NAMESPACE value: mcp-host - name: CONTEXT_MAPPER_HOST_IMAGE diff --git a/deploy/overlays/minikube/configmaps/rpc-proxy-config.yaml b/deploy/overlays/minikube/configmaps/rpc-proxy-config.yaml index f873193e03..153c36aa1a 100644 --- a/deploy/overlays/minikube/configmaps/rpc-proxy-config.yaml +++ b/deploy/overlays/minikube/configmaps/rpc-proxy-config.yaml @@ -35,6 +35,10 @@ data: RPC_PROXY_UPSTREAM_TIMEOUT_MS: "60000" RPC_PROXY_MAX_TOKEN_LENGTH: "4096" RPC_PROXY_ARTIFACT_DOWNLOAD_MAX_MB: "250" + # Body-transfer budget of one artifact download (the header wait is bounded by + # RPC_PROXY_UPSTREAM_TIMEOUT_MS). 300 s moves the 250 MB cap above at a + # sustained 0.84 MB/s; the unset default (60 s) would need 4.2 MB/s. + RPC_PROXY_ARTIFACT_DOWNLOAD_TIMEOUT_MS: "300000" RPC_PROXY_ALLOWED_METHOD_PATTERN: "^[a-zA-Z0-9_./:-]{1,120}$" RPC_PROXY_STREAM_MAX_LIFETIME_MS: "600000" RPC_PROXY_STREAM_INTERVAL_MS: "3000" diff --git a/deploy/overlays/minikube/kustomization.yaml b/deploy/overlays/minikube/kustomization.yaml index 3e20987799..912e2dee34 100644 --- a/deploy/overlays/minikube/kustomization.yaml +++ b/deploy/overlays/minikube/kustomization.yaml @@ -83,6 +83,7 @@ images: newTag: test patchesStrategicMerge: + - patches/hcc-stateless-lifecycle.yaml - patches/hcc-policy-resync.yaml - patches/resource-limits.yaml - patches/dynamic-images.yaml diff --git a/deploy/overlays/minikube/patches/hcc-stateless-lifecycle.yaml b/deploy/overlays/minikube/patches/hcc-stateless-lifecycle.yaml new file mode 100644 index 0000000000..79bba8029a --- /dev/null +++ b/deploy/overlays/minikube/patches/hcc-stateless-lifecycle.yaml @@ -0,0 +1,22 @@ +# Local Minikube mirrors the product stateless idle policy explicitly. This is +# also the baseline that pre-gate reconciliation restores after an accelerated +# lifecycle test mutates the live Deployment. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: host-context-controller + namespace: control-plane +spec: + template: + spec: + containers: + - name: host-context-controller + env: + - name: CONTEXT_MAPPER_STATELESS_IDLE_MINUTES + value: '30' + - name: CONTEXT_MAPPER_STATELESS_IDLE_FLOOR_MINUTES + value: '15' + - name: CONTEXT_MAPPER_STATELESS_DRAIN_GRACE_MS + value: '60000' + - name: CONTEXT_MAPPER_HEARTBEAT_POLL_MS + value: '10000' diff --git a/desktop-app/src/__tests__/appService.bootstrap.test.ts b/desktop-app/src/__tests__/appService.bootstrap.test.ts index ba2eb86c83..e4a5f21d5e 100644 --- a/desktop-app/src/__tests__/appService.bootstrap.test.ts +++ b/desktop-app/src/__tests__/appService.bootstrap.test.ts @@ -132,7 +132,7 @@ describe('AppService invitation configuration lookup', () => { expect(bindChatStoreForUser).toHaveBeenCalledWith( 'user-1', resolveEnvKey('https://api.example.com', 'https://rpc.example.com'), - { legacyEnvKeys: [getActiveLegacyRestOnlyEnvKey()] } + { legacyEnvKeys: [getActiveLegacyRestOnlyEnvKey()], teamId: 'team-1' } ) expect(config.externalRestApiBaseUrl).toBe('https://api.example.com') expect(config.rpcProxyBaseUrl).toBe('https://rpc.example.com') diff --git a/desktop-app/src/__tests__/appService.chatStoreTeamHop.test.ts b/desktop-app/src/__tests__/appService.chatStoreTeamHop.test.ts new file mode 100644 index 0000000000..7d933b9941 --- /dev/null +++ b/desktop-app/src/__tests__/appService.chatStoreTeamHop.test.ts @@ -0,0 +1,235 @@ +/** + * A transient `runWithTeamContext` hop installs another team's token (and `me`) + * for the duration of one operation and then restores the original team. The + * chat store scope and the delete-fence authority must NOT follow that hop: a + * confirmed deletion is bound to the store's scope, and a store rebound to the + * hop team would accept a late append to a chat the user already deleted. + */ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' +import { AppService } from '../appService.js' +import { + __setChatStoreBaseDirForTests, + requireChatStore, + unbindChatStore, +} from '../chatStoreBinding.js' +import type { ChatMessage } from '../types.js' + +let base: string + +beforeEach(async () => { + base = await fs.mkdtemp(path.join(os.tmpdir(), 'chat-store-team-hop-')) + __setChatStoreBaseDirForTests(base) +}) + +afterEach(async () => { + unbindChatStore() + __setChatStoreBaseDirForTests(null) + await fs.rm(base, { recursive: true, force: true }) +}) + +const member = (teamId: string) => ({ + id: 'user-1', + email: 'user@example.test', + name: 'Test User', + picture: null, + teamId, + teamName: teamId, + role: 'member', +}) + +/** An AppService signed in on team-a whose auth client answers team switches. */ +function signedInService(getMeAnswers: string[]): AppService { + const service = new AppService() as unknown as Record + service.sessionToken = 'team-a-token' + service.me = member('team-a') + const getMe = vi.fn() + for (const teamId of getMeAnswers) getMe.mockResolvedValueOnce(member(teamId)) + service.authClient = { + getMe, + switchTeam: vi.fn(async (_token: string, teamId: string) => ({ + token: `${teamId}-token`, + team: { id: teamId, name: teamId, role: 'member' }, + })), + } + service.tokenStore = { setSessionToken: vi.fn() } + return service as unknown as AppService +} + +const lateMessage = (): ChatMessage => ({ + id: 'late', + role: 'user', + content: 'late append', + timestamp: Date.now(), +}) + +describe('chat store scope across a transient team hop', () => { + it('keeps the store scope and the delete authority on the home team for the whole hop', async () => { + const service = signedInService(['team-b', 'team-a']) + const internals = service as unknown as { + bindCurrentChatStore(userId: string): Promise + runWithTeamContext(teamId: string, op: (token: string) => Promise): Promise + me: { teamId: string } + } + await internals.bindCurrentChatStore('user-1') + const store = requireChatStore() + const fence = service.getChatDeletionFenceAuthority() + await store.createChat('agent-1', 'c1') + await store.deleteChat('agent-1', 'c1', fence.authorityScope) + // Liveness witness: the deletion took effect in team-a. + expect(await store.loadMessages('agent-1', 'c1')).toEqual([]) + + const during: { sessionTeam?: string; store?: unknown; fence?: unknown; loaded?: string[] } = {} + await internals.runWithTeamContext('team-b', async () => { + during.sessionTeam = internals.me.teamId + during.store = store.getAuthorityScope() + during.fence = service.getChatDeletionFenceAuthority().authorityScope + await store.appendMessages('agent-1', 'c1', [lateMessage()]) + during.loaded = (await store.loadMessages('agent-1', 'c1')).map(m => m.id) + }) + + // Liveness witness: the session really was on team-b during the operation. + expect(during.sessionTeam).toBe('team-b') + expect(during.store).toMatchObject({ userId: 'user-1', teamId: 'team-a' }) + expect(during.fence).toMatchObject({ userId: 'user-1', teamId: 'team-a' }) + // The late append to the deleted chat was rejected, not stored. + expect(during.loaded).toEqual([]) + // Restored: same team, and the chat is still gone. + expect(internals.me.teamId).toBe('team-a') + expect(store.getAuthorityScope()?.teamId).toBe('team-a') + expect(service.getChatDeletionFenceAuthority().authorityScope.teamId).toBe('team-a') + expect(await store.loadMessages('agent-1', 'c1')).toEqual([]) + }) + + it('releases the home-team pin once the hop completes so the store follows the session again', async () => { + const service = signedInService(['team-b', 'team-a', 'team-c']) + const internals = service as unknown as { + bindCurrentChatStore(userId: string): Promise + runWithTeamContext(teamId: string, op: (token: string) => Promise): Promise + switchSessionToTeam(teamId: string): Promise + chatStoreTeamId(): string | null + me: { teamId: string } + } + await internals.bindCurrentChatStore('user-1') + + const during: { sessionTeam?: string; chatStoreTeam?: string | null } = {} + await internals.runWithTeamContext('team-b', async () => { + during.sessionTeam = internals.me.teamId + during.chatStoreTeam = internals.chatStoreTeamId() + }) + + // Liveness witness: the pin was really held during the hop (the session was on + // team-b while the store team stayed on team-a) and the hop restored team-a. + expect(during).toEqual({ sessionTeam: 'team-b', chatStoreTeam: 'team-a' }) + expect(internals.me.teamId).toBe('team-a') + expect(internals.chatStoreTeamId()).toBe('team-a') + + // A genuine team switch after the hop: the store team follows `me.teamId`, it + // is not stuck on the team the finished hop was pinned to. + await internals.switchSessionToTeam('team-c') + + expect(internals.me.teamId).toBe('team-c') + expect(internals.chatStoreTeamId()).toBe('team-c') + expect(requireChatStore().getAuthorityScope()?.teamId).toBe('team-c') + expect(service.getChatDeletionFenceAuthority().authorityScope.teamId).toBe('team-c') + }) + + describe('when the switch back to the home team fails', () => { + type HopInternals = { + bindCurrentChatStore(userId: string): Promise + runWithTeamContext(teamId: string, op: (token: string) => Promise): Promise + me: { teamId: string } + } + + /** team-a service whose auth client accepts the hop to team-b and refuses the way back. */ + function serviceFailingToRestore(): { service: AppService; internals: HopInternals } { + const service = signedInService(['team-b']) + const authClient = (service as unknown as { authClient: { switchTeam: unknown } }).authClient + authClient.switchTeam = vi.fn(async (_token: string, teamId: string) => { + if (teamId === 'team-a') throw new Error('network down') + return { token: `${teamId}-token`, team: { id: teamId, name: teamId, role: 'member' } } + }) + return { service, internals: service as unknown as HopInternals } + } + + it('rebinds the store to the team the session is really on so the delete fence still matches', async () => { + const { service, internals } = serviceFailingToRestore() + await internals.bindCurrentChatStore('user-1') + const store = requireChatStore() + await store.createChat('agent-1', 'c1') + + const during: { store?: string | null; fence?: string | null } = {} + await expect( + internals.runWithTeamContext('team-b', async () => { + during.store = store.getAuthorityScope()?.teamId + during.fence = service.getChatDeletionFenceAuthority().authorityScope.teamId + return 'done' + }) + ).rejects.toThrow('network down') + + // Liveness witness: the hop really happened (store and fence stayed on + // team-a while the operation ran) and the switch back really failed + // (the session is left on team-b). + expect(during).toEqual({ store: 'team-a', fence: 'team-a' }) + expect(internals.me.teamId).toBe('team-b') + // The pin is gone and the store follows the session again. + const fence = service.getChatDeletionFenceAuthority() + expect(fence.authorityScope.teamId).toBe('team-b') + expect(requireChatStore().getAuthorityScope()?.teamId).toBe('team-b') + // A deletion authorized by that fence is accepted instead of throwing + // "Chat authority scope changed". + await requireChatStore().createChat('agent-1', 'c2') + await requireChatStore().deleteChat('agent-1', 'c2', fence.authorityScope) + expect(await requireChatStore().loadMessages('agent-1', 'c2')).toEqual([]) + }) + + it('logs a failed rebind and keeps the error of the failed operation', async () => { + const { internals } = serviceFailingToRestore() + await internals.bindCurrentChatStore('user-1') + const realBind = internals.bindCurrentChatStore.bind(internals) + let hopFinished = false + const rebindAttempts: string[] = [] + internals.bindCurrentChatStore = vi.fn(async (userId: string) => { + if (!hopFinished) return realBind(userId) + rebindAttempts.push(userId) + throw new Error('rebind exploded') + }) + const errorLog = vi.spyOn(console, 'error').mockImplementation(() => undefined) + + try { + await expect( + internals.runWithTeamContext('team-b', async () => { + hopFinished = true + throw new Error('operation failed') + }) + ).rejects.toThrow('operation failed') + + // Liveness witness: the rebind ran and failed, and it was reported. + expect(rebindAttempts).toEqual(['user-1']) + expect(errorLog).toHaveBeenCalledWith( + expect.stringContaining('Failed to rebind the chat store'), + expect.objectContaining({ message: 'rebind exploded' }) + ) + } finally { + errorLog.mockRestore() + } + }) + }) + + it('still rebinds the store when the team changes without a restoring hop', async () => { + const service = signedInService(['team-b']) + const internals = service as unknown as { + bindCurrentChatStore(userId: string): Promise + switchSessionToTeam(teamId: string): Promise + } + await internals.bindCurrentChatStore('user-1') + expect(requireChatStore().getAuthorityScope()?.teamId).toBe('team-a') + + await internals.switchSessionToTeam('team-b') + + expect(requireChatStore().getAuthorityScope()?.teamId).toBe('team-b') + expect(service.getChatDeletionFenceAuthority().authorityScope.teamId).toBe('team-b') + }) +}) diff --git a/desktop-app/src/__tests__/appService.prewarm.test.ts b/desktop-app/src/__tests__/appService.prewarm.test.ts index 4c124a8c09..2fb75eeaad 100644 --- a/desktop-app/src/__tests__/appService.prewarm.test.ts +++ b/desktop-app/src/__tests__/appService.prewarm.test.ts @@ -7,6 +7,8 @@ import { ApiError } from '../httpClient.js' // ── mock dependencies before AppService is imported ────────────────────────── vi.mock('../config.js', () => ({ + getActiveEnvKey: () => 'test-env', + getActiveLegacyEnvKeys: () => [], config: { rpcProxyBaseUrl: 'http://proxy', externalRestApiBaseUrl: 'http://rest', @@ -258,6 +260,252 @@ describe('AppService.prewarmHost — bounded wake re-emission', () => { infoSpy.mockRestore() }) + it('logout cancels pending re-emissions and does not leak cooldown to the next session', async () => { + mockPrewarmHost + .mockResolvedValueOnce({ status: 'wake-requested' }) + .mockResolvedValueOnce({ status: 'active' }) + const svc = makeService() + vi.spyOn( + svc as unknown as { suspendDesktopGfsUploadsForAuthBoundary: () => Promise }, + 'suspendDesktopGfsUploadsForAuthBoundary' + ).mockResolvedValue() + + await expect(svc.prewarmHost('chatllm')).resolves.toEqual({ + requested: true, + status: 'wake-requested', + }) + expect(vi.getTimerCount()).toBe(1) + + await svc.logout() + + expect(vi.getTimerCount()).toBe(0) + await vi.advanceTimersByTimeAsync(30_000) + expect(mockPrewarmHost).toHaveBeenCalledTimes(1) + ;(svc as unknown as { sessionToken: string }).sessionToken = 'next-session' + ;(svc as unknown as { me: unknown }).me = { id: 2, teamId: 'team-2' } + await expect(svc.prewarmHost('chatllm')).resolves.toEqual({ + requested: true, + status: 'active', + }) + expect(mockPrewarmHost).toHaveBeenCalledTimes(2) + expect(vi.getTimerCount()).toBe(0) + }) + + it('a late prewarm result cannot erase the next session cooldown', async () => { + const grant = await mockGetOrIssue() + mockGetOrIssue.mockClear() + let complete!: (value: typeof grant) => void + const pending = new Promise(resolve => { + complete = resolve + }) + mockGetOrIssue.mockReturnValueOnce(pending) + mockPrewarmHost.mockResolvedValue({ status: 'active' }) + const svc = makeService() + vi.spyOn( + svc as unknown as { suspendDesktopGfsUploadsForAuthBoundary: () => Promise }, + 'suspendDesktopGfsUploadsForAuthBoundary' + ).mockResolvedValue() + + const oldAttempt = svc.prewarmHost('chatllm') + for (let i = 0; i < 10 && mockGetOrIssue.mock.calls.length === 0; i++) await Promise.resolve() + expect(mockGetOrIssue).toHaveBeenCalledTimes(1) + await svc.logout() + ;(svc as unknown as { sessionToken: string }).sessionToken = 'next-session' + ;(svc as unknown as { me: unknown }).me = { id: 2, teamId: 'team-2' } + + await expect(svc.prewarmHost('chatllm')).resolves.toEqual({ + requested: true, + status: 'active', + }) + complete(grant) + await expect(oldAttempt).resolves.toEqual({ requested: false, skipped: 'auth-changed' }) + await expect(svc.prewarmHost('chatllm')).resolves.toEqual({ + requested: false, + skipped: 'cooldown', + }) + expect(mockPrewarmHost).toHaveBeenCalledTimes(1) + }) + + it('a pending explicit team switch blocks prewarm and a failed switch releases the guard', async () => { + let rejectSwitch!: (error: Error) => void + const pendingSwitch = new Promise((_resolve, reject) => { + rejectSwitch = reject + }) + const svc = makeService() + vi.spyOn( + svc as unknown as { switchSessionToTeam: () => Promise }, + 'switchSessionToTeam' + ).mockReturnValue(pendingSwitch) + mockPrewarmHost.mockResolvedValue({ status: 'active' }) + + await expect(svc.prewarmHost('first-host')).resolves.toEqual({ + requested: true, + status: 'active', + }) + const switching = svc.switchTeam('team-2') + await expect(svc.prewarmHost('chatllm')).resolves.toEqual({ + requested: false, + skipped: 'auth-changed', + }) + expect(mockPrewarmHost).toHaveBeenCalledTimes(1) + + rejectSwitch(new Error('switch canceled')) + await expect(switching).rejects.toThrow('switch canceled') + await expect(svc.prewarmHost('chatllm')).resolves.toEqual({ + requested: true, + status: 'active', + }) + expect(mockPrewarmHost).toHaveBeenCalledTimes(2) + }) + + it('a transient session revision does not cancel the bounded wake loop', async () => { + mockPrewarmHost + .mockResolvedValueOnce({ status: 'wake-requested' }) + .mockResolvedValueOnce({ status: 'active' }) + const svc = makeService() + + await svc.prewarmHost('chatllm') + ;(svc as unknown as { sessionGeneration: number }).sessionGeneration += 1 + await vi.advanceTimersByTimeAsync(10_000) + + expect(mockPrewarmHost).toHaveBeenCalledTimes(2) + expect(vi.getTimerCount()).toBe(0) + }) + + it('an internal team hop preserves the bounded wake loop and cooldown', async () => { + mockPrewarmHost + .mockResolvedValueOnce({ status: 'wake-requested' }) + .mockResolvedValueOnce({ status: 'active' }) + const svc = makeService() + const switchSessionToTeam = vi + .spyOn( + svc as unknown as { switchSessionToTeam: () => Promise }, + 'switchSessionToTeam' + ) + .mockImplementation(async () => (svc as unknown as { sessionToken: string }).sessionToken) + let finishOperation!: () => void + const operation = new Promise(resolve => { + finishOperation = resolve + }) + let markOperationStarted!: () => void + const operationStarted = new Promise(resolve => { + markOperationStarted = resolve + }) + + await svc.prewarmHost('chatllm') + const hop = ( + svc as unknown as { + runWithTeamContext: (teamId: string, operation: () => Promise) => Promise + } + ).runWithTeamContext('team-2', () => { + markOperationStarted() + return operation + }) + await operationStarted + expect(switchSessionToTeam).toHaveBeenCalledWith('team-2', expect.any(String)) + expect( + ( + svc as unknown as { prewarmAttemptAtByHostRef: Map } + ).prewarmAttemptAtByHostRef.has('chatllm') + ).toBe(true) + await vi.advanceTimersByTimeAsync(10_000) + expect(mockPrewarmHost).toHaveBeenCalledTimes(2) + expect(vi.getTimerCount()).toBe(0) + + finishOperation() + await hop + expect(switchSessionToTeam).toHaveBeenCalledWith('team-1', expect.any(String)) + await expect(svc.prewarmHost('chatllm')).resolves.toEqual({ + requested: false, + skipped: 'cooldown', + }) + }) + + it('nested explicit transitions keep prewarm fenced until both rejected switches settle', async () => { + const rejectSwitches: Array<(error: Error) => void> = [] + const svc = makeService() + vi.spyOn( + svc as unknown as { switchSessionToTeam: () => Promise }, + 'switchSessionToTeam' + ).mockImplementation( + () => + new Promise((_resolve, reject) => { + rejectSwitches.push(reject) + }) + ) + mockPrewarmHost.mockResolvedValue({ status: 'active' }) + + const firstSwitch = svc.switchTeam('team-2') + const secondSwitch = svc.switchTeam('team-3') + expect(rejectSwitches).toHaveLength(2) + await expect(svc.prewarmHost('chatllm')).resolves.toEqual({ + requested: false, + skipped: 'auth-changed', + }) + + rejectSwitches[0](new Error('first switch rejected')) + await expect(firstSwitch).rejects.toThrow('first switch rejected') + await expect(svc.prewarmHost('chatllm')).resolves.toEqual({ + requested: false, + skipped: 'auth-changed', + }) + expect(mockPrewarmHost).not.toHaveBeenCalled() + + rejectSwitches[1](new Error('second switch rejected')) + await expect(secondSwitch).rejects.toThrow('second switch rejected') + await expect(svc.prewarmHost('chatllm')).resolves.toEqual({ + requested: true, + status: 'active', + }) + expect(mockPrewarmHost).toHaveBeenCalledTimes(1) + }) + + it('a late prior response cannot clear the new session wake loop', async () => { + let completeOld!: (value: { status: string }) => void + const pendingOld = new Promise<{ status: string }>(resolve => { + completeOld = resolve + }) + mockPrewarmHost + .mockResolvedValueOnce({ status: 'wake-requested' }) + .mockReturnValueOnce(pendingOld) + .mockResolvedValueOnce({ status: 'wake-requested' }) + .mockResolvedValueOnce({ status: 'active' }) + const svc = makeService() + vi.spyOn( + svc as unknown as { suspendDesktopGfsUploadsForAuthBoundary: () => Promise }, + 'suspendDesktopGfsUploadsForAuthBoundary' + ).mockResolvedValue() + + await svc.prewarmHost('chatllm') + await vi.advanceTimersByTimeAsync(10_000) + expect(mockPrewarmHost).toHaveBeenCalledTimes(2) + await svc.logout() + ;(svc as unknown as { sessionToken: string }).sessionToken = 'next-session' + ;(svc as unknown as { me: unknown }).me = { id: 2, teamId: 'team-2' } + await svc.prewarmHost('chatllm') + + completeOld({ status: 'active' }) + await Promise.resolve() + await Promise.resolve() + const loops = (svc as unknown as { prewarmReemitLoopHostRefs: Set }) + .prewarmReemitLoopHostRefs + expect(loops.has('chatllm')).toBe(true) + await expect(svc.prewarmHost('chatllm')).resolves.toEqual({ + requested: false, + skipped: 'cooldown', + }) + ;( + svc as unknown as { prewarmAttemptAtByHostRef: Map } + ).prewarmAttemptAtByHostRef.delete('chatllm') + await expect(svc.prewarmHost('chatllm')).resolves.toEqual({ + requested: false, + skipped: 'in-flight', + }) + await vi.advanceTimersByTimeAsync(10_000) + expect(mockPrewarmHost).toHaveBeenCalledTimes(4) + expect(loops.has('chatllm')).toBe(false) + }) + it('immediate 200 or 409 schedules no re-emission at all', async () => { mockPrewarmHost.mockResolvedValueOnce({ status: 'active' }) const svc = makeService() diff --git a/desktop-app/src/__tests__/chatStore.test.ts b/desktop-app/src/__tests__/chatStore.test.ts index 7c78ed7a0c..08742a4bed 100644 --- a/desktop-app/src/__tests__/chatStore.test.ts +++ b/desktop-app/src/__tests__/chatStore.test.ts @@ -8,10 +8,17 @@ import type { ChatMessage } from '../types.js' let tempDir: string let store: ChatStore +const TEAM_A_SCOPE = { + environmentKey: 'env-a', + userId: 'user-a', + teamId: 'team-a', +} as const +const TEAM_B_SCOPE = { ...TEAM_A_SCOPE, teamId: 'team-b' } as const beforeEach(async () => { tempDir = await fs.mkdtemp(join(tmpdir(), 'chatstore-test-')) store = new ChatStore(tempDir) + store.setAuthorityScope(TEAM_A_SCOPE) }) afterEach(async () => { @@ -178,12 +185,63 @@ describe('renameChat', () => { // ── deleteChat ─────────────────────────────────────────────────────────────── describe('deleteChat', () => { + it('persists a confirmed deletion across a new store instance', async () => { + await store.createChat('agent-1', 'deleted-session') + await store.createChat('agent-1', 'kept-session') + await store.deleteChat('agent-1', 'deleted-session', TEAM_A_SCOPE) + + const restartedStore = new ChatStore(tempDir) + expect((await restartedStore.listChats('agent-1')).map(chat => chat.id)).toEqual([ + 'kept-session', + ]) + expect((await restartedStore.getIndex('agent-1')).deletedChatTombstones).toContainEqual({ + chatId: 'deleted-session', + authorityScope: TEAM_A_SCOPE, + }) + }) + + it('does not recreate a deleted transcript from queued or restarted writes', async () => { + const agentRef = 'agent-1' + const chatId = 'deleted-while-sending' + const message: ChatMessage = { + id: 'late-message', + role: 'user', + content: 'arrived after delete', + timestamp: 1, + } + await store.createChat(agentRef, chatId) + await store.saveMessages(agentRef, chatId, [ + { id: 'original', role: 'user', content: 'original', timestamp: 0 }, + ]) + + // Invocation order is the per-chat serialization order: both writes must + // observe the tombstone published by the preceding confirmed delete. + const deletion = store.deleteChat(agentRef, chatId, TEAM_A_SCOPE) + const queuedAppend = store.appendMessages(agentRef, chatId, [message]) + const queuedReplace = store.replaceMessages(agentRef, chatId, [message]) + await Promise.all([deletion, queuedAppend, queuedReplace]) + + const restartedStore = new ChatStore(tempDir) + await restartedStore.appendMessages(agentRef, chatId, [message]) + await restartedStore.replaceMessages(agentRef, chatId, [message]) + await expect(restartedStore.createChat(agentRef, chatId)).rejects.toThrow( + 'Chat was deleted locally' + ) + expect(await restartedStore.loadMessages(agentRef, chatId)).toEqual([]) + expect(await restartedStore.listChats(agentRef)).toEqual([]) + expect((await restartedStore.getIndex(agentRef)).deletedChatTombstones).toContainEqual({ + chatId, + authorityScope: TEAM_A_SCOPE, + }) + await expect(fs.access(chatCacheDir(chatId))).rejects.toThrow() + }) + it('removes from index and deletes message cache', async () => { await store.createChat('agent-1', 'del-1') await store.saveMessages('agent-1', 'del-1', [ { id: 'm1', role: 'user', content: 'hello', timestamp: Date.now() }, ]) - await store.deleteChat('agent-1', 'del-1') + await store.deleteChat('agent-1', 'del-1', TEAM_A_SCOPE) const chats = await store.listChats('agent-1') expect(chats).toHaveLength(0) @@ -194,10 +252,875 @@ describe('deleteChat', () => { it('clears lastActiveChatId when deleting active chat', async () => { await store.createChat('agent-1', 'active-1') await store.setLastActiveChatId('agent-1', 'active-1') - await store.deleteChat('agent-1', 'active-1') + await store.deleteChat('agent-1', 'active-1', TEAM_A_SCOPE) const lastActive = await store.getLastActiveChatId('agent-1') expect(lastActive).toBeNull() }) + + it('keeps a confirmed deletion across a session-team rebind for the same user and Host', async () => { + const agentRef = 'agent-1' + const chatId = 'team-owned-chat' + const message: ChatMessage = { id: 'late', role: 'user', content: 'late', timestamp: 1 } + await store.createChat(agentRef, chatId) + await store.saveMessages(agentRef, chatId, [ + { id: 'original', role: 'user', content: 'original', timestamp: 0 }, + ]) + // Liveness witness: the chat existed on disk before the delete. + await expect(fs.access(chatCacheDir(chatId))).resolves.toBeUndefined() + await store.deleteChat(agentRef, chatId, TEAM_A_SCOPE) + await expect(store.createChat(agentRef, chatId)).rejects.toThrow('Chat was deleted locally') + expect(await store.loadMessages(agentRef, chatId)).toEqual([]) + + // The store scope is rebound from `me.teamId` on every team switch. + store.setAuthorityScope(TEAM_B_SCOPE) + + await expect(store.createChat(agentRef, chatId)).rejects.toThrow('Chat was deleted locally') + await store.appendMessages(agentRef, chatId, [message]) + await store.replaceMessages(agentRef, chatId, [message]) + await store.setLastActiveChatId(agentRef, chatId) + expect(await store.loadMessages(agentRef, chatId)).toEqual([]) + expect(await store.getLastActiveChatId(agentRef)).toBeNull() + expect(await store.listChats(agentRef)).toEqual([]) + await expect(fs.access(chatCacheDir(chatId))).rejects.toThrow() + }) + + it('binds a tombstone to environment and user, not to the session team', async () => { + await store.createChat('agent-1', 'identity-chat') + await store.deleteChat('agent-1', 'identity-chat', TEAM_A_SCOPE) + + // The persisted tombstone keeps its full authority scope (format unchanged). + expect((await store.getIndex('agent-1')).deletedChatTombstones).toContainEqual({ + chatId: 'identity-chat', + authorityScope: TEAM_A_SCOPE, + }) + // Same store directory but a different user identity is a different Host + // identity: the deletion does not apply to it. + const otherUser = new ChatStore(tempDir) + otherUser.setAuthorityScope({ ...TEAM_A_SCOPE, userId: 'user-b' }) + await expect(otherUser.createChat('agent-1', 'identity-chat')).resolves.toMatchObject({ + id: 'identity-chat', + }) + }) + + it('keeps cleanup queued after a failure and retries it only in the owning scope', async () => { + await store.createChat('agent-1', 'retry-cleanup') + await store.saveMessages('agent-1', 'retry-cleanup', [ + { id: 'message', role: 'user', content: 'private', timestamp: 1 }, + ]) + const cleanupReadError = Object.assign(new Error('permission denied'), { code: 'EACCES' }) + vi.spyOn(fs, 'readdir').mockRejectedValueOnce(cleanupReadError) + + await expect(store.deleteChat('agent-1', 'retry-cleanup', TEAM_A_SCOPE)).resolves.toEqual({ + cleanupPending: true, + }) + expect((await store.getIndex('agent-1')).pendingChatCleanup).toContainEqual({ + chatId: 'retry-cleanup', + authorityScope: TEAM_A_SCOPE, + }) + await expect(fs.access(chatCacheDir('retry-cleanup'))).resolves.toBeUndefined() + + // A caller holding the previous user's scope is stale once the session user + // changed: the retry refuses instead of running for the wrong identity. + store.setAuthorityScope({ ...TEAM_A_SCOPE, userId: 'user-b' }) + await expect(store.retryPendingDeleteCleanups(TEAM_A_SCOPE)).rejects.toThrow( + 'Chat deletion authority scope changed' + ) + await expect(fs.access(chatCacheDir('retry-cleanup'))).resolves.toBeUndefined() + + store.setAuthorityScope(TEAM_A_SCOPE) + await store.retryPendingDeleteCleanups(TEAM_A_SCOPE) + expect((await store.getIndex('agent-1')).pendingChatCleanup).toEqual([]) + await expect(fs.access(chatCacheDir('retry-cleanup'))).rejects.toThrow() + }) + + it('retries a cleanup left pending in team A while the session is in team B (same environment and user)', async () => { + await store.createChat('agent-1', 'team-switch-cleanup') + await store.saveMessages('agent-1', 'team-switch-cleanup', [ + { id: 'message', role: 'user', content: 'private', timestamp: 1 }, + ]) + vi.spyOn(fs, 'readdir').mockRejectedValueOnce( + Object.assign(new Error('permission denied'), { code: 'EACCES' }) + ) + vi.spyOn(console, 'warn').mockImplementation(() => undefined) + await expect(store.deleteChat('agent-1', 'team-switch-cleanup', TEAM_A_SCOPE)).resolves.toEqual( + { cleanupPending: true } + ) + // Liveness witness: the entry is really queued and the artifacts really exist. + expect((await store.getIndex('agent-1')).pendingChatCleanup).toEqual([ + { chatId: 'team-switch-cleanup', authorityScope: TEAM_A_SCOPE }, + ]) + await expect(fs.access(chatCacheDir('team-switch-cleanup'))).resolves.toBeUndefined() + + store.setAuthorityScope(TEAM_B_SCOPE) + await store.retryPendingDeleteCleanups(TEAM_B_SCOPE) + + expect((await store.getIndex('agent-1')).pendingChatCleanup).toEqual([]) + await expect(fs.access(chatCacheDir('team-switch-cleanup'))).rejects.toThrow() + // The tombstone keeps holding in the new team: the chat is not resurrected. + expect((await store.getIndex('agent-1')).deletedChatTombstones).toContainEqual({ + chatId: 'team-switch-cleanup', + authorityScope: TEAM_A_SCOPE, + }) + }) + + it.each([ + ['a different user', { ...TEAM_A_SCOPE, userId: 'user-b' }], + ['a different environment', { ...TEAM_A_SCOPE, environmentKey: 'env-b' }], + ])('does not retry a cleanup pending for %s', async (_label, otherScope) => { + vi.spyOn(console, 'warn').mockImplementation(() => undefined) + const readdir = vi.spyOn(fs, 'readdir') + // agent-1: a cleanup left pending by the original identity. + await store.createChat('agent-1', 'foreign-cleanup') + await store.saveMessages('agent-1', 'foreign-cleanup', [ + { id: 'message', role: 'user', content: 'private', timestamp: 1 }, + ]) + readdir.mockRejectedValueOnce(Object.assign(new Error('denied'), { code: 'EACCES' })) + await store.deleteChat('agent-1', 'foreign-cleanup', TEAM_A_SCOPE) + // agent-2: a cleanup left pending by the other identity, which the retry must process. + store.setAuthorityScope(otherScope) + await store.createChat('agent-2', 'own-cleanup') + await store.saveMessages('agent-2', 'own-cleanup', [ + { id: 'message', role: 'user', content: 'mine', timestamp: 1 }, + ]) + readdir.mockRejectedValueOnce(Object.assign(new Error('denied'), { code: 'EACCES' })) + await store.deleteChat('agent-2', 'own-cleanup', otherScope) + expect((await store.getIndex('agent-2')).pendingChatCleanup).toHaveLength(1) + + await store.retryPendingDeleteCleanups(otherScope) + + // Liveness witness: the retry ran and finished the entry that belongs to this identity. + expect((await store.getIndex('agent-2')).pendingChatCleanup).toEqual([]) + await expect(fs.access(join(tempDir, 'agent-2', 'chats', 'own-cleanup'))).rejects.toThrow() + // The other identity's entry was left alone: still queued, artifacts still on disk. + expect((await store.getIndex('agent-1')).pendingChatCleanup).toEqual([ + { chatId: 'foreign-cleanup', authorityScope: TEAM_A_SCOPE }, + ]) + await expect(fs.access(chatCacheDir('foreign-cleanup'))).resolves.toBeUndefined() + }) +}) + +// ── deletion identity guards ───────────────────────────────────────────────── + +/** + * The store's authority scope can change while a deletion is in flight (sign-out, + * account or environment switch). Every `sameChatDeletionIdentity` guard inside + * the cleanup path exists for that window, so each test moves the scope from + * inside a filesystem call the cleanup is awaiting and then checks what the + * cleanup did NOT do afterwards. Every negative check is paired with a witness + * that the call which moved the scope really ran. + */ +describe('deleted-chat cleanup when the identity changes mid-operation', () => { + const FOREIGN_SCOPES = [ + ['a different user', { ...TEAM_A_SCOPE, userId: 'user-b' }], + ['a different environment', { ...TEAM_A_SCOPE, environmentKey: 'env-b' }], + ] as const + const CHAT_ID = 'mid-operation-chat' + + function snapshotRootPath(chatId: string): string { + return agentPath('chats', '.snapshots', Buffer.from(chatId, 'utf8').toString('base64url')) + } + + async function seedChat(chatId: string): Promise { + await store.createChat('agent-1', chatId) + await store.saveMessages('agent-1', chatId, [ + { id: 'm1', role: 'user', content: 'private', timestamp: 1 }, + ]) + const sibling = chatSnapshotDir(chatId, 'previous', 'sibling') + await fs.mkdir(sibling, { recursive: true }) + await fs.writeFile(join(sibling, 'marker'), 'sibling') + await expect(fs.access(chatCacheDir(chatId))).resolves.toBeUndefined() + await expect(fs.access(join(sibling, 'marker'))).resolves.toBeUndefined() + } + + /** Records every `fs.rm` path and runs `afterRm` once the real removal finished. */ + function interceptRm(afterRm: (path: string) => void): string[] { + const realRm = fs.rm.bind(fs) + const paths: string[] = [] + vi.spyOn(fs, 'rm').mockImplementation(async (...args) => { + paths.push(String(args[0])) + await realRm(...(args as Parameters)) + afterRm(String(args[0])) + }) + return paths + } + + async function expectTombstoneAndCleanupStillQueued(chatId: string): Promise { + const index = await store.getIndex('agent-1') + expect(index.deletedChatTombstones).toContainEqual({ + chatId, + authorityScope: TEAM_A_SCOPE, + }) + expect(index.pendingChatCleanup).toContainEqual({ chatId, authorityScope: TEAM_A_SCOPE }) + } + + /** The interrupted cleanup stays resumable: the owning identity finishes it. */ + async function expectOwningIdentityFinishesCleanup(chatId: string): Promise { + vi.restoreAllMocks() + store.setAuthorityScope(TEAM_A_SCOPE) + await store.retryPendingDeleteCleanups(TEAM_A_SCOPE) + expect((await store.getIndex('agent-1')).pendingChatCleanup).toEqual([]) + await expect(fs.access(chatCacheDir(chatId))).rejects.toThrow() + await expect(fs.access(snapshotRootPath(chatId))).rejects.toThrow() + } + + function failingRm(): void { + vi.spyOn(fs, 'rm').mockRejectedValue(Object.assign(new Error('denied'), { code: 'EACCES' })) + vi.spyOn(console, 'warn').mockImplementation(() => undefined) + } + + type CleanupInternals = { + finishPendingDeleteCleanup( + agentRef: string, + tombstone: { chatId: string; authorityScope: unknown } + ): Promise + } + + it.each(FOREIGN_SCOPES)( + 'stops before the next removal when the identity changes to %s during one', + async (_label, foreign) => { + await seedChat(CHAT_ID) + const removed = interceptRm(() => store.setAuthorityScope(foreign)) + + await expect(store.deleteChat('agent-1', CHAT_ID, TEAM_A_SCOPE)).resolves.toEqual({ + cleanupPending: true, + }) + + // Witness: the first removal ran (and moved the scope); none followed it. + expect(removed).toHaveLength(1) + await expect(fs.access(chatCacheDir(CHAT_ID))).resolves.toBeUndefined() + await expectTombstoneAndCleanupStillQueued(CHAT_ID) + await expectOwningIdentityFinishesCleanup(CHAT_ID) + } + ) + + it.each(FOREIGN_SCOPES)( + 'removes nothing when the identity changes to %s while the legacy quarantine is listed', + async (_label, foreign) => { + await seedChat(CHAT_ID) + const realReaddir = fs.readdir.bind(fs) + const listed: string[] = [] + vi.spyOn(fs, 'readdir').mockImplementation(async (...args) => { + listed.push(String(args[0])) + try { + // The quarantine directory does not exist here: the real listing fails + // with ENOENT, which the cleanup treats as "nothing quarantined". + return await realReaddir(...(args as Parameters)) + } finally { + if (String(args[0]) === agentPath('.corrupt')) store.setAuthorityScope(foreign) + } + }) + const removed = interceptRm(() => undefined) + + await expect(store.deleteChat('agent-1', CHAT_ID, TEAM_A_SCOPE)).resolves.toEqual({ + cleanupPending: true, + }) + + // Witness: the quarantine listing ran (and moved the scope) before any removal. + expect(listed).toContain(agentPath('.corrupt')) + expect(removed).toEqual([]) + await expect(fs.access(chatCacheDir(CHAT_ID))).resolves.toBeUndefined() + await expectTombstoneAndCleanupStillQueued(CHAT_ID) + await expectOwningIdentityFinishesCleanup(CHAT_ID) + } + ) + + it.each(FOREIGN_SCOPES)( + 'leaves the snapshot siblings alone when the identity changes to %s during the last artifact removal', + async (_label, foreign) => { + await seedChat(CHAT_ID) + const removed = interceptRm(path => { + if (path === chatCacheDir(CHAT_ID)) store.setAuthorityScope(foreign) + }) + + await expect(store.deleteChat('agent-1', CHAT_ID, TEAM_A_SCOPE)).resolves.toEqual({ + cleanupPending: true, + }) + + // Witness: the chat directory really was removed (moving the scope), and the + // snapshot siblings, which the cleanup removes right after it, were not. + expect(removed).toContain(chatCacheDir(CHAT_ID)) + await expect(fs.access(chatCacheDir(CHAT_ID))).rejects.toThrow() + expect(removed).not.toContain(snapshotRootPath(CHAT_ID)) + await expect( + fs.access(join(chatSnapshotDir(CHAT_ID, 'previous', 'sibling'), 'marker')) + ).resolves.toBeUndefined() + await expectTombstoneAndCleanupStillQueued(CHAT_ID) + await expectOwningIdentityFinishesCleanup(CHAT_ID) + } + ) + + it.each(FOREIGN_SCOPES)( + 'keeps the cleanup queued when the identity changes to %s during the snapshot sibling removal', + async (_label, foreign) => { + await seedChat(CHAT_ID) + const removed = interceptRm(path => { + if (path === snapshotRootPath(CHAT_ID)) store.setAuthorityScope(foreign) + }) + + await expect(store.deleteChat('agent-1', CHAT_ID, TEAM_A_SCOPE)).resolves.toEqual({ + cleanupPending: true, + }) + + // Witness: every artifact removal ran, the last one moved the scope, and the + // pending entry was still not cleared for an identity that no longer owns it. + expect(removed.at(-1)).toBe(snapshotRootPath(CHAT_ID)) + await expect(fs.access(chatCacheDir(CHAT_ID))).rejects.toThrow() + await expect(fs.access(snapshotRootPath(CHAT_ID))).rejects.toThrow() + await expectTombstoneAndCleanupStillQueued(CHAT_ID) + await expectOwningIdentityFinishesCleanup(CHAT_ID) + } + ) + + it.each(FOREIGN_SCOPES)( + 'does not start the cleanup when the identity changes to %s while the deletion is persisted', + async (_label, foreign) => { + await seedChat(CHAT_ID) + const realRename = fs.rename.bind(fs) + const indexRenames: string[] = [] + vi.spyOn(fs, 'rename').mockImplementation(async (...args) => { + await realRename(...(args as Parameters)) + if (String(args[1]) === agentPath('index.json')) { + indexRenames.push(String(args[1])) + store.setAuthorityScope(foreign) + } + }) + const realReaddir = fs.readdir.bind(fs) + const listed: string[] = [] + vi.spyOn(fs, 'readdir').mockImplementation(async (...args) => { + listed.push(String(args[0])) + return realReaddir(...(args as Parameters)) + }) + const removed = interceptRm(() => undefined) + + await expect(store.deleteChat('agent-1', CHAT_ID, TEAM_A_SCOPE)).resolves.toEqual({ + cleanupPending: true, + }) + + // Witness: the deletion was persisted (moving the scope) before the cleanup + // would have listed the quarantine or removed anything. + expect(indexRenames.length).toBeGreaterThan(0) + expect(listed).not.toContain(agentPath('.corrupt')) + expect(removed).toEqual([]) + await expectTombstoneAndCleanupStillQueued(CHAT_ID) + await expectOwningIdentityFinishesCleanup(CHAT_ID) + } + ) + + it.each(FOREIGN_SCOPES)( + "keeps another identity's pending entry for the same chat when the owner finishes (%s)", + async (_label, foreign) => { + failingRm() + await store.deleteChat('agent-1', 'shared-chat-id', TEAM_A_SCOPE) + store.setAuthorityScope(foreign) + await store.deleteChat('agent-1', 'shared-chat-id', foreign) + expect((await store.getIndex('agent-1')).pendingChatCleanup).toEqual([ + { chatId: 'shared-chat-id', authorityScope: TEAM_A_SCOPE }, + { chatId: 'shared-chat-id', authorityScope: foreign }, + ]) + vi.restoreAllMocks() + + store.setAuthorityScope(TEAM_A_SCOPE) + await store.retryPendingDeleteCleanups(TEAM_A_SCOPE) + + // Witness: the owner's own entry was processed and cleared; the other + // identity's entry for the same chat id is still queued, its tombstone too. + const index = await store.getIndex('agent-1') + expect(index.pendingChatCleanup).toEqual([ + { chatId: 'shared-chat-id', authorityScope: foreign }, + ]) + expect(index.deletedChatTombstones).toHaveLength(2) + } + ) + + it('hands only entries of the calling identity to the cleanup', async () => { + const foreign = { ...TEAM_A_SCOPE, userId: 'user-b' } + failingRm() + await store.deleteChat('agent-1', 'foreign-entry', TEAM_A_SCOPE) + store.setAuthorityScope(foreign) + await store.deleteChat('agent-1', 'own-entry', foreign) + vi.restoreAllMocks() + expect((await store.getIndex('agent-1')).pendingChatCleanup).toHaveLength(2) + const finish = vi.spyOn(store as unknown as CleanupInternals, 'finishPendingDeleteCleanup') + + await store.retryPendingDeleteCleanups(foreign) + + // Witness: the retry did hand work to the cleanup (the calling identity's entry). + expect(finish.mock.calls.map(call => call[1].chatId)).toEqual(['own-entry']) + expect((await store.getIndex('agent-1')).pendingChatCleanup).toEqual([ + { chatId: 'foreign-entry', authorityScope: TEAM_A_SCOPE }, + ]) + }) + + it('stops the retry sweep when the identity changes to another user during it', async () => { + failingRm() + await store.deleteChat('agent-1', 'sweep-1', TEAM_A_SCOPE) + await store.deleteChat('agent-1', 'sweep-2', TEAM_A_SCOPE) + vi.restoreAllMocks() + expect((await store.getIndex('agent-1')).pendingChatCleanup).toHaveLength(2) + const finish = vi.spyOn(store as unknown as CleanupInternals, 'finishPendingDeleteCleanup') + const removed = interceptRm(() => + store.setAuthorityScope({ ...TEAM_A_SCOPE, userId: 'user-b' }) + ) + + await store.retryPendingDeleteCleanups(TEAM_A_SCOPE) + + // Witness: the first entry was handed to the cleanup and its removal moved the + // scope; the second entry was never handed over, and both stay queued. + expect(removed).toHaveLength(1) + expect(finish).toHaveBeenCalledTimes(1) + expect((await store.getIndex('agent-1')).pendingChatCleanup).toHaveLength(2) + }) + + it('records one tombstone and one pending entry when the same chat is deleted twice', async () => { + await seedChat(CHAT_ID) + failingRm() + + await expect(store.deleteChat('agent-1', CHAT_ID, TEAM_A_SCOPE)).resolves.toEqual({ + cleanupPending: true, + }) + // Witness: the first deletion queued its entries before the repeat. + const afterFirst = await store.getIndex('agent-1') + expect(afterFirst.deletedChatTombstones).toHaveLength(1) + expect(afterFirst.pendingChatCleanup).toHaveLength(1) + + await expect(store.deleteChat('agent-1', CHAT_ID, TEAM_A_SCOPE)).resolves.toEqual({ + cleanupPending: true, + }) + + const afterSecond = await store.getIndex('agent-1') + expect(afterSecond.deletedChatTombstones).toEqual([ + { chatId: CHAT_ID, authorityScope: TEAM_A_SCOPE }, + ]) + expect(afterSecond.pendingChatCleanup).toEqual([ + { chatId: CHAT_ID, authorityScope: TEAM_A_SCOPE }, + ]) + }) +}) + +// ── tombstone durability ───────────────────────────────────────────────────── + +describe('tombstone durability', () => { + const tombstone = (chatId: string) => ({ chatId, authorityScope: TEAM_A_SCOPE }) + + it('drops only a malformed tombstone entry and keeps the catalog and other tombstones', async () => { + await store.createChat('agent-1', 'kept-chat') + await store.createChat('agent-1', 'gone-1') + await store.deleteChat('agent-1', 'gone-1', TEAM_A_SCOPE) + const index = await readJsonFile>(agentPath('index.json')) + index.deletedChatTombstones = [ + { chatId: 'malformed-entry', authorityScope: { environmentKey: '', userId: 7 } }, + ...(index.deletedChatTombstones as unknown[]), + ] + await fs.writeFile(agentPath('index.json'), JSON.stringify(index)) + // The sidecar must not paper over the malformed entry in this scenario. + await fs.rm(agentPath('.tombstones.json'), { force: true }) + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + + const restarted = new ChatStore(tempDir) + restarted.setAuthorityScope(TEAM_A_SCOPE) + const first = await restarted.getIndex('agent-1') + await restarted.getIndex('agent-1') + + expect(first.chats.map(chat => chat.id)).toEqual(['kept-chat']) + expect(first.deletedChatTombstones).toEqual([tombstone('gone-1')]) + // Not quarantined: the catalog stays in place. + await expect(fs.access(agentPath('index.json'))).resolves.toBeUndefined() + await expect(fs.access(agentPath('.corrupt'))).rejects.toThrow() + // Reported once, not once per read. + const dropped = warn.mock.calls.filter(call => String(call[0]).includes('malformed tombstone')) + expect(dropped).toHaveLength(1) + await expect(restarted.createChat('agent-1', 'gone-1')).rejects.toThrow( + 'Chat was deleted locally' + ) + }) + + it('restores tombstones from the sidecar when a corrupt catalog is quarantined', async () => { + await store.createChat('agent-1', 'kept-chat') + await store.createChat('agent-1', 'deleted-chat') + await store.saveMessages('agent-1', 'deleted-chat', [ + { id: 'm1', role: 'user', content: 'private', timestamp: 1 }, + ]) + // Cleanup fails, so the chat stays in pendingChatCleanup as well. + vi.spyOn(fs, 'rm').mockRejectedValue(Object.assign(new Error('denied'), { code: 'EACCES' })) + vi.spyOn(console, 'warn').mockImplementation(() => undefined) + await expect(store.deleteChat('agent-1', 'deleted-chat', TEAM_A_SCOPE)).resolves.toEqual({ + cleanupPending: true, + }) + vi.restoreAllMocks() + vi.spyOn(console, 'warn').mockImplementation(() => undefined) + // Liveness witness: the catalog held the tombstone before it was corrupted. + expect((await store.getIndex('agent-1')).deletedChatTombstones).toEqual([ + tombstone('deleted-chat'), + ]) + + await fs.writeFile(agentPath('index.json'), '{"version":2,"chats":[') + + const restarted = new ChatStore(tempDir) + restarted.setAuthorityScope(TEAM_A_SCOPE) + const recovered = await restarted.getIndex('agent-1') + expect(recovered.chats).toEqual([]) + expect(recovered.deletedChatTombstones).toEqual([tombstone('deleted-chat')]) + expect(recovered.pendingChatCleanup).toEqual([tombstone('deleted-chat')]) + // The corrupt bytes stay recoverable in quarantine. + const quarantined = await fs.readdir(agentPath('.corrupt')) + expect(quarantined).toHaveLength(1) + expect(await fs.readFile(agentPath('.corrupt', quarantined[0]!), 'utf-8')).toBe( + '{"version":2,"chats":[' + ) + // The catalog is gone from its writable path, yet the deletion still holds. + await expect(fs.access(agentPath('index.json'))).rejects.toThrow() + expect((await restarted.getIndex('agent-1')).deletedChatTombstones).toEqual([ + tombstone('deleted-chat'), + ]) + await expect(restarted.createChat('agent-1', 'deleted-chat')).rejects.toThrow( + 'Chat was deleted locally' + ) + await expect(restarted.createChat('agent-1', 'kept-chat')).resolves.toMatchObject({ + id: 'kept-chat', + }) + }) + + it('does not write a tombstone sidecar for an agent that never deleted a chat', async () => { + await store.createChat('agent-1', 'never-deleted') + expect((await store.listChats('agent-1')).map(chat => chat.id)).toEqual(['never-deleted']) + await expect(fs.access(agentPath('.tombstones.json'))).rejects.toThrow() + }) + + it('reports an unreadable sidecar once and returns the empty catalog', async () => { + await store.createChat('agent-1', 'c') + await store.deleteChat('agent-1', 'c', TEAM_A_SCOPE) + await fs.writeFile(agentPath('index.json'), 'not json') + await fs.writeFile(agentPath('.tombstones.json'), 'not json either') + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + + const restarted = new ChatStore(tempDir) + restarted.setAuthorityScope(TEAM_A_SCOPE) + expect((await restarted.getIndex('agent-1')).deletedChatTombstones).toBeUndefined() + await restarted.getIndex('agent-1') + + const unreadable = warn.mock.calls.filter(call => String(call[0]).includes('tombstone sidecar')) + expect(unreadable).toHaveLength(1) + }) + + it('commits nothing when the sidecar write of a delete fails, and a retry commits both', async () => { + await store.createChat('agent-1', 'a') + await store.createChat('agent-1', 'b') + const originalWriteFile = fs.writeFile.bind(fs) + const written: string[] = [] + const writeFile = vi.spyOn(fs, 'writeFile').mockImplementation(async (...args) => { + const filePath = String(args[0]) + written.push(filePath) + if (filePath.endsWith('.tombstones.json.tmp')) { + throw Object.assign(new Error('no space left on device'), { code: 'ENOSPC' }) + } + return originalWriteFile(...(args as Parameters)) + }) + + await expect(store.deleteChat('agent-1', 'a', TEAM_A_SCOPE)).rejects.toMatchObject({ + code: 'ENOSPC', + }) + + // Witness: the rejected save reached the sidecar write. + expect(written.some(path => path.endsWith('.tombstones.json.tmp'))).toBe(true) + // The rejected delete left the catalog as it was. + expect((await store.listChats('agent-1')).map(chat => chat.id)).toEqual(['a', 'b']) + expect(written.some(path => path.endsWith('index.json.tmp'))).toBe(false) + + writeFile.mockRestore() + await expect(store.deleteChat('agent-1', 'a', TEAM_A_SCOPE)).resolves.toEqual({ + cleanupPending: false, + }) + const sidecar = await readJsonFile<{ deletedChatTombstones: unknown[] }>( + agentPath('.tombstones.json') + ) + expect(sidecar.deletedChatTombstones).toEqual([tombstone('a')]) + expect((await store.listChats('agent-1')).map(chat => chat.id)).toEqual(['b']) + }) + + /** Records every readFile path and fails the sidecar read with `code` when given. */ + function spyReadFile(failSidecarWith?: string): string[] { + const originalReadFile = fs.readFile.bind(fs) + const read: string[] = [] + vi.spyOn(fs, 'readFile').mockImplementation(async (...args) => { + const filePath = String(args[0]) + read.push(filePath) + if (failSidecarWith && filePath === agentPath('.tombstones.json')) { + throw transientFileReadError(failSidecarWith) + } + return originalReadFile(...(args as Parameters)) + }) + return read + } + + it('propagates a sidecar read failure other than ENOENT instead of dropping the tombstones', async () => { + await store.createChat('agent-1', 'gone') + await store.deleteChat('agent-1', 'gone', TEAM_A_SCOPE) + // A missing catalog makes the read recover tombstones from the sidecar. + await fs.rm(agentPath('index.json')) + const read = spyReadFile('EACCES') + + const restarted = new ChatStore(tempDir) + restarted.setAuthorityScope(TEAM_A_SCOPE) + await expect(restarted.getIndex('agent-1')).rejects.toMatchObject({ code: 'EACCES' }) + + // Witness: the rejection came from the sidecar read. + expect(read).toContain(agentPath('.tombstones.json')) + }) + + it('reports a sidecar with an unknown shape once and recovers as if it were absent', async () => { + await store.createChat('agent-1', 'gone') + await store.deleteChat('agent-1', 'gone', TEAM_A_SCOPE) + await fs.rm(agentPath('index.json')) + // Well-formed tombstones under an unknown version: the version alone must reject it. + await fs.writeFile( + agentPath('.tombstones.json'), + JSON.stringify({ + version: 99, + deletedChatTombstones: [tombstone('gone')], + pendingChatCleanup: [], + }) + ) + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + + const restarted = new ChatStore(tempDir) + restarted.setAuthorityScope(TEAM_A_SCOPE) + const first = await restarted.getIndex('agent-1') + await restarted.getIndex('agent-1') + + expect(first.chats).toEqual([]) + expect(first.deletedChatTombstones).toBeUndefined() + // Witness: the rejection was reported, once, naming the agent. + const reports = warn.mock.calls.filter( + call => String(call[0]).includes('tombstone sidecar') && String(call[0]).includes('"agent-1"') + ) + expect(reports).toHaveLength(1) + }) + + it('does not merge the sidecar into a readable index, only into a recovered one', async () => { + await store.createChat('agent-1', 'kept') + await store.createChat('agent-1', 'gone') + await store.deleteChat('agent-1', 'gone', TEAM_A_SCOPE) + const sidecar = await readJsonFile<{ deletedChatTombstones: unknown[] }>( + agentPath('.tombstones.json') + ) + sidecar.deletedChatTombstones.push(tombstone('sidecar-only')) + await fs.writeFile(agentPath('.tombstones.json'), JSON.stringify(sidecar)) + const read = spyReadFile() + vi.spyOn(console, 'warn').mockImplementation(() => undefined) + const restarted = new ChatStore(tempDir) + restarted.setAuthorityScope(TEAM_A_SCOPE) + + const valid = await restarted.getIndex('agent-1') + + expect(valid.chats.map(chat => chat.id)).toEqual(['kept']) + expect(valid.deletedChatTombstones).toEqual([tombstone('gone')]) + expect(read).toContain(agentPath('index.json')) + expect(read).not.toContain(agentPath('.tombstones.json')) + + // Control: once the catalog is corrupt, the same read does merge the sidecar. + await fs.writeFile(agentPath('index.json'), '{"version":2,"chats":[') + const recovered = await restarted.getIndex('agent-1') + + expect(read).toContain(agentPath('.tombstones.json')) + expect(recovered.deletedChatTombstones).toEqual([tombstone('gone'), tombstone('sidecar-only')]) + }) +}) + +describe('deleted-chat cleanup failure reporting', () => { + it('reports a permanent cleanup failure once per agent, chat and code', async () => { + await store.createChat('agent-1', 'stuck-chat') + await store.saveMessages('agent-1', 'stuck-chat', [ + { id: 'm1', role: 'user', content: 'x', timestamp: 1 }, + ]) + await store.createChat('agent-2', 'clean-chat') + await store.saveMessages('agent-2', 'clean-chat', [ + { id: 'm1', role: 'user', content: 'x', timestamp: 1 }, + ]) + const realRm = fs.rm.bind(fs) + // Only agent-1's directory is unwritable; agent-2 cleans normally. + vi.spyOn(fs, 'rm').mockImplementation(async (path, options) => { + if (String(path).includes(join('agent-1'))) { + throw Object.assign(new Error('permission denied'), { code: 'EACCES' }) + } + return realRm(path, options) + }) + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + + await expect(store.deleteChat('agent-1', 'stuck-chat', TEAM_A_SCOPE)).resolves.toEqual({ + cleanupPending: true, + }) + await expect(store.deleteChat('agent-2', 'clean-chat', TEAM_A_SCOPE)).resolves.toEqual({ + cleanupPending: false, + }) + + await store.retryPendingDeleteCleanups(TEAM_A_SCOPE) + await store.retryPendingDeleteCleanups(TEAM_A_SCOPE) + + const stuckReports = warn.mock.calls.filter( + call => + String(call[0]).includes('Deleted-chat cleanup failed') && + String(call[0]).includes('"agent-1"') && + String(call[0]).includes('"stuck-chat"') && + String(call[0]).includes('EACCES') + ) + expect(stuckReports).toHaveLength(1) + // The failing agent stays queued; the healthy one finished on its own. + expect((await store.getIndex('agent-1')).pendingChatCleanup).toEqual([ + { chatId: 'stuck-chat', authorityScope: TEAM_A_SCOPE }, + ]) + expect((await store.getIndex('agent-2')).pendingChatCleanup).toEqual([]) + }) + + it('reports an unreadable agent catalog during retry and still cleans the other agents', async () => { + await store.createChat('agent-2', 'clean-chat') + await store.saveMessages('agent-2', 'clean-chat', [ + { id: 'm1', role: 'user', content: 'x', timestamp: 1 }, + ]) + const realRm = fs.rm.bind(fs) + vi.spyOn(console, 'warn').mockImplementation(() => undefined) + const rm = vi + .spyOn(fs, 'rm') + .mockRejectedValue(Object.assign(new Error('permission denied'), { code: 'EACCES' })) + await store.deleteChat('agent-2', 'clean-chat', TEAM_A_SCOPE) + rm.mockImplementation((path, options) => realRm(path, options)) + // Sorts before agent-2, so the retry hits it first. + await fs.mkdir(join(tempDir, 'agent-0-broken'), { recursive: true }) + await fs.writeFile( + join(tempDir, 'agent-0-broken', 'index.json'), + JSON.stringify({ version: 99, chats: [] }) + ) + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + + await store.retryPendingDeleteCleanups(TEAM_A_SCOPE) + await store.retryPendingDeleteCleanups(TEAM_A_SCOPE) + + const brokenReports = warn.mock.calls.filter( + call => + String(call[0]).includes('Deleted-chat cleanup failed') && + String(call[0]).includes('"agent-0-broken"') + ) + expect(brokenReports).toHaveLength(1) + expect((await store.getIndex('agent-2')).pendingChatCleanup).toEqual([]) + }) +}) + +/** + * `chat:getIndex` retries the pending cleanups before every read and the + * renderer reads every agent's index in a fan-out, so a retry that walks every + * agent costs N·(N+1) index reads per fan-out. The first retry after a bind + * walks every agent once; later retries touch only the agent being read. + */ +describe('pending cleanup retry per agent', () => { + const AGENTS = ['agent-1', 'agent-2', 'agent-3', 'agent-4', 'agent-5'] + const indexPathOf = (agentRef: string) => join(tempDir, agentRef, 'index.json') + + async function queueFailedCleanup(agentRef: string, chatId: string): Promise { + await store.createChat(agentRef, chatId) + await store.saveMessages(agentRef, chatId, [ + { id: 'm1', role: 'user', content: 'x', timestamp: 1 }, + ]) + const rm = vi + .spyOn(fs, 'rm') + .mockRejectedValue(Object.assign(new Error('denied'), { code: 'EACCES' })) + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + await expect(store.deleteChat(agentRef, chatId, TEAM_A_SCOPE)).resolves.toEqual({ + cleanupPending: true, + }) + rm.mockRestore() + warn.mockRestore() + } + + /** Records every read of an agent's index.json until the mocks are restored. */ + function recordIndexReads(): string[] { + const originalReadFile = fs.readFile.bind(fs) + const reads: string[] = [] + vi.spyOn(fs, 'readFile').mockImplementation(async (...args) => { + const filePath = String(args[0]) + if (filePath.endsWith('index.json')) reads.push(filePath) + return originalReadFile(...(args as Parameters)) + }) + return reads + } + + async function pendingOf(agentRef: string) { + return (await store.getIndex(agentRef)).pendingChatCleanup ?? [] + } + + it('walks every agent once after a bind, then reads only the requested agent', async () => { + for (const agentRef of AGENTS) await queueFailedCleanup(agentRef, `pending-${agentRef}`) + // Witness: every agent really has a queued cleanup. + for (const agentRef of AGENTS) expect(await pendingOf(agentRef)).toHaveLength(1) + store.setAuthorityScope(TEAM_A_SCOPE) + const reads = recordIndexReads() + + await store.retryPendingDeleteCleanups(TEAM_A_SCOPE, 'agent-1') + // The sweep reads each agent's index to find its queue, and once more to + // drop the finished entry. + expect(reads).toHaveLength(2 * AGENTS.length) + const afterSweep = reads.length + for (const agentRef of AGENTS.slice(1)) { + await store.retryPendingDeleteCleanups(TEAM_A_SCOPE, agentRef) + } + + expect(reads.slice(afterSweep)).toEqual(AGENTS.slice(1).map(indexPathOf)) + expect(reads).toHaveLength(3 * AGENTS.length - 1) + vi.restoreAllMocks() + for (const agentRef of AGENTS) expect(await pendingOf(agentRef)).toEqual([]) + }) + + it('finishes a cleanup queued after the sweep on the next read of that agent', async () => { + await store.createChat('agent-1', 'kept') + store.setAuthorityScope(TEAM_A_SCOPE) + await store.retryPendingDeleteCleanups(TEAM_A_SCOPE, 'agent-1') + await queueFailedCleanup('agent-3', 'late-delete') + // Witness: the cleanup is really queued and its artifacts are on disk. + expect(await pendingOf('agent-3')).toHaveLength(1) + await expect(fs.access(join(tempDir, 'agent-3', 'chats', 'late-delete'))).resolves.toBe( + undefined + ) + const reads = recordIndexReads() + + await store.retryPendingDeleteCleanups(TEAM_A_SCOPE, 'agent-3') + + const retryReads = [...reads] + vi.restoreAllMocks() + expect(await pendingOf('agent-3')).toEqual([]) + await expect(fs.access(join(tempDir, 'agent-3', 'chats', 'late-delete'))).rejects.toThrow() + // Only agent-3 was touched: read to find its queue, then to drop the entry. + expect(retryReads).toEqual([indexPathOf('agent-3'), indexPathOf('agent-3')]) + }) + + it('walks every agent again after the next bind', async () => { + for (const agentRef of AGENTS.slice(0, 3)) await store.createChat(agentRef, 'kept') + store.setAuthorityScope(TEAM_A_SCOPE) + const reads = recordIndexReads() + + await store.retryPendingDeleteCleanups(TEAM_A_SCOPE, 'agent-1') + await store.retryPendingDeleteCleanups(TEAM_A_SCOPE, 'agent-1') + // A team switch of the same user rebinds the store. + store.setAuthorityScope(TEAM_B_SCOPE) + await store.retryPendingDeleteCleanups(TEAM_B_SCOPE, 'agent-1') + + const sweep = AGENTS.slice(0, 3).map(indexPathOf) + expect(reads).toEqual([...sweep, indexPathOf('agent-1'), ...sweep]) + }) + + it('keeps the sweep armed when the agents directory cannot be listed', async () => { + for (const agentRef of AGENTS.slice(0, 2)) await store.createChat(agentRef, 'kept') + store.setAuthorityScope(TEAM_A_SCOPE) + const readdir = vi + .spyOn(fs, 'readdir') + .mockRejectedValueOnce(Object.assign(new Error('permission denied'), { code: 'EACCES' })) + + await expect(store.retryPendingDeleteCleanups(TEAM_A_SCOPE, 'agent-1')).rejects.toMatchObject({ + code: 'EACCES', + }) + const reads = recordIndexReads() + await store.retryPendingDeleteCleanups(TEAM_A_SCOPE, 'agent-1') + + expect(readdir.mock.calls.map(call => String(call[0]))).toEqual([tempDir, tempDir]) + expect(reads).toEqual(AGENTS.slice(0, 2).map(indexPathOf)) + }) }) // ── messages ───────────────────────────────────────────────────────────────── @@ -215,6 +1138,37 @@ describe('messages', () => { expect(messages[1]!.content).toBe('hello') }) + it('does not load transcript messages while deleted-chat cleanup is pending', async () => { + await store.createChat('agent-1', 'pending-delete') + await store.saveMessages('agent-1', 'pending-delete', [ + { id: 'private-message', role: 'user', content: 'still on disk', timestamp: 1000 }, + ]) + vi.spyOn(fs, 'rm').mockRejectedValue(transientFileReadError('EACCES')) + + await expect(store.deleteChat('agent-1', 'pending-delete', TEAM_A_SCOPE)).resolves.toEqual({ + cleanupPending: true, + }) + expect((await store.getIndex('agent-1')).pendingChatCleanup).toContainEqual({ + chatId: 'pending-delete', + authorityScope: TEAM_A_SCOPE, + }) + await expect(store.loadMessages('agent-1', 'pending-delete')).resolves.toEqual([]) + await expect(fs.access(chatMetaPath('pending-delete'))).resolves.toBeUndefined() + }) + + it('upserts an optimistic outgoing message and later updates it by the same id', async () => { + await store.createChat('agent-1', 'upsert-send') + const optimistic = { id: 'outgoing-1', role: 'user' as const, content: 'hello', timestamp: 1 } + await store.upsertMessages('agent-1', 'upsert-send', [optimistic]) + await store.upsertMessages('agent-1', 'upsert-send', [ + { ...optimistic, task_id: 'task-1', timestamp: 2 }, + ]) + + await expect(store.loadMessages('agent-1', 'upsert-send')).resolves.toEqual([ + expect.objectContaining({ id: 'outgoing-1', task_id: 'task-1' }), + ]) + }) + it('pagination with limit and offset', async () => { await store.createChat('agent-1', 'pag-1') const msgs = Array.from({ length: 10 }, (_, i) => ({ @@ -243,6 +1197,7 @@ describe('messages', () => { pageSize: 3, maxLocalSyncedMessages: Number.POSITIVE_INFINITY, }) + pagedStore.setAuthorityScope(TEAM_A_SCOPE) await pagedStore.createChat('agent-1', 'paged-1') const msgs = Array.from({ length: 10 }, (_, i) => ({ id: `m${i}`, @@ -273,6 +1228,7 @@ describe('messages', () => { pageSize: 2, maxLocalSyncedMessages: Number.POSITIVE_INFINITY, }) + pagedStore.setAuthorityScope(TEAM_A_SCOPE) await pagedStore.createChat('agent-1', 'append-pages') await pagedStore.saveMessages('agent-1', 'append-pages', [ { id: 'm0', role: 'user', content: 'msg-0', timestamp: 0 }, @@ -663,6 +1619,7 @@ describe('messages', () => { pageSize: 2, maxLocalSyncedMessages: Number.POSITIVE_INFINITY, }) + pagedStore.setAuthorityScope(TEAM_A_SCOPE) await pagedStore.createChat('agent-1', 'delete-siblings') await pagedStore.saveMessages('agent-1', 'delete-siblings', [ { id: 'm0', role: 'user', content: 'msg-0', timestamp: 0 }, @@ -673,7 +1630,7 @@ describe('messages', () => { [{ id: 'backup-0', role: 'user', content: 'backup-0', timestamp: 1 }] ) - await pagedStore.deleteChat('agent-1', 'delete-siblings') + await pagedStore.deleteChat('agent-1', 'delete-siblings', TEAM_A_SCOPE) expect(await pagedStore.loadMessages('agent-1', 'delete-siblings')).toEqual([]) await expect( @@ -1570,6 +2527,18 @@ describe('lastActiveChatId', () => { const id = await store.getLastActiveChatId('agent-1') expect(id).toBe('last-1') }) + + it('does not reactivate a deleted chat when a late selection write arrives', async () => { + await store.createChat('agent-1', 'retained-1') + await store.setLastActiveChatId('agent-1', 'retained-1') + await store.createChat('agent-1', 'deleted-1') + await store.deleteChat('agent-1', 'deleted-1', TEAM_A_SCOPE) + + await store.setLastActiveChatId('agent-1', 'deleted-1') + + expect(await store.getLastActiveChatId('agent-1')).toBe('retained-1') + expect((await store.listChats('agent-1')).map(chat => chat.id)).toEqual(['retained-1']) + }) }) // ── onboarding ─────────────────────────────────────────────────────────────── @@ -1847,7 +2816,7 @@ describe('corrupt/missing files', () => { await fs.writeFile(join(corruptDir, `${encodedChatId}-${legacyUuid}.json`), '{}') await fs.writeFile(join(corruptDir, `${otherEncodedChatId}-${legacyUuid}.json`), '{}') - await store.deleteChat('agent-1', chatId) + await store.deleteChat('agent-1', chatId, TEAM_A_SCOPE) await expect(fs.access(join(corruptDir, encodedChatId))).rejects.toMatchObject({ code: 'ENOENT', diff --git a/desktop-app/src/__tests__/chatStoreBinding.test.ts b/desktop-app/src/__tests__/chatStoreBinding.test.ts index 8a9b9f715e..d4bb93200c 100644 --- a/desktop-app/src/__tests__/chatStoreBinding.test.ts +++ b/desktop-app/src/__tests__/chatStoreBinding.test.ts @@ -5,7 +5,9 @@ import path from 'node:path' import { __setChatStoreBaseDirForTests, bindChatStoreForUser, + getChatStoreBindingGeneration, requireChatStore, + requireChatStoreForBindingGeneration, unbindChatStore, } from '../chatStoreBinding' @@ -39,6 +41,27 @@ describe('chatStoreBinding', () => { expect(() => requireChatStore()).toThrow(/Not authenticated/) }) + it('fences destructive work to the binding that issued it', async () => { + await bindChatStoreForUser('user-a', ENV_A) + const firstBinding = getChatStoreBindingGeneration() + expect(requireChatStoreForBindingGeneration(firstBinding)).toBe(requireChatStore()) + + // Refreshing the same user's binding does not invalidate their work. + await bindChatStoreForUser('user-a', ENV_A) + expect(getChatStoreBindingGeneration()).toBe(firstBinding) + + await bindChatStoreForUser('user-b', ENV_A) + const nextBinding = getChatStoreBindingGeneration() + expect(nextBinding).not.toBe(firstBinding) + expect(() => requireChatStoreForBindingGeneration(firstBinding)).toThrow( + /Chat store binding changed/ + ) + expect(requireChatStoreForBindingGeneration(nextBinding)).toBe(requireChatStore()) + + unbindChatStore() + expect(() => requireChatStoreForBindingGeneration(nextBinding)).toThrow() + }) + it('does not follow symlinked quarantine roots during retention cleanup', async () => { const externalDir = await fs.mkdtemp(path.join(os.tmpdir(), 'clerum-external-quarantine-')) const sentinel = path.join(externalDir, 'corrupt-old') diff --git a/desktop-app/src/__tests__/httpClient.upstreamErrors.test.ts b/desktop-app/src/__tests__/httpClient.upstreamErrors.test.ts new file mode 100644 index 0000000000..d2d0dbaf4f --- /dev/null +++ b/desktop-app/src/__tests__/httpClient.upstreamErrors.test.ts @@ -0,0 +1,299 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { AuthClient } from '../authClient.js' +import { ApiError, requestJson } from '../httpClient.js' +import { RpcProxyClient, SandboxUiSessionError } from '../rpcProxyClient.js' +import { SharedFilesClient } from '../sharedFilesClient.js' +import { + ERROR_EXCERPT_MAX_CHARS, + HOST_ACCESS_DENIED_MESSAGE, + HOST_ACCESS_REVOKED_MESSAGE, + boundedErrorExcerpt, + hostAccessDenialMessage, +} from '../upstreamErrors.js' + +vi.mock('../config.js', () => ({ + config: { + rpcProxyBaseUrl: 'http://proxy', + externalRestApiBaseUrl: 'http://rest', + enableDevLoginUi: false, + requestTimeoutMs: 60000, + appName: 'test', + }, +})) + +afterEach(() => { + vi.unstubAllGlobals() +}) + +function stubResponse(status: number, statusText: string, body: string, headers = {}) { + const fetchMock = vi.fn(async () => new Response(body, { status, statusText, headers })) + vi.stubGlobal('fetch', fetchMock) + return fetchMock +} + +async function rejection(promise: Promise): Promise { + const error = await promise.then( + () => null, + (e: unknown) => e + ) + expect(error).toBeInstanceOf(ApiError) + return error as ApiError +} + +// The rpc-proxy own-denial body, exactly as the cross-layer contract defines it. +const denialBody = (code?: string) => + JSON.stringify({ error: 'Forbidden: user cannot access this host', ...(code ? { code } : {}) }) + +describe('requestJson Host-access 403 mapping', () => { + it('maps code host_access_revoked to the single confirmed-revocation message', async () => { + const fetchMock = stubResponse(403, 'Forbidden', denialBody('host_access_revoked'), { + 'retry-after': '7', + }) + + const error = await rejection(requestJson('GET', 'http://proxy/x')) + + expect(fetchMock).toHaveBeenCalledTimes(1) + expect(error.message).toBe(HOST_ACCESS_REVOKED_MESSAGE) + expect(error.message).toBe('403 Forbidden: host_access_revoked') + expect(error.status).toBe(403) + expect(error.bodyText).toBe(denialBody('host_access_revoked')) + expect(error.retryAfter).toBe('7') + }) + + it('maps code host_access_denied to a message that is not a confirmed revocation', async () => { + stubResponse(403, 'Forbidden', denialBody('host_access_denied')) + + const error = await rejection(requestJson('GET', 'http://proxy/x')) + + expect(error.message).toBe(HOST_ACCESS_DENIED_MESSAGE) + expect(error.message).toBe('403 Forbidden: host_access_denied') + expect(error.message).not.toContain('host_access_revoked') + }) + + it.each([ + [ + 'a JSON body whose error is the revoked token but has no code', + '{"error":"host_access_revoked"}', + ], + ['the pre-contract denial body without a code', denialBody()], + ['a text body ending in the revoked token', 'Forbidden: host_access_revoked'], + ['a JSON message field carrying the revoked token', '{"message":"host_access_revoked"}'], + ['a code of another kind', '{"code":"missing_scope","error":"host_access_revoked"}'], + ])('never forges a confirmed revocation from %s', async (_label, body) => { + const fetchMock = stubResponse(403, 'Forbidden', body) + + const error = await rejection(requestJson('GET', 'http://proxy/x')) + + // Liveness witness: the 403 path ran and produced the ordinary status text. + expect(fetchMock).toHaveBeenCalledTimes(1) + expect(error.message.startsWith('403 Forbidden: ')).toBe(true) + expect(error.status).toBe(403) + expect(error.message).not.toContain('host_access_revoked') + }) + + it('leaves a non-403 body that mentions the token as an ordinary failure', async () => { + stubResponse(502, 'Bad Gateway', '{"code":"host_access_revoked"}') + + const error = await rejection(requestJson('GET', 'http://proxy/x')) + + expect(error.message.startsWith('502 Bad Gateway: ')).toBe(true) + expect(error.message).not.toContain('host_access_revoked') + }) +}) + +describe('bounded upstream error excerpts', () => { + const HUGE = 'y'.repeat(100 * 1024) + + it('keeps a 100 KB body out of the requestJson message but on bodyText, status first', async () => { + const body = JSON.stringify({ detail: HUGE }) + stubResponse(500, 'Internal Server Error', body) + + const error = await rejection(requestJson('GET', 'http://proxy/x')) + + expect(error.message.startsWith('500 Internal Server Error: ')).toBe(true) + expect(error.message.length).toBeLessThan(600) + expect(error.bodyText).toBe(body) + expect(error.status).toBe(500) + expect(/^(\d{3}) /.exec(error.message)?.[1]).toBe('500') + }) + + it('bounds an oversized error string and message field of a JSON body', async () => { + stubResponse(400, 'Bad Request', JSON.stringify({ error: HUGE, message: HUGE })) + + const error = await rejection(requestJson('GET', 'http://proxy/x')) + + expect(error.message.startsWith('400 Bad Request: ')).toBe(true) + expect(error.message.length).toBeLessThan(600) + }) + + it.each([ + ['prewarmHost', (c: RpcProxyClient) => c.prewarmHost('t', 'host'), 'Prewarm failed (500): '], + [ + 'approveToolCall', + (c: RpcProxyClient) => c.approveToolCall('t', 'host', 'task', 'tool'), + 'Approve failed (500): ', + ], + [ + 'listArtifacts', + (c: RpcProxyClient) => c.listArtifacts('t', 'host'), + 'List artifacts failed (500): ', + ], + [ + 'getHostModels', + (c: RpcProxyClient) => c.getHostModels('t', 'host', 'chat'), + 'Get host models failed (500): ', + ], + [ + 'setHostModel', + (c: RpcProxyClient) => c.setHostModel('t', 'host', 'chat', 'model'), + 'Set host model failed (500): ', + ], + ])('bounds the raw-fetch message of %s', async (_name, call, prefix) => { + const fetchMock = stubResponse(500, 'Internal Server Error', HUGE) + + const error = await rejection(call(new RpcProxyClient())) + + expect(fetchMock).toHaveBeenCalledTimes(1) + expect(error.message.startsWith(prefix)).toBe(true) + expect(error.message.length).toBeLessThan(600) + expect(error.bodyText).toBe(HUGE) + expect(error.status).toBe(500) + }) +}) + +describe('bounded upstream error excerpts at every remaining call site', () => { + const HUGE = 'y'.repeat(100 * 1024) + const BOUNDED = `${'y'.repeat(ERROR_EXCERPT_MAX_CHARS)}…` + const signal = () => new AbortController().signal + + // Each entry drives one `boundedErrorExcerpt` call site through the public + // method that owns it and names the exact message it must produce. + const SITES: Array<[string, () => Promise, string]> = [ + [ + 'RpcProxyClient.mintSandboxUiSession', + () => new RpcProxyClient().mintSandboxUiSession('t', 'ns', 'app'), + `sandbox-ui session mint failed (500): ${BOUNDED}`, + ], + [ + 'RpcProxyClient.requestSandboxUiOauthAuthorizeUrl', + () => new RpcProxyClient().requestSandboxUiOauthAuthorizeUrl('t', 'ns', 'app', 'client'), + `sandbox-ui authorize-url request failed (500): ${BOUNDED}`, + ], + [ + 'RpcProxyClient.requestMcpOauthAuthorizeUrl', + () => new RpcProxyClient().requestMcpOauthAuthorizeUrl('t', 'server'), + `mcp-oauth authorize-url request failed (500): ${BOUNDED}`, + ], + [ + 'RpcProxyClient.openHostStatusStream', + () => new RpcProxyClient().openHostStatusStream('t', 'host', () => undefined, signal()), + `Host stream failed (500): ${BOUNDED}`, + ], + [ + 'RpcProxyClient.denyToolCall', + () => new RpcProxyClient().denyToolCall('t', 'host', 'task', 'tool', 'reason'), + `Deny failed (500): ${BOUNDED}`, + ], + [ + 'RpcProxyClient.cancelTask', + () => new RpcProxyClient().cancelTask('t', 'host', 'task'), + `cancelTask failed (500): ${BOUNDED}`, + ], + [ + 'RpcProxyClient.downloadArtifact', + () => new RpcProxyClient().downloadArtifact('t', 'host', 'file.txt'), + `Download artifact failed (500): ${BOUNDED}`, + ], + [ + 'RpcProxyClient.getContextBreakdown', + () => new RpcProxyClient().getContextBreakdown('t', 'host', 'agent', 'chat'), + `Get context breakdown failed (500): ${BOUNDED}`, + ], + [ + 'RpcProxyClient.postDesktopSession', + () => new RpcProxyClient().postDesktopSession('t', 'host'), + `Desktop session exchange failed: 500 ${BOUNDED}`, + ], + [ + 'AuthClient.openWorkflowNotificationStream', + () => new AuthClient().openWorkflowNotificationStream('t', () => undefined, signal()), + `Notification stream failed (500): ${BOUNDED}`, + ], + [ + 'AuthClient.downloadWorkflowRunArtifact', + () => new AuthClient().downloadWorkflowRunArtifact('t', 'ns', 'flow', 'run', 'out.txt'), + `Download workflow artifact failed (500): ${BOUNDED}`, + ], + [ + 'SharedFilesClient.downloadFile', + () => new SharedFilesClient().downloadFile('t', 'ctx', 'sfs', 'a.txt'), + `500 Internal Server Error: ${BOUNDED}`, + ], + ] + + it.each(SITES)('bounds the upstream body in the message of %s', async (_name, call, message) => { + const fetchMock = stubResponse(500, 'Internal Server Error', HUGE) + + const error = await call().then( + () => null, + (e: unknown) => e as Error & { status?: number; bodyText?: string; body?: string } + ) + + // Liveness witness: the request went out and the failure path really ran. + expect(fetchMock).toHaveBeenCalledTimes(1) + expect(error).toBeInstanceOf(Error) + // The message carries the 512-character excerpt and nothing beyond it, while the + // full upstream body stays on the typed error for diagnostics. + expect(error?.message).toBe(message) + expect(error?.message.length).toBeLessThan(700) + const fullBody = error?.bodyText ?? error?.body + if (fullBody !== undefined) expect(fullBody).toBe(HUGE) + }) + + it('keeps the typed status of the sandbox-ui session error while bounding its message', async () => { + stubResponse(500, 'Internal Server Error', HUGE) + + const error = await new RpcProxyClient().mintSandboxUiSession('t', 'ns', 'app').then( + () => null, + (e: unknown) => e + ) + + expect(error).toBeInstanceOf(SandboxUiSessionError) + expect((error as SandboxUiSessionError).status).toBe(500) + expect((error as SandboxUiSessionError).body).toBe(HUGE) + }) +}) + +describe('upstreamErrors helpers', () => { + it('collapses whitespace, cuts at the bound with an ellipsis and defuses reserved tokens', () => { + expect(boundedErrorExcerpt(' a\n\n b\t c ')).toBe('a b c') + const cut = boundedErrorExcerpt('z'.repeat(ERROR_EXCERPT_MAX_CHARS + 50)) + expect(cut).toBe(`${'z'.repeat(ERROR_EXCERPT_MAX_CHARS)}…`) + const defused = boundedErrorExcerpt('a host_access_revoked and HOST_ACCESS_DENIED here') + expect(defused).toBe('a host-access-revoked and host-access-DENIED here') + }) + + it('cuts exactly above the bound: 511 and 512 characters are kept, 513 gets the ellipsis', () => { + expect(ERROR_EXCERPT_MAX_CHARS).toBe(512) + const kept511 = 'q'.repeat(ERROR_EXCERPT_MAX_CHARS - 1) + const kept512 = 'q'.repeat(ERROR_EXCERPT_MAX_CHARS) + const cut513 = 'q'.repeat(ERROR_EXCERPT_MAX_CHARS + 1) + + expect(boundedErrorExcerpt(kept511)).toBe(kept511) + expect(boundedErrorExcerpt(kept512)).toBe(kept512) + expect(boundedErrorExcerpt(cut513)).toBe(`${kept512}…`) + }) + + it('decides Host-access messages from the parsed code of a 403 only', () => { + expect(hostAccessDenialMessage(403, '{"code":"host_access_revoked"}')).toBe( + HOST_ACCESS_REVOKED_MESSAGE + ) + expect(hostAccessDenialMessage(403, '{"code":"host_access_denied"}')).toBe( + HOST_ACCESS_DENIED_MESSAGE + ) + expect(hostAccessDenialMessage(401, '{"code":"host_access_revoked"}')).toBeNull() + expect(hostAccessDenialMessage(403, 'host_access_revoked')).toBeNull() + expect(hostAccessDenialMessage(403, '{"error":"host_access_revoked"}')).toBeNull() + expect(hostAccessDenialMessage(403, '')).toBeNull() + }) +}) diff --git a/desktop-app/src/__tests__/ipc.chatGetIndex.test.ts b/desktop-app/src/__tests__/ipc.chatGetIndex.test.ts new file mode 100644 index 0000000000..c72bf81c09 --- /dev/null +++ b/desktop-app/src/__tests__/ipc.chatGetIndex.test.ts @@ -0,0 +1,171 @@ +/** + * `chat:getIndex` must keep returning the index WITH its tombstones when the + * deleted-chat cleanup retry that precedes it fails (scope changed mid-flight, + * no session, unreadable chats directory). The renderer filters deleted chats + * with those tombstones; a rejected read leaves them visible. + */ +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from 'vitest' +import { promises as fs } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import type { AppService } from '../appService.js' +import type { ChatIndex } from '../types.js' + +type Handler = (event: unknown, payload: unknown) => Promise + +const electron = vi.hoisted(() => ({ handlers: new Map() })) + +vi.mock('electron', () => ({ + BrowserWindow: { fromWebContents: vi.fn(), getAllWindows: vi.fn(() => []) }, + Notification: vi.fn(), + app: { on: vi.fn(), getPath: vi.fn(() => '/nonexistent'), getVersion: vi.fn(), isPackaged: true }, + clipboard: { writeText: vi.fn() }, + dialog: { showSaveDialog: vi.fn() }, + ipcMain: { + handle: vi.fn((channel: string, handler: Handler) => { + electron.handlers.set(channel, handler) + }), + on: vi.fn(), + removeHandler: vi.fn(), + }, +})) +// ipc.ts only uses AppService as a type at registration. +vi.mock('../appService.js', () => ({ AppService: class {} })) + +const TEAM_A = { environmentKey: 'env-a', userId: 'user-a', teamId: 'team-a' } +const TEAM_B = { ...TEAM_A, teamId: 'team-b' } +const OTHER_USER = { ...TEAM_A, userId: 'user-b' } + +let authority: { authorityScope: typeof TEAM_A; sessionGeneration: number } | Error = { + authorityScope: TEAM_A, + sessionGeneration: 1, +} +const service = { + getChatDeletionFenceAuthority: vi.fn(() => { + if (authority instanceof Error) throw authority + return authority + }), +} +const trusted = { senderFrame: { url: 'file:///app/index.html' } } +const call = (channel: string, payload: unknown) => { + const handler = electron.handlers.get(channel) + if (!handler) throw new Error(`no IPC handler registered for ${channel}`) + return handler(trusted, payload) +} +let base: string + +beforeAll(async () => { + base = await fs.mkdtemp(join(tmpdir(), 'ipc-chat-getindex-')) + const binding = await import('../chatStoreBinding.js') + binding.__setChatStoreBaseDirForTests(base) + await binding.bindChatStoreForUser('user-a', 'env-a', { teamId: 'team-a' }) + const { registerIpcHandlers } = await import('../ipc.js') + registerIpcHandlers(service as unknown as AppService) + await call('chat:create', { agentRef: 'agent-1', chatId: 'deleted-chat' }) + const fence = await call('chat:captureDeleteFence', { expectedAuthorityScope: TEAM_A }) + await call('chat:delete', { version: 3, fence, agentRef: 'agent-1', chatId: 'deleted-chat' }) +}) + +afterEach(() => { + authority = { authorityScope: TEAM_A, sessionGeneration: 1 } + vi.restoreAllMocks() +}) + +afterAll(async () => { + const binding = await import('../chatStoreBinding.js') + binding.__setChatStoreBaseDirForTests(null) + binding.unbindChatStore() + await fs.rm(base, { recursive: true, force: true }) +}) + +const tombstones = (index: unknown) => + (index as ChatIndex).deletedChatTombstones?.map(item => item.chatId) + +describe('chat:getIndex', () => { + it('returns the tombstones when the cleanup retry runs normally', async () => { + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + + const index = await call('chat:getIndex', { agentRef: 'agent-1' }) + + expect(tombstones(index)).toEqual(['deleted-chat']) + expect(service.getChatDeletionFenceAuthority).toHaveBeenCalled() + expect(warn).not.toHaveBeenCalled() + }) + + it('still returns the tombstones when the session scope changed during the retry', async () => { + // A transient identity change: the main service reports another user while + // the store is still bound to user-a, so the retry rejects with a + // scope-changed error. (A team hop alone no longer rejects: the retry is + // bound to environment + user.) + authority = { authorityScope: OTHER_USER, sessionGeneration: 2 } + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + + const index = await call('chat:getIndex', { agentRef: 'agent-1' }) + + // Liveness witness: the retry ran and failed, and the failure was reported. + const report = warn.mock.calls.find(entry => String(entry[0]).includes('[chat:getIndex]')) + expect(String(report?.[0])).toContain('agent "agent-1"') + expect(String((report?.[1] as Error).message)).toContain('scope changed') + expect(tombstones(index)).toEqual(['deleted-chat']) + }) + + it('runs the cleanup retry without a failure report after a team switch of the same user', async () => { + authority = { authorityScope: TEAM_B, sessionGeneration: 2 } + service.getChatDeletionFenceAuthority.mockClear() + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + + const index = await call('chat:getIndex', { agentRef: 'agent-1' }) + + // Liveness witness: the retry consulted the authority (it did run) and returned the index. + expect(service.getChatDeletionFenceAuthority).toHaveBeenCalledTimes(1) + expect(tombstones(index)).toEqual(['deleted-chat']) + expect(warn).not.toHaveBeenCalled() + }) + + it('still returns the tombstones when there is no authenticated session', async () => { + authority = new Error('Not authenticated') + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + + const index = await call('chat:getIndex', { agentRef: 'agent-1' }) + + const report = warn.mock.calls.find(entry => String(entry[0]).includes('[chat:getIndex]')) + expect(String((report?.[1] as Error).message)).toBe('Not authenticated') + expect(tombstones(index)).toEqual(['deleted-chat']) + }) + + it('retries only the cleanups of the agent being read once the bind sweep ran', async () => { + const binding = await import('../chatStoreBinding.js') + // Earlier reads in this file already consumed the sweep of the bind. + const retry = vi.spyOn(binding.requireChatStore(), 'retryPendingDeleteCleanups') + const readdir = vi.spyOn(fs, 'readdir') + + const index = await call('chat:getIndex', { agentRef: 'agent-1' }) + + expect(retry).toHaveBeenCalledWith(TEAM_A, 'agent-1') + expect(readdir.mock.calls.map(entry => String(entry[0]))).not.toContain( + join(base, 'env-a', 'user-a') + ) + expect(tombstones(index)).toEqual(['deleted-chat']) + }) + + it('still returns the tombstones and reports the code when the chats directory is unreadable', async () => { + const binding = await import('../chatStoreBinding.js') + // Re-binding the same user (a team switch or a catalog refresh) arms the + // sweep over every agent directory for the next read. + await binding.bindChatStoreForUser('user-a', 'env-a', { teamId: 'team-a' }) + const readdir = vi + .spyOn(fs, 'readdir') + .mockRejectedValueOnce(Object.assign(new Error('permission denied'), { code: 'EACCES' })) + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + + const index = await call('chat:getIndex', { agentRef: 'agent-1' }) + + const report = warn.mock.calls.find(entry => String(entry[0]).includes('[chat:getIndex]')) + expect(String(report?.[0])).toContain('(EACCES)') + expect(tombstones(index)).toEqual(['deleted-chat']) + // The sweep that failed stays armed: the next read walks the directory again. + await call('chat:getIndex', { agentRef: 'agent-1' }) + const userDir = join(base, 'env-a', 'user-a') + expect(readdir.mock.calls.map(entry => String(entry[0]))).toEqual([userDir, userDir]) + }) +}) diff --git a/desktop-app/src/__tests__/rpcProxyClient.test.ts b/desktop-app/src/__tests__/rpcProxyClient.test.ts index c7f5de222b..e4170c8733 100644 --- a/desktop-app/src/__tests__/rpcProxyClient.test.ts +++ b/desktop-app/src/__tests__/rpcProxyClient.test.ts @@ -44,6 +44,111 @@ describe('RpcProxyClient.listSessions', () => { } ) + // The rpc-proxy denial bodies exactly as the contract defines them: status 403, + // the human `error` text, and a machine `code` that says whether access was + // actually removed (`host_access_revoked`) or merely refused (`host_access_denied`). + const rpcProxyDenial = (code?: string) => + JSON.stringify({ error: 'Forbidden: user cannot access this host', ...(code ? { code } : {}) }) + + const stubForbidden = (body: string) => { + const fetchMock = vi.fn().mockResolvedValue({ ok: false, status: 403, text: async () => body }) + vi.stubGlobal('fetch', fetchMock) + return fetchMock + } + + it('projects a confirmed revocation (code host_access_revoked) from every raw-fetch site', async () => { + const fetchMock = stubForbidden(rpcProxyDenial('host_access_revoked')) + + await expect(client.listSessions('token', 'host')).rejects.toThrow( + '403 Forbidden: host_access_revoked' + ) + await expect(client.loadSessionMessages('token', 'host', 'agent-a', 'chat-a')).rejects.toThrow( + '403 Forbidden: host_access_revoked' + ) + await expect(client.renameSession('token', 'host', 'agent-a', 'chat-a', 'T')).rejects.toThrow( + '403 Forbidden: host_access_revoked' + ) + expect(fetchMock).toHaveBeenCalledTimes(3) + }) + + it('projects an own-denial without revocation (code host_access_denied) as its own message', async () => { + const fetchMock = stubForbidden(rpcProxyDenial('host_access_denied')) + + for (const call of [ + () => client.listSessions('token', 'host'), + () => client.loadSessionMessages('token', 'host', 'agent-a', 'chat-a'), + () => client.renameSession('token', 'host', 'agent-a', 'chat-a', 'T'), + ]) { + const error = await call().then( + () => null, + (e: unknown) => e as Error + ) + expect(error?.message).toBe('403 Forbidden: host_access_denied') + expect(error?.message).not.toContain('host_access_revoked') + } + expect(fetchMock).toHaveBeenCalledTimes(3) + }) + + it.each([ + ['the pre-contract body (error text only, no code)', rpcProxyDenial()], + [ + 'a JSON body whose error is the revoked token but has no code', + '{"error":"host_access_revoked"}', + ], + [ + 'a JSON body carrying the revoked token in a non-code field', + '{"reason":"host_access_revoked"}', + ], + ['a text body ending in the revoked token', 'Forbidden: host_access_revoked'], + ['a code of another kind', '{"code":"missing_scope","error":"host_access_revoked"}'], + ])('does not forge a confirmed revocation from %s', async (_label, body) => { + const fetchMock = stubForbidden(body) + + for (const call of [ + () => client.listSessions('token', 'host'), + () => client.loadSessionMessages('token', 'host', 'agent-a', 'chat-a'), + () => client.renameSession('token', 'host', 'agent-a', 'chat-a', 'T'), + ]) { + const error = await call().then( + () => null, + (e: unknown) => e as Error & { status?: number } + ) + // Liveness witness: the failure path ran and kept the plain 403 text. + expect(error?.message).toMatch(/failed \(403\)/) + expect(error?.status).toBe(403) + expect(error?.message).not.toContain('host_access_revoked') + } + expect(fetchMock).toHaveBeenCalledTimes(3) + }) + + it('bounds an oversized upstream body in the message while ApiError keeps it whole', async () => { + const huge = `{"detail":"${'x'.repeat(100 * 1024)}"}` + stubForbidden(huge) + + const error = (await client.listSessions('token', 'host').then( + () => null, + (e: unknown) => e + )) as Error & { status?: number; bodyText?: string } + expect(error).toBeInstanceOf(Error) + // Status stays first so the renderer's status parsing keeps working. + expect(error.message.startsWith('List sessions failed (403): ')).toBe(true) + expect(error.message.length).toBeLessThan(600) + expect(error.bodyText).toBe(huge) + }) + + it('keeps a generic catalog 403 uncertain', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockResolvedValue({ + ok: false, + status: 403, + text: async () => JSON.stringify({ error: 'missing scope' }), + }) + ) + + await expect(client.listSessions('token', 'host')).rejects.toThrow('List sessions failed (403)') + }) + it('accepts legacy session catalog responses that omit optional metadata', async () => { vi.stubGlobal( 'fetch', @@ -110,6 +215,47 @@ describe('RpcProxyClient.listSessions', () => { }) }) + // R3-L10: the `rpc:listSessions` IPC handler refuses a cursor longer than + // 2048 characters, so a page cursor that long must never reach the renderer. + it.each([ + [2048, true], + [3000, false], + ])('keeps a %s-character nextCursor only within the IPC cursor limit', async (length, kept) => { + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + const nextCursor = 'c'.repeat(length) + vi.stubGlobal( + 'fetch', + vi.fn().mockResolvedValue({ + ok: true, + json: async () => ({ + items: [ + { + agent: 'agent-a', + chatId: 'chat-a', + turnCount: 1, + lastActivityAt: '2026-01-01T00:00:00.000Z', + }, + ], + nextCursor, + }), + }) + ) + + const result = await client.listSessions('token', 'host') + + // Witness: the page itself is still delivered in both cases. + expect(result.items.map(item => item.chatId)).toEqual(['chat-a']) + expect(result.nextCursor).toBe(kept ? nextCursor : undefined) + if (kept) { + expect(warn).not.toHaveBeenCalled() + } else { + expect(warn).toHaveBeenCalledTimes(1) + const logged = warn.mock.calls[0].map(String).join(' ') + expect(logged).toContain(String(length)) + expect(logged).not.toContain('ccc') + } + }) + it('keeps valid catalog entries when one item is malformed and omits unknown states', async () => { vi.stubGlobal( 'fetch', diff --git a/desktop-app/src/appService.ts b/desktop-app/src/appService.ts index c64b322e03..850b24632f 100644 --- a/desktop-app/src/appService.ts +++ b/desktop-app/src/appService.ts @@ -42,6 +42,7 @@ import { AccessCatalog, AgentWithMcpServers, ApprovalDecisionResult, + ChatAuthorityScope, ContextBreakdownResult, DependencyHealth, DesktopAppInfo, @@ -871,6 +872,12 @@ export class AppService { // token in sessionToken. Existing GFS jobs keep their captured token and // scope, but new GFS operations must not capture that temporary token. private gfsTransientTeamHopDepth = 0 + // While a `runWithTeamContext` hop that will restore the original team is in + // flight, `me.teamId` holds the borrowed team. The chat store (and the delete + // fence derived from it) stays on the original team: rebinding it for the hop + // and back would expose the deliberate team's chats to the borrowed scope for + // the length of an RPC. + private chatStoreHomeTeamId: string | null = null // A finite RPC may temporarily borrow another team session. Keep the // deliberate user-owned GFS scope stable across that hop so in-flight jobs // are not invalidated by the transient `me` value. @@ -899,6 +906,9 @@ export class AppService { private workflowTeamByKey = new Map() private readonly prewarmAttemptAtByHostRef = new Map() private readonly prewarmReemitLoopHostRefs = new Set() + private readonly prewarmReemitAbortControllers = new Map() + private prewarmAuthEpoch = 0 + private prewarmAuthTransitionCount = 0 private hostStatusStreams = new Map< string, { @@ -1005,12 +1015,35 @@ export class AppService { } } + /** The team the chat store and the delete fence are scoped to (see chatStoreHomeTeamId). */ + private chatStoreTeamId(): string | null { + if (this.chatStoreHomeTeamId !== null) return this.chatStoreHomeTeamId + return this.me?.teamId ?? null + } + private async bindCurrentChatStore(userId: string): Promise { await bindChatStoreForUser(userId, getActiveEnvKey(), { legacyEnvKeys: getActiveLegacyEnvKeys(), + teamId: this.me?.id === userId ? this.chatStoreTeamId() : null, }) } + getChatDeletionFenceAuthority(): { + authorityScope: ChatAuthorityScope + sessionGeneration: number + } { + this.requireSessionToken() + if (!this.me) throw new Error('Not authenticated') + return { + authorityScope: { + environmentKey: getActiveEnvKey(), + userId: this.me.id, + teamId: String(this.chatStoreTeamId() || '').trim() || null, + }, + sessionGeneration: this.sessionGeneration, + } + } + private async commitSessionToken( token: string, options: { refreshMe?: boolean } = {} @@ -1150,6 +1183,7 @@ export class AppService { const shouldSwitch = originalTeamId !== targetTeamId const shouldRestore = Boolean(originalTeamId && shouldSwitch) const releaseTransientHop = shouldSwitch ? this.enterGfsTransientTeamHop() : undefined + if (shouldRestore) this.chatStoreHomeTeamId = originalTeamId try { if (shouldSwitch) { @@ -1175,6 +1209,25 @@ export class AppService { } } } finally { + if (shouldRestore) { + this.chatStoreHomeTeamId = null + // A failed switch back leaves the session on the hop team while the + // store is still bound to the pinned home team. Rebind so the store + // scope and the delete-fence authority (both derived from + // chatStoreTeamId) agree again. A rebind failure is logged and never + // replaces the error of the operation or of the failed restore. + const sessionUserId = this.me?.id + if (sessionUserId && this.me?.teamId !== originalTeamId) { + try { + await this.bindCurrentChatStore(sessionUserId) + } catch (rebindError) { + console.error( + '[AppService] Failed to rebind the chat store after a failed team restore:', + rebindError + ) + } + } + } releaseTransientHop?.() } } finally { @@ -1246,6 +1299,7 @@ export class AppService { this.gfsAuthEpoch += 1 this.gfsDispatchBlocked = true this.gfsScopeIdentity = null + this.cancelPrewarmReemissions() this.stopAllStreams() this.sessionToken = null this.me = null @@ -1264,6 +1318,26 @@ export class AppService { this.gfsScopeIdentity = identityOverride ?? (this.me ? desktopGfsUploadIdentity(this.me) : null) } + private cancelPrewarmReemissions(): void { + this.prewarmAuthEpoch += 1 + this.prewarmAttemptAtByHostRef.clear() + this.prewarmReemitLoopHostRefs.clear() + for (const controller of this.prewarmReemitAbortControllers.values()) controller.abort() + this.prewarmReemitAbortControllers.clear() + } + + private beginPrewarmAuthTransition(): () => void { + this.prewarmAuthTransitionCount += 1 + this.cancelPrewarmReemissions() + let released = false + return () => { + if (released) return + released = true + this.cancelPrewarmReemissions() + this.prewarmAuthTransitionCount -= 1 + } + } + private currentDesktopGfsUploadScope( driveValue: string | undefined, options: { allowTransientTeamHop?: boolean } = {} @@ -1393,7 +1467,10 @@ export class AppService { if (this.sessionGeneration !== restoreGeneration || this.sessionToken !== token) { return { authenticated: Boolean(this.sessionToken && this.me), me: this.me } } - await bindChatStoreForUser(restoredMe.id, envKey, { legacyEnvKeys }) + await bindChatStoreForUser(restoredMe.id, envKey, { + legacyEnvKeys, + teamId: restoredMe.teamId, + }) if (this.sessionGeneration !== restoreGeneration || this.sessionToken !== token) { if (this.me) { await this.bindCurrentChatStore(this.me.id) @@ -1460,44 +1537,49 @@ export class AppService { token: string me: SessionMe }): Promise { - const previousToken = this.sessionToken - const previousMe = this.me - const previousGeneration = this.sessionGeneration - const hadAuthenticatedScope = Boolean(previousToken && previousMe) - if (hadAuthenticatedScope) { - try { - await this.suspendDesktopGfsUploadsForAuthBoundary() - } catch (error) { + const releasePrewarm = this.beginPrewarmAuthTransition() + try { + const previousToken = this.sessionToken + const previousMe = this.me + const previousGeneration = this.sessionGeneration + const hadAuthenticatedScope = Boolean(previousToken && previousMe) + if (hadAuthenticatedScope) { + try { + await this.suspendDesktopGfsUploadsForAuthBoundary() + } catch (error) { + if ( + this.sessionGeneration === previousGeneration && + this.sessionToken === previousToken && + this.me === previousMe + ) { + this.activateGfsAuthScope() + } + throw error + } if ( - this.sessionGeneration === previousGeneration && - this.sessionToken === previousToken && - this.me === previousMe + this.sessionGeneration !== previousGeneration || + this.sessionToken !== previousToken || + this.me !== previousMe ) { - this.activateGfsAuthScope() + throw new Error('stale_auth_epoch: authenticated scope changed during login replacement') } - throw error - } - if ( - this.sessionGeneration !== previousGeneration || - this.sessionToken !== previousToken || - this.me !== previousMe - ) { - throw new Error('stale_auth_epoch: authenticated scope changed during login replacement') } + this.logoutInProgress = false + this.sessionGeneration += 1 + this.sessionToken = result.token + this.me = result.me + await this.bindCurrentChatStore(result.me.id) + this.accessCatalog = null + this.teamDirectoryCache = null + this.workflowApprovalTeamById.clear() + this.workflowTeamByKey.clear() + this.rpcTokenManager.clear() + await this.tokenStore.setSessionToken(result.token, getActiveEnvKey()) + this.activateGfsAuthScope() + return { authenticated: true, me: result.me } + } finally { + releasePrewarm() } - this.logoutInProgress = false - this.sessionGeneration += 1 - this.sessionToken = result.token - this.me = result.me - await this.bindCurrentChatStore(result.me.id) - this.accessCatalog = null - this.teamDirectoryCache = null - this.workflowApprovalTeamById.clear() - this.workflowTeamByKey.clear() - this.rpcTokenManager.clear() - await this.tokenStore.setSessionToken(result.token, getActiveEnvKey()) - this.activateGfsAuthScope() - return { authenticated: true, me: result.me } } private async completePasswordLogin(email: string, password: string): Promise { @@ -1540,28 +1622,33 @@ export class AppService { } private async applyRuntimeEnvironmentChange(operation: () => Promise): Promise { - const oldEnvKey = getActiveEnvKey() - const oldBaseUrl = normalizeDesktopUploadBaseUrl(config.externalRestApiBaseUrl) - const oldLegacyEnvKeys = getActiveLegacyEnvKeys() - const hadAuthenticatedScope = Boolean(this.sessionToken && this.me) - // Invalidate a restore that may still be awaiting keychain/getMe before it - // can bind an old-environment token to the newly selected runtime. - this.sessionGeneration += 1 - if (hadAuthenticatedScope) await this.suspendDesktopGfsUploadsForAuthBoundary() + const releasePrewarm = this.beginPrewarmAuthTransition() try { - await operation() - } catch (error) { - if (hadAuthenticatedScope && this.sessionToken && this.me) this.activateGfsAuthScope() - throw error - } - const boundaryChanged = - getActiveEnvKey() !== oldEnvKey || - normalizeDesktopUploadBaseUrl(config.externalRestApiBaseUrl) !== oldBaseUrl - if (boundaryChanged) { - this.clearAuthenticatedSessionState() - await this.tokenStore.clearSessionToken(oldEnvKey, { legacyEnvKeys: oldLegacyEnvKeys }) - } else if (hadAuthenticatedScope && this.sessionToken && this.me) { - this.activateGfsAuthScope() + const oldEnvKey = getActiveEnvKey() + const oldBaseUrl = normalizeDesktopUploadBaseUrl(config.externalRestApiBaseUrl) + const oldLegacyEnvKeys = getActiveLegacyEnvKeys() + const hadAuthenticatedScope = Boolean(this.sessionToken && this.me) + // Invalidate a restore that may still be awaiting keychain/getMe before it + // can bind an old-environment token to the newly selected runtime. + this.sessionGeneration += 1 + if (hadAuthenticatedScope) await this.suspendDesktopGfsUploadsForAuthBoundary() + try { + await operation() + } catch (error) { + if (hadAuthenticatedScope && this.sessionToken && this.me) this.activateGfsAuthScope() + throw error + } + const boundaryChanged = + getActiveEnvKey() !== oldEnvKey || + normalizeDesktopUploadBaseUrl(config.externalRestApiBaseUrl) !== oldBaseUrl + if (boundaryChanged) { + this.clearAuthenticatedSessionState() + await this.tokenStore.clearSessionToken(oldEnvKey, { legacyEnvKeys: oldLegacyEnvKeys }) + } else if (hadAuthenticatedScope && this.sessionToken && this.me) { + this.activateGfsAuthScope() + } + } finally { + releasePrewarm() } } @@ -1903,6 +1990,7 @@ export class AppService { async logout(): Promise { this.logoutInProgress = true + const releasePrewarm = this.beginPrewarmAuthTransition() try { const envKey = getActiveEnvKey() const legacyEnvKeys = getActiveLegacyEnvKeys() @@ -1913,6 +2001,7 @@ export class AppService { // result must not: the next user of this machine gets nothing of this one's. tryGetPluginSdkRuntime()?.notifySessionChanged(false) } finally { + releasePrewarm() this.logoutInProgress = false } } @@ -3059,6 +3148,9 @@ export class AppService { async switchTeam(teamId: string): Promise { const targetTeamId = String(teamId || '').trim() if (!targetTeamId) throw new Error('teamId is required') + // Explicit team changes fence opportunistic wake immediately. A failed + // switch can safely leave prewarm canceled; the message path remains live. + const releasePrewarm = this.beginPrewarmAuthTransition() const previousIdentity = this.gfsScopeIdentity ?? (this.me ? desktopGfsUploadIdentity(this.me) : undefined) // Keep the new team token fenced until the old GFS jobs have been @@ -3110,6 +3202,7 @@ export class AppService { tryGetPluginSdkRuntime()?.notifySessionChanged(true) return { authenticated: true, me: this.me } } finally { + releasePrewarm() releaseTransientHop() } } @@ -3477,6 +3570,10 @@ export class AppService { if (!targetHostRef) { throw new Error('hostRef is required') } + if (this.prewarmAuthTransitionCount > 0) { + return { requested: false, skipped: 'auth-changed' } + } + const authEpoch = this.prewarmAuthEpoch const now = Date.now() const lastAttemptAt = this.prewarmAttemptAtByHostRef.get(targetHostRef) if (lastAttemptAt !== undefined && now - lastAttemptAt < PREWARM_COOLDOWN_MS) { @@ -3502,14 +3599,23 @@ export class AppService { HOST_WAKEABLE_OPERATION_SCOPES, effectiveHostRefs ) + if (this.prewarmAuthEpoch !== authEpoch || this.prewarmAuthTransitionCount > 0) { + return { requested: false, skipped: 'auth-changed' } + } const result = await this.rpcClient.prewarmHost(rpc.token, targetHostRef) - if (result.status === 'wake-requested') { + if ( + result.status === 'wake-requested' && + this.prewarmAuthEpoch === authEpoch && + this.prewarmAuthTransitionCount === 0 + ) { // HCC has not acted yet and its watch may have lost the event. Run // the bounded background re-emission tied to THIS catalog-driven // invocation — the renderer never awaits it, and the cooldown gate // (checked above, not refreshed by re-emits) keeps it single-flight. this.prewarmReemitLoopHostRefs.add(targetHostRef) - void this.runPrewarmReemissionLoop(targetHostRef, rpc.token) + const controller = new AbortController() + this.prewarmReemitAbortControllers.set(targetHostRef, controller) + void this.runPrewarmReemissionLoop(targetHostRef, rpc.token, authEpoch, controller.signal) } return { requested: true, status: result.status } } catch (error) { @@ -3520,8 +3626,23 @@ export class AppService { } /** Injectable so tests stay deterministic; fake timers advance it. */ - private prewarmReemitDelay = (ms: number): Promise => - new Promise(resolve => setTimeout(resolve, ms)) + private prewarmReemitDelay = (ms: number, signal?: AbortSignal): Promise => + new Promise(resolve => { + if (signal?.aborted) { + resolve() + return + } + const timer = setTimeout(() => { + signal?.removeEventListener('abort', abort) + resolve() + }, ms) + const abort = () => { + clearTimeout(timer) + signal?.removeEventListener('abort', abort) + resolve() + } + signal?.addEventListener('abort', abort, { once: true }) + }) /** * Bounded background wake re-emission (see PREWARM_REEMIT_* constants). @@ -3533,10 +3654,16 @@ export class AppService { * the cooldown window. Deliberately NOT reachable from the host status * stream lifecycle — the anti-flap invariant is unchanged. */ - private async runPrewarmReemissionLoop(hostRef: string, rpcToken: string): Promise { + private async runPrewarmReemissionLoop( + hostRef: string, + rpcToken: string, + authEpoch: number, + signal: AbortSignal + ): Promise { try { for (let attempt = 1; attempt <= PREWARM_REEMIT_MAX_ATTEMPTS; attempt++) { - await this.prewarmReemitDelay(PREWARM_REEMIT_INTERVAL_MS) + await this.prewarmReemitDelay(PREWARM_REEMIT_INTERVAL_MS, signal) + if (signal.aborted || this.prewarmAuthEpoch !== authEpoch) return const result = await this.rpcClient.prewarmHost(rpcToken, hostRef) console.info( `[AppService] prewarmHost re-emit host=${hostRef} attempt=${attempt} status=${result.status}` @@ -3552,7 +3679,10 @@ export class AppService { const message = error instanceof Error ? error.message : String(error) console.warn(`[AppService] prewarmHost re-emit failed host=${hostRef}: ${message}`) } finally { - this.prewarmReemitLoopHostRefs.delete(hostRef) + if (this.prewarmReemitAbortControllers.get(hostRef)?.signal === signal) { + this.prewarmReemitLoopHostRefs.delete(hostRef) + this.prewarmReemitAbortControllers.delete(hostRef) + } } } @@ -4283,12 +4413,10 @@ export class AppService { try { return await this.rpcClient.listSessions(rpc.token, targetHostRef, query) } catch (error) { - const message = error instanceof Error ? error.message : String(error) - if (message.includes('401') && message.toLowerCase().includes('missing token')) { - console.warn( - '[AppService] Session catalog unavailable because the runtime rejected the session token.' - ) - return { items: [] } + if (AppService.shouldRefreshRpcToken(error)) { + this.rpcTokenManager.clear() + const retried = await this.issueRpcTokenForHostRefs(HOST_SESSION_SCOPES, effectiveHostRefs) + return this.rpcClient.listSessions(retried.token, targetHostRef, query) } throw error } @@ -4316,19 +4444,16 @@ export class AppService { query ) } catch (error) { - const message = error instanceof Error ? error.message : String(error) - if (message.includes('401') && message.toLowerCase().includes('missing token')) { - console.warn( - '[AppService] Session messages unavailable because the runtime rejected the session token.' - ) - return { + if (AppService.shouldRefreshRpcToken(error)) { + this.rpcTokenManager.clear() + const retried = await this.issueRpcTokenForHostRefs(HOST_SESSION_SCOPES, effectiveHostRefs) + return this.rpcClient.loadSessionMessages( + retried.token, + targetHostRef, agent, chatId, - turns: [], - totalTurns: 0, - hasMoreBefore: false, - hasMoreAfter: false, - } + query + ) } throw error } diff --git a/desktop-app/src/authClient.ts b/desktop-app/src/authClient.ts index 74901a6c3e..ec20e13eda 100644 --- a/desktop-app/src/authClient.ts +++ b/desktop-app/src/authClient.ts @@ -25,6 +25,7 @@ import { WorkflowRunCompletedNotification, WorkflowRunsResult, } from './types.js' +import { boundedErrorExcerpt } from './upstreamErrors.js' function url(path: string): string { return `${config.externalRestApiBaseUrl.replace(/\/+$/, '')}${path}` @@ -463,7 +464,7 @@ export class AuthClient { if (!response.ok) { const body = await readErrorBody(response) throw new ApiError( - `Notification stream failed (${response.status}): ${body || response.statusText}`, + `Notification stream failed (${response.status}): ${boundedErrorExcerpt(body) || response.statusText}`, response.status, body ) @@ -602,7 +603,7 @@ export class AuthClient { if (!response.ok) { const body = await readErrorBody(response) throw new ApiError( - `Download workflow artifact failed (${response.status}): ${body || response.statusText}`, + `Download workflow artifact failed (${response.status}): ${boundedErrorExcerpt(body) || response.statusText}`, response.status, body ) diff --git a/desktop-app/src/chatStore.ts b/desktop-app/src/chatStore.ts index 563a7842fe..92459aed7b 100644 --- a/desktop-app/src/chatStore.ts +++ b/desktop-app/src/chatStore.ts @@ -4,6 +4,9 @@ import { dirname, join } from 'node:path' import { mergeAuthoritativeServerMessages, messageServerTurnNumber } from './chatMessageMerge.js' import { assertSafeFilesystemSegment } from './pathSafety.js' import type { + ChatAuthorityScope, + ChatDeleteResult, + ChatDeleteTombstone, ChatFile, ChatIndex, ChatMessage, @@ -69,6 +72,43 @@ function isPositiveSafeInteger(value: unknown): value is number { return typeof value === 'number' && Number.isSafeInteger(value) && value >= 1 } +function isChatAuthorityScope(value: unknown): value is ChatAuthorityScope { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false + const scope = value as Record + return ( + typeof scope.environmentKey === 'string' && + scope.environmentKey.length > 0 && + typeof scope.userId === 'string' && + scope.userId.length > 0 && + (scope.teamId === null || typeof scope.teamId === 'string') + ) +} + +function isChatDeleteTombstone(value: unknown): value is ChatDeleteTombstone { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false + const tombstone = value as Record + return typeof tombstone.chatId === 'string' && isChatAuthorityScope(tombstone.authorityScope) +} + +function sameChatAuthorityScope(left: ChatAuthorityScope, right: ChatAuthorityScope): boolean { + return ( + left.environmentKey === right.environmentKey && + left.userId === right.userId && + left.teamId === right.teamId + ) +} + +/** + * The identity a local deletion is bound to: environment + user. The store is + * rooted at `//`, so together with the per-agent index this + * is the Host/agent identity the chat belongs to. The session team is NOT part + * of it: it is rebound from `me.teamId` on every team switch, and a deletion the + * user confirmed for a Host's chat must keep holding after that switch. + */ +function sameChatDeletionIdentity(left: ChatAuthorityScope, right: ChatAuthorityScope): boolean { + return left.environmentKey === right.environmentKey && left.userId === right.userId +} + function normalizePositiveInteger(value: number | undefined, fallback: number): number { if (value === undefined) return fallback if (!Number.isFinite(value)) return fallback @@ -149,7 +189,13 @@ function samePageEntry(left: ChatPageEntry, right: ChatPageEntry): boolean { ) } -function parseChatIndex(raw: string): ChatIndex { +interface ParsedChatIndex { + index: ChatIndex + /** Tombstone / pending-cleanup entries dropped because they were malformed. */ + droppedTombstoneEntries: number +} + +function parseChatIndex(raw: string): ParsedChatIndex { const parsed = JSON.parse(raw) as unknown if (!parsed || typeof parsed !== 'object') { throw new Error('Invalid chat index') @@ -166,7 +212,10 @@ function parseChatIndex(raw: string): ChatIndex { (candidate.version !== 2 && candidate.version !== 3) || (candidate.lastActiveChatId !== null && typeof candidate.lastActiveChatId !== 'string') || typeof candidate.onboardingDismissed !== 'boolean' || - !Array.isArray(candidate.chats) + !Array.isArray(candidate.chats) || + (candidate.deletedChatTombstones !== undefined && + !Array.isArray(candidate.deletedChatTombstones)) || + (candidate.pendingChatCleanup !== undefined && !Array.isArray(candidate.pendingChatCleanup)) ) { throw new Error('Invalid chat index') } @@ -187,7 +236,62 @@ function parseChatIndex(raw: string): ChatIndex { throw new Error('Invalid chat index entry') } } - return { ...candidate, version: INDEX_VERSION } as unknown as ChatIndex + // One malformed tombstone must not condemn the whole catalog (and with it every + // other tombstone): drop only the malformed entries and report the count. + let droppedTombstoneEntries = 0 + const keepValidTombstones = ( + entries: unknown[] | undefined + ): ChatDeleteTombstone[] | undefined => { + if (entries === undefined) return undefined + const valid = entries.filter(isChatDeleteTombstone) + droppedTombstoneEntries += entries.length - valid.length + return valid + } + const deletedChatTombstones = keepValidTombstones(candidate.deletedChatTombstones as unknown[]) + const pendingChatCleanup = keepValidTombstones(candidate.pendingChatCleanup as unknown[]) + const index = { ...candidate, version: INDEX_VERSION } as unknown as ChatIndex + if (deletedChatTombstones) index.deletedChatTombstones = deletedChatTombstones + if (pendingChatCleanup) index.pendingChatCleanup = pendingChatCleanup + return { index, droppedTombstoneEntries } +} + +/** + * Durable copy of the tombstone and pending-cleanup lists, written next to + * `index.json`. A corrupt catalog is quarantined and replaced by an empty one; + * without this copy every confirmed local deletion of that agent would be + * forgotten and the deleted chats could be recreated from the server. + */ +const TOMBSTONE_SIDECAR_VERSION = 1 + +interface TombstoneSidecar { + version: typeof TOMBSTONE_SIDECAR_VERSION + deletedChatTombstones: ChatDeleteTombstone[] + pendingChatCleanup: ChatDeleteTombstone[] +} + +/** + * `deletedChatTombstones` are intentionally unbounded and never pruned: a + * deleted chat is never reopened, and the tombstone is what keeps a late write + * (a server session sync, a delayed append) from resurrecting it. An entry costs + * ~209 bytes (measured), so the growth is negligible. Do not add a cap or an + * expiry: evicting a tombstone would let exactly that late write bring the + * deleted chat back. Only `pendingChatCleanup` entries are removed, once the + * chat's artifacts are gone. + */ +function mergeTombstones( + primary: ChatDeleteTombstone[] | undefined, + recovered: ChatDeleteTombstone[] +): ChatDeleteTombstone[] { + const merged = [...(primary ?? [])] + for (const candidate of recovered) { + const present = merged.some( + existing => + existing.chatId === candidate.chatId && + sameChatAuthorityScope(existing.authorityScope, candidate.authorityScope) + ) + if (!present) merged.push(candidate) + } + return merged } class UnsupportedChatIndexVersionError extends Error { @@ -289,6 +393,9 @@ function emptyIndex(): ChatIndex { export class ChatStore { private readonly pageSize: number private readonly maxLocalSyncedMessages: number + private authorityScope: ChatAuthorityScope | null = null + /** Set by every bind; the next cleanup retry walks every agent once. */ + private pendingCleanupSweep = true constructor( private readonly baseDir: string, @@ -298,6 +405,47 @@ export class ChatStore { this.maxLocalSyncedMessages = normalizeMaxLocalSyncedMessages(options.maxLocalSyncedMessages) } + setAuthorityScope(scope: ChatAuthorityScope): void { + this.authorityScope = Object.freeze({ ...scope }) + this.pendingCleanupSweep = true + } + + getAuthorityScope(): ChatAuthorityScope | null { + return this.authorityScope ? { ...this.authorityScope } : null + } + + private requireAuthorityScope(expected?: ChatAuthorityScope): ChatAuthorityScope { + const current = this.authorityScope + if (!current || (expected && !sameChatAuthorityScope(current, expected))) { + throw new Error('Chat deletion authority scope changed') + } + return current + } + + /** + * Same check as `requireAuthorityScope` but on the deletion identity + * (environment + user): pending-cleanup retries must keep working after a team + * switch, and must still refuse another user or environment. + */ + private requireDeletionIdentity(expected: ChatAuthorityScope): ChatAuthorityScope { + const current = this.authorityScope + if (!current || !sameChatDeletionIdentity(current, expected)) { + throw new Error('Chat deletion authority scope changed') + } + return current + } + + private isDeletedInScope(index: ChatIndex, chatId: string): boolean { + return Boolean( + index.deletedChatTombstones?.some( + tombstone => + tombstone.chatId === chatId && + (!this.authorityScope || + sameChatDeletionIdentity(tombstone.authorityScope, this.authorityScope)) + ) + ) + } + /** * Index read-modify-write sequences are serialized per agent because every * chat shares one `index.json`. Paged transcript operations use a separate @@ -306,6 +454,9 @@ export class ChatStore { */ private indexChains = new Map>() private chatChains = new Map>() + private tombstoneSidecarSerialized = new Map() + private warnedTombstoneAgents = new Set() + private warnedCleanupFailures = new Set() private cleanedSnapshotSiblingKeys = new Set() private compatibilityCheckedKeys = new Set() private compatibilityWindows = new Map() @@ -373,6 +524,14 @@ export class ChatStore { return join(this.agentDir(agentRef), 'index.json') } + /** + * Dot-prefixed, so it can never equal `${chatId}.json` (chat ids may not start + * with a dot) nor the `index.json` catalog. + */ + private tombstoneSidecarPath(agentRef: string): string { + return join(this.agentDir(agentRef), '.tombstones.json') + } + private corruptIndexPath(agentRef: string): string { return join(this.agentDir(agentRef), '.corrupt', `index-${randomUUID()}.json`) } @@ -1440,22 +1599,91 @@ export class ChatStore { return pruned.meta } + /** + * The tombstones persisted in the sidecar, or null when there is none. + * An unreadable sidecar is reported (once per agent) and treated as absent: + * the catalog it protects is being rebuilt from an empty one already, and the + * next tombstone write replaces it. + */ + private async readTombstoneSidecar(agentRef: string): Promise { + let raw: string + try { + raw = await fs.readFile(this.tombstoneSidecarPath(agentRef), 'utf-8') + } catch (error) { + if (isNotFoundError(error)) return null + throw error + } + try { + const parsed = JSON.parse(raw) as Record + if ( + parsed?.version !== TOMBSTONE_SIDECAR_VERSION || + !Array.isArray(parsed.deletedChatTombstones) || + !Array.isArray(parsed.pendingChatCleanup) + ) { + throw new Error('Invalid tombstone sidecar') + } + return { + version: TOMBSTONE_SIDECAR_VERSION, + deletedChatTombstones: parsed.deletedChatTombstones.filter(isChatDeleteTombstone), + pendingChatCleanup: parsed.pendingChatCleanup.filter(isChatDeleteTombstone), + } + } catch (error) { + if (!this.warnedTombstoneAgents.has(`sidecar:${agentRef}`)) { + this.warnedTombstoneAgents.add(`sidecar:${agentRef}`) + console.warn( + `[chatStore] Unreadable tombstone sidecar for "${agentRef}"; deleted-chat tombstones cannot be recovered`, + error + ) + } + return null + } + } + + /** Adds the sidecar's tombstones (deduplicated) to an index read from disk. */ + private async withRecoveredTombstones(agentRef: string, index: ChatIndex): Promise { + const sidecar = await this.readTombstoneSidecar(agentRef) + if (!sidecar) return index + const deletedChatTombstones = mergeTombstones( + index.deletedChatTombstones, + sidecar.deletedChatTombstones + ) + const pendingChatCleanup = mergeTombstones(index.pendingChatCleanup, sidecar.pendingChatCleanup) + if (deletedChatTombstones.length > 0) index.deletedChatTombstones = deletedChatTombstones + if (pendingChatCleanup.length > 0) index.pendingChatCleanup = pendingChatCleanup + return index + } + async getIndex(agentRef: string): Promise { let raw: string try { raw = await fs.readFile(this.indexPath(agentRef), 'utf-8') } catch (error) { - if (isNotFoundError(error)) return emptyIndex() + // A missing catalog (fresh agent, or one quarantined earlier) still owes + // the tombstones its sidecar holds. + if (isNotFoundError(error)) { + return this.withRecoveredTombstones(agentRef, emptyIndex()) + } throw error } try { - return parseChatIndex(raw) + const { index, droppedTombstoneEntries } = parseChatIndex(raw) + if (droppedTombstoneEntries === 0) return index + const warnKey = `dropped:${agentRef}` + if (!this.warnedTombstoneAgents.has(warnKey)) { + this.warnedTombstoneAgents.add(warnKey) + console.warn( + `[chatStore] Dropped ${droppedTombstoneEntries} malformed tombstone entr${droppedTombstoneEntries === 1 ? 'y' : 'ies'} from the chat index of "${agentRef}"` + ) + } + return this.withRecoveredTombstones(agentRef, index) } catch (error) { if (error instanceof UnsupportedChatIndexVersionError) throw error // Preserve unreadable catalog bytes for bounded recovery/diagnostics, but // detach them from the writable path so a torn index cannot brick every // chat mutation. Paged transcripts live under chats/ and remain untouched; // the catalog can be repopulated by the server as normal writes resume. + // The tombstone sidecar survives the quarantine, so confirmed local + // deletions keep holding on the rebuilt catalog. const source = this.indexPath(agentRef) const target = this.corruptIndexPath(agentRef) await fs.mkdir(dirname(target), { recursive: true, mode: 0o700 }) @@ -1470,7 +1698,7 @@ export class ChatStore { }) } } - return emptyIndex() + return this.withRecoveredTombstones(agentRef, emptyIndex()) } } @@ -1486,9 +1714,33 @@ export class ChatStore { // Flush the temp file before rename and the containing directory afterward. // Visibility atomicity alone does not make the catalog durable across power loss. const target = this.indexPath(agentRef) + await this.saveTombstoneSidecar(agentRef, normalized) await this.writeJsonAtomic(target, normalized, 2) } + /** + * Mirrors the tombstone lists into the sidecar (atomically) whenever they + * change. Skipped for an agent that never had a tombstone. It is written + * before the index, so a rejected save has committed no catalog change. A + * crash between the two writes leaves the sidecar ahead of the index; an + * index read without dropped tombstone entries is authoritative and ignores + * it, and the next save realigns the sidecar with the index. + */ + private async saveTombstoneSidecar(agentRef: string, index: ChatIndex): Promise { + const sidecar: TombstoneSidecar = { + version: TOMBSTONE_SIDECAR_VERSION, + deletedChatTombstones: index.deletedChatTombstones ?? [], + pendingChatCleanup: index.pendingChatCleanup ?? [], + } + const serialized = JSON.stringify(sidecar) + const known = this.tombstoneSidecarSerialized.get(agentRef) + const nothingToProtect = + sidecar.deletedChatTombstones.length === 0 && sidecar.pendingChatCleanup.length === 0 + if (known === serialized || (known === undefined && nothingToProtect)) return + await this.writeJsonAtomic(this.tombstoneSidecarPath(agentRef), sidecar) + this.tombstoneSidecarSerialized.set(agentRef, serialized) + } + async listChats(agentRef: string): Promise { const index = await this.getIndex(agentRef) return index.chats @@ -1502,6 +1754,7 @@ export class ChatStore { async setLastActiveChatId(agentRef: string, chatId: string): Promise { return this.serializeIndex(agentRef, async () => { const index = await this.getIndex(agentRef) + if (this.isDeletedInScope(index, chatId)) return index.lastActiveChatId = chatId await this.saveIndex(agentRef, index) }) @@ -1518,6 +1771,9 @@ export class ChatStore { async createChat(agentRef: string, chatId: string): Promise { return this.serializeIndex(agentRef, async () => { const index = await this.getIndex(agentRef) + if (this.isDeletedInScope(index, chatId)) { + throw new Error('Chat was deleted locally') + } const existing = index.chats.find(c => c.id === chatId) if (existing) return existing @@ -1546,35 +1802,226 @@ export class ChatStore { }) } - async deleteChat(agentRef: string, chatId: string): Promise { - return this.serializeChat(agentRef, chatId, async () => { + async deleteChat( + agentRef: string, + chatId: string, + authorityScope: ChatAuthorityScope + ): Promise { + const scope = this.requireAuthorityScope(authorityScope) + const tombstone: ChatDeleteTombstone = { chatId, authorityScope: { ...scope } } + await this.serializeChat(agentRef, chatId, async () => { + this.requireAuthorityScope(scope) await this.serializeIndex(agentRef, async () => { + this.requireAuthorityScope(scope) const index = await this.getIndex(agentRef) index.chats = index.chats.filter(c => c.id !== chatId) - if (index.lastActiveChatId === chatId) { - index.lastActiveChatId = null + index.deletedChatTombstones = index.deletedChatTombstones ?? [] + if ( + !index.deletedChatTombstones.some( + existing => + existing.chatId === chatId && sameChatAuthorityScope(existing.authorityScope, scope) + ) + ) { + index.deletedChatTombstones.push(tombstone) + } + index.pendingChatCleanup = index.pendingChatCleanup ?? [] + if ( + !index.pendingChatCleanup.some( + existing => + existing.chatId === chatId && sameChatAuthorityScope(existing.authorityScope, scope) + ) + ) { + index.pendingChatCleanup.push(tombstone) } + if (index.lastActiveChatId === chatId) index.lastActiveChatId = null await this.saveIndex(agentRef, index) }) - await fs.rm(this.chatFilePath(agentRef, chatId), { force: true }) - const corruptLegacyDir = join(this.agentDir(agentRef), '.corrupt') - await fs.rm(join(corruptLegacyDir, encodedChatId(chatId)), { recursive: true, force: true }) - const corruptLegacyEntries = await fs.readdir(corruptLegacyDir).catch(() => []) - await Promise.all( - corruptLegacyEntries - .filter(name => legacyCorruptFileBelongsToChat(name, chatId)) - .map(name => fs.rm(join(corruptLegacyDir, name), { force: true })) - ) - await fs.rm(`${this.chatFilePath(agentRef, chatId)}.tmp`, { force: true }) - await fs.rm(this.chatCompatibilitySignaturePath(agentRef, chatId), { force: true }) - await fs.rm(`${this.chatCompatibilitySignaturePath(agentRef, chatId)}.tmp`, { force: true }) - await fs.rm(this.chatDirPath(agentRef, chatId), { recursive: true, force: true }) - await this.removePagedChatSnapshotSiblings(agentRef, chatId) const cacheKey = this.chatSnapshotCleanupKey(agentRef, chatId) this.cleanedSnapshotSiblingKeys.delete(cacheKey) this.compatibilityCheckedKeys.delete(cacheKey) this.dropCompatibilityWindow(cacheKey) }) + + const cleaned = await this.finishPendingDeleteCleanup(agentRef, tombstone) + return { cleanupPending: !cleaned } + } + + /** + * A deleted chat's cleanup is retried on every catalog read, so a permanent + * failure (EACCES on the chat dir) would repeat forever. Report it once per + * (agent, chat, error code) instead of on every retry, and never hide it. + */ + private warnCleanupFailureOnce(agentRef: string, chatId: string | null, error: unknown): void { + const code = (error as NodeJS.ErrnoException | undefined)?.code ?? 'UNKNOWN' + const key = JSON.stringify([agentRef, chatId, code]) + if (this.warnedCleanupFailures.has(key)) return + this.warnedCleanupFailures.add(key) + console.warn( + `[chatStore] Deleted-chat cleanup failed for agent "${agentRef}"${chatId ? ` chat "${chatId}"` : ''} (${code}); it stays queued and is retried`, + error + ) + } + + private async removeChatArtifacts( + agentRef: string, + chatId: string, + authorityScope: ChatAuthorityScope + ): Promise { + const corruptLegacyDir = join(this.agentDir(agentRef), '.corrupt') + let corruptLegacyEntries: string[] + try { + corruptLegacyEntries = await fs.readdir(corruptLegacyDir) + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') corruptLegacyEntries = [] + else { + this.warnCleanupFailureOnce(agentRef, chatId, error) + return false + } + } + if (!this.authorityScope || !sameChatDeletionIdentity(this.authorityScope, authorityScope)) { + return false + } + const paths = [ + { path: this.chatFilePath(agentRef, chatId), recursive: false }, + { path: join(corruptLegacyDir, encodedChatId(chatId)), recursive: true }, + ...corruptLegacyEntries + .filter(name => legacyCorruptFileBelongsToChat(name, chatId)) + .map(name => ({ path: join(corruptLegacyDir, name), recursive: false })), + { path: `${this.chatFilePath(agentRef, chatId)}.tmp`, recursive: false }, + { path: this.chatCompatibilitySignaturePath(agentRef, chatId), recursive: false }, + { path: `${this.chatCompatibilitySignaturePath(agentRef, chatId)}.tmp`, recursive: false }, + { path: this.chatDirPath(agentRef, chatId), recursive: true }, + ] + let allRemoved = true + for (const target of paths) { + if (!this.authorityScope || !sameChatDeletionIdentity(this.authorityScope, authorityScope)) { + return false + } + try { + await fs.rm(target.path, { recursive: target.recursive, force: true }) + } catch (error) { + this.warnCleanupFailureOnce(agentRef, chatId, error) + allRemoved = false + } + } + if (!this.authorityScope || !sameChatDeletionIdentity(this.authorityScope, authorityScope)) { + return false + } + try { + await this.removePagedChatSnapshotSiblings(agentRef, chatId) + } catch (error) { + this.warnCleanupFailureOnce(agentRef, chatId, error) + allRemoved = false + } + return ( + allRemoved && + Boolean(this.authorityScope && sameChatDeletionIdentity(this.authorityScope, authorityScope)) + ) + } + + private async finishPendingDeleteCleanup( + agentRef: string, + tombstone: ChatDeleteTombstone + ): Promise { + if ( + !this.authorityScope || + !sameChatDeletionIdentity(this.authorityScope, tombstone.authorityScope) + ) { + return false + } + let artifactsRemoved = false + try { + artifactsRemoved = await this.removeChatArtifacts( + agentRef, + tombstone.chatId, + tombstone.authorityScope + ) + } catch (error) { + this.warnCleanupFailureOnce(agentRef, tombstone.chatId, error) + return false + } + if ( + !artifactsRemoved || + !this.authorityScope || + !sameChatDeletionIdentity(this.authorityScope, tombstone.authorityScope) + ) { + return false + } + try { + await this.serializeIndex(agentRef, async () => { + this.requireDeletionIdentity(tombstone.authorityScope) + const index = await this.getIndex(agentRef) + index.pendingChatCleanup = (index.pendingChatCleanup ?? []).filter( + existing => + existing.chatId !== tombstone.chatId || + !sameChatDeletionIdentity(existing.authorityScope, tombstone.authorityScope) + ) + await this.saveIndex(agentRef, index) + }) + return true + } catch (error) { + this.warnCleanupFailureOnce(agentRef, tombstone.chatId, error) + return false + } + } + + /** + * Retries the calling identity's queued artifact cleanups. With `agentRef` + * only that agent's queue is retried, except on the first retry after a bind + * (`setAuthorityScope`), which walks every agent once so the queues of agents + * that are not read again still finish. Without `agentRef` every agent is + * walked. + */ + async retryPendingDeleteCleanups( + authorityScope: ChatAuthorityScope, + agentRef?: string + ): Promise { + const scope = this.requireDeletionIdentity(authorityScope) + if (agentRef !== undefined && !this.pendingCleanupSweep) { + await this.retryAgentPendingDeleteCleanups(agentRef, scope) + return + } + this.pendingCleanupSweep = false + let entries + try { + entries = await fs.readdir(this.baseDir, { withFileTypes: true }) + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return + // The walk did not happen: the next retry attempts it again. + this.pendingCleanupSweep = true + throw error + } + for (const entry of entries) { + if (!entry.isDirectory()) continue + if (!(await this.retryAgentPendingDeleteCleanups(entry.name, scope))) return + } + } + + /** Retries one agent's queue; false once the identity changed and retrying must stop. */ + private async retryAgentPendingDeleteCleanups( + agentRef: string, + scope: ChatAuthorityScope + ): Promise { + try { + const index = await this.getIndex(agentRef) + const pending = (index.pendingChatCleanup ?? []).filter(item => + sameChatDeletionIdentity(item.authorityScope, scope) + ) + for (const tombstone of pending) { + if (!this.authorityScope || !sameChatDeletionIdentity(this.authorityScope, scope)) { + return false + } + await this.serializeChat(agentRef, tombstone.chatId, () => + this.finishPendingDeleteCleanup(agentRef, tombstone) + ) + } + } catch (error) { + // An unrelated or unreadable agent index must not prevent other durable + // cleanup entries from being retried on the next authorized read, but the + // failure is reported (once per agent and code), never swallowed. + this.warnCleanupFailureOnce(agentRef, null, error) + } + return true } async loadMessages( @@ -1585,6 +2032,8 @@ export class ChatStore { ): Promise { try { return await this.serializeChat(agentRef, chatId, async () => { + const index = await this.getIndex(agentRef) + if (this.isDeletedInScope(index, chatId)) return [] const meta = await this.readOrMigratePagedChatUnlocked(agentRef, chatId) if (!meta) return [] return this.readMessagesFromPagedMeta(agentRef, chatId, meta, limit, offset) @@ -1639,7 +2088,8 @@ export class ChatStore { options: { preserveExistingTotals?: boolean } = {} ): Promise { return this.serializeChat(agentRef, chatId, async () => { - await this.getIndex(agentRef) + const index = await this.getIndex(agentRef) + if (this.isDeletedInScope(index, chatId)) return const indexedCount = options.preserveExistingTotals ? await this.indexedMessageCount(agentRef, chatId) : undefined @@ -1672,7 +2122,8 @@ export class ChatStore { options: ReplaceChatMessagesOptions = {} ): Promise { await this.serializeChat(agentRef, chatId, async () => { - await this.getIndex(agentRef) + const index = await this.getIndex(agentRef) + if (this.isDeletedInScope(index, chatId)) return const existingMeta = await this.readOrMigratePagedChatUnlocked(agentRef, chatId) const existingMessages = existingMeta ? await this.readMessagesFromPagedMeta(agentRef, chatId, existingMeta) @@ -1713,7 +2164,8 @@ export class ChatStore { newMessages: ChatMessage[] ): Promise { return this.serializeChat(agentRef, chatId, async () => { - await this.getIndex(agentRef) + const indexBeforeAppend = await this.getIndex(agentRef) + if (this.isDeletedInScope(indexBeforeAppend, chatId)) return const existing = (await this.readOrMigratePagedChatUnlocked(agentRef, chatId)) ?? (await this.writePagedChatUnlocked(agentRef, chatId, [])) @@ -1734,6 +2186,31 @@ export class ChatStore { }) } + /** + * Insert or update messages by reconciled identity before a send's task id is + * known. A send is first persisted optimistically, then updated in place after + * its acknowledgement; append-only writes would duplicate the same user row. + */ + async upsertMessages(agentRef: string, chatId: string, messages: ChatMessage[]): Promise { + if (!messages.length) return + return this.serializeChat(agentRef, chatId, async () => { + const index = await this.getIndex(agentRef) + if (this.isDeletedInScope(index, chatId)) return + const existingMeta = + (await this.readOrMigratePagedChatUnlocked(agentRef, chatId)) ?? + (await this.writePagedChatUnlocked(agentRef, chatId, [])) + const existingMessages = await this.readMessagesFromPagedMeta(agentRef, chatId, existingMeta) + const merged = mergeReconciledMessages(existingMessages, messages) + const indexedCount = await this.indexedMessageCount(agentRef, chatId) + await this.writePagedChatUnlocked(agentRef, chatId, merged, { + messageCount: Math.max(merged.length, indexedCount ?? 0), + }) + await this.updateChatIndexFromMessages(agentRef, chatId, merged, { + preserveExistingTotals: true, + }) + }) + } + /** * Flag that a task terminated while the chat was not the active view (D.5). * Idempotent: a no-op when already flagged, so a noisy task can't spam FS writes. diff --git a/desktop-app/src/chatStoreBinding.ts b/desktop-app/src/chatStoreBinding.ts index 0a7b61d26b..5e9ca2764c 100644 --- a/desktop-app/src/chatStoreBinding.ts +++ b/desktop-app/src/chatStoreBinding.ts @@ -2,7 +2,9 @@ import { app } from 'electron' import { promises as fs } from 'node:fs' import { join } from 'node:path' import { ChatStore } from './chatStore.js' +import { resolveDevIsolation } from './devIsolation.js' import { assertSafeFilesystemSegment } from './pathSafety.js' +import type { ChatAuthorityScope } from './types.js' /** * Keep the shared catalog at v2 so pre-paging desktop builds can read the @@ -13,12 +15,14 @@ const PREVIOUS_PAGED_INDEX_VERSION = 3 const LEGACY_INDEX_VERSIONS = new Set([1]) const CORRUPT_QUARANTINE_RETENTION_MS = 30 * 24 * 60 * 60 * 1000 -// Default base dir derived from Electron's userData path. Tests override via +// Default base dir derived from the user's home. Tests override via // __setChatStoreBaseDirForTests to point at a tmpdir, which means the // app.getPath call below is only reached in production. let baseDirOverride: string | null = null function currentBaseDir(): string { if (baseDirOverride !== null) return baseDirOverride + const isolation = resolveDevIsolation(process.env, process.argv, app.isPackaged) + if (isolation.mode === 'isolated') return join(isolation.plan.runDir, 'chats') return join(app.getPath('home'), '.clerum', 'chats') } @@ -49,7 +53,7 @@ const PRE_ENV_MIGRATION_MARKER = '.env-scoped' export async function bindChatStoreForUser( userId: string, envKey: string, - options: { legacyEnvKeys?: readonly string[] } = {} + options: { legacyEnvKeys?: readonly string[]; teamId?: string | null } = {} ): Promise { assertSafeFilesystemSegment('userId', userId) assertSafeFilesystemSegment('envKey', envKey) @@ -60,9 +64,23 @@ export async function bindChatStoreForUser( // refreshes re-call this with an unchanged `me.id`; tearing the store down // just to rebuild it opens a window where every concurrent chat IPC fails // with "Not authenticated" (seen as an empty "Latest sessions" at boot). - if (activeChatStore && activeUserId === userId && activeEnvKey === envKey) return + const authorityScope: ChatAuthorityScope = { + environmentKey: envKey, + userId, + teamId: String(options.teamId || '').trim() || null, + } + if (activeChatStore && activeUserId === userId && activeEnvKey === envKey) { + activeChatStore.setAuthorityScope(authorityScope) + return + } const bindKey = `${envKey}::${userId}` - if (bindInFlight?.key === bindKey) return bindInFlight.promise + if (bindInFlight?.key === bindKey) { + await bindInFlight.promise + if (activeChatStore && activeUserId === userId && activeEnvKey === envKey) { + activeChatStore.setAuthorityScope(authorityScope) + } + return + } const generation = ++bindingGeneration const promise = (async () => { @@ -84,6 +102,7 @@ export async function bindChatStoreForUser( await sweepExpiredCorruptQuarantines(userDir) if (generation !== bindingGeneration) return activeChatStore = new ChatStore(userDir) + activeChatStore.setAuthorityScope(authorityScope) activeUserId = userId activeEnvKey = envKey })() @@ -343,6 +362,19 @@ export function requireChatStore(): ChatStore { return activeChatStore } +/** Fences destructive IPC work to the store binding that issued the request. */ +export function getChatStoreBindingGeneration(): number { + requireChatStore() + return bindingGeneration +} + +export function requireChatStoreForBindingGeneration(expectedGeneration: number): ChatStore { + if (!Number.isSafeInteger(expectedGeneration) || expectedGeneration !== bindingGeneration) { + throw new Error('Chat store binding changed') + } + return requireChatStore() +} + /** Test-only helper. Do not call from production code. */ export function __setChatStoreBaseDirForTests(base: string | null): void { baseDirOverride = base diff --git a/desktop-app/src/httpClient.ts b/desktop-app/src/httpClient.ts index f90eef145a..a7f637bbe3 100644 --- a/desktop-app/src/httpClient.ts +++ b/desktop-app/src/httpClient.ts @@ -1,4 +1,5 @@ import { config } from './config.js' +import { boundedErrorExcerpt, hostAccessDenialMessage } from './upstreamErrors.js' export class ApiError extends Error { status: number @@ -141,6 +142,11 @@ export async function requestJson( const raw = await response.text() if (!response.ok) { + const retryAfter = response.headers.get('retry-after') + const hostAccessMessage = hostAccessDenialMessage(response.status, raw) + if (hostAccessMessage) { + throw new ApiError(hostAccessMessage, response.status, raw, retryAfter) + } let msg = raw || response.statusText try { const parsed = JSON.parse(raw) as { error?: unknown; message?: unknown } @@ -149,11 +155,12 @@ export async function requestJson( } catch { // Keep raw text as message. } + // The full body stays on `bodyText`; the message carries a bounded excerpt. throw new ApiError( - `${response.status} ${response.statusText}: ${msg}`, + `${response.status} ${response.statusText}: ${boundedErrorExcerpt(msg)}`, response.status, raw, - response.headers.get('retry-after') + retryAfter ) } diff --git a/desktop-app/src/ipc.ts b/desktop-app/src/ipc.ts index e5540b7c36..436abdaa3a 100644 --- a/desktop-app/src/ipc.ts +++ b/desktop-app/src/ipc.ts @@ -12,7 +12,11 @@ import { promises as fs } from 'node:fs' import path from 'node:path' import { parseFileReferenceV1 } from '@clerum/gfs-interaction-policy' import { AppService } from './appService.js' -import { requireChatStore } from './chatStoreBinding.js' +import { + getChatStoreBindingGeneration, + requireChatStore, + requireChatStoreForBindingGeneration, +} from './chatStoreBinding.js' import { GFS_PREVIEW_MAX_BYTES } from './gfs/previewLimits.js' import { assertSafeRouteSegment } from './pathSafety.js' import { @@ -23,6 +27,8 @@ import { PLUGIN_SDK_REQUEST_CHANNEL, } from './pluginSdkProtocol.js' import { + type ChatAuthorityScope, + type ChatDeleteFence, DesktopRuntimeConfig, HostActivityStreamEvent, HostMessageRequest, @@ -35,6 +41,54 @@ function sanitizeString(input: unknown): string { return String(input || '').trim() } +function sanitizeChatAuthorityScope(input: unknown): ChatAuthorityScope { + if (!input || typeof input !== 'object' || Array.isArray(input)) { + throw new Error('Invalid chat deletion authority scope') + } + const value = input as Record + const environmentKey = sanitizeString(value.environmentKey) + const userId = sanitizeString(value.userId) + const teamId = value.teamId === null ? null : sanitizeString(value.teamId) + if (!environmentKey || !userId || (value.teamId !== null && !teamId)) { + throw new Error('Invalid chat deletion authority scope') + } + return { environmentKey, userId, teamId } +} + +function sameChatAuthorityScope(left: ChatAuthorityScope, right: ChatAuthorityScope): boolean { + return ( + left.environmentKey === right.environmentKey && + left.userId === right.userId && + left.teamId === right.teamId + ) +} + +function sanitizeChatDeleteFence(input: unknown): ChatDeleteFence { + if (!input || typeof input !== 'object' || Array.isArray(input)) { + throw new Error('Scoped chat deletion is required') + } + const value = input as Record + const bindingGeneration = value.bindingGeneration + const sessionGeneration = value.sessionGeneration + if ( + value.version !== 1 || + typeof bindingGeneration !== 'number' || + !Number.isSafeInteger(bindingGeneration) || + bindingGeneration < 1 || + typeof sessionGeneration !== 'number' || + !Number.isSafeInteger(sessionGeneration) || + sessionGeneration < 0 + ) { + throw new Error('Invalid chat deletion fence') + } + return { + version: 1, + authorityScope: sanitizeChatAuthorityScope(value.authorityScope), + bindingGeneration, + sessionGeneration, + } +} + /** * Sanitize a bulk subject-key array (grant/share to many subjects in one call). * Fails loud on a non-array or empty payload rather than silently no-op'ing — an @@ -1426,25 +1480,85 @@ export function registerIpcHandlers(service: AppService): void { ipcMain.handle( 'chat:rename', - async (event, payload: { agentRef: string; chatId: string; title: string }) => { + async ( + event, + payload: { + agentRef: string + chatId: string + title: string + bindingGeneration: number + } + ) => { assertTrustedSender(event) const agentRef = sanitizeString(payload?.agentRef) const chatId = sanitizeString(payload?.chatId) const title = sanitizeString(payload?.title) if (!agentRef || !chatId || !title) throw new Error('agentRef, chatId, and title are required') - await requireChatStore().renameChat(agentRef, chatId, title) + const bindingGeneration = payload?.bindingGeneration + if (typeof bindingGeneration !== 'number' || !Number.isSafeInteger(bindingGeneration)) { + throw new Error('A valid chat store binding generation is required') + } + await requireChatStoreForBindingGeneration(bindingGeneration).renameChat( + agentRef, + chatId, + title + ) } ) - ipcMain.handle('chat:delete', async (event, payload: { agentRef: string; chatId: string }) => { + ipcMain.handle('chat:bindingGeneration', event => { assertTrustedSender(event) - const agentRef = sanitizeString(payload?.agentRef) - const chatId = sanitizeString(payload?.chatId) - if (!agentRef || !chatId) throw new Error('agentRef and chatId are required') - await requireChatStore().deleteChat(agentRef, chatId) + return getChatStoreBindingGeneration() }) + ipcMain.handle( + 'chat:captureDeleteFence', + (event, payload: { expectedAuthorityScope?: unknown }) => { + assertTrustedSender(event) + const expectedAuthorityScope = sanitizeChatAuthorityScope(payload?.expectedAuthorityScope) + const authority = service.getChatDeletionFenceAuthority() + if (!sameChatAuthorityScope(expectedAuthorityScope, authority.authorityScope)) { + throw new Error('Chat deletion authority scope changed before confirmation') + } + return { + version: 1, + authorityScope: authority.authorityScope, + bindingGeneration: getChatStoreBindingGeneration(), + sessionGeneration: authority.sessionGeneration, + } satisfies ChatDeleteFence + } + ) + + ipcMain.handle( + 'chat:delete', + async ( + event, + payload: { + version?: number + fence?: unknown + agentRef: string + chatId: string + } + ) => { + assertTrustedSender(event) + const agentRef = sanitizeString(payload?.agentRef) + const chatId = sanitizeString(payload?.chatId) + if (!agentRef || !chatId) throw new Error('agentRef and chatId are required') + if (payload?.version !== 3) throw new Error('Scoped chat deletion is required') + const fence = sanitizeChatDeleteFence(payload.fence) + const authority = service.getChatDeletionFenceAuthority() + if ( + !sameChatAuthorityScope(fence.authorityScope, authority.authorityScope) || + fence.sessionGeneration !== authority.sessionGeneration + ) { + throw new Error('Chat deletion authority changed before confirmation') + } + const store = requireChatStoreForBindingGeneration(fence.bindingGeneration) + return await store.deleteChat(agentRef, chatId, fence.authorityScope) + } + ) + ipcMain.handle( 'chat:loadMessages', async ( @@ -1476,6 +1590,22 @@ export function registerIpcHandlers(service: AppService): void { } ) + ipcMain.handle( + 'chat:upsertMessages', + async (event, payload: { agentRef: string; chatId: string; messages: unknown[] }) => { + assertTrustedSender(event) + const agentRef = sanitizeString(payload?.agentRef) + const chatId = sanitizeString(payload?.chatId) + if (!agentRef || !chatId) throw new Error('agentRef and chatId are required') + const messages = Array.isArray(payload?.messages) ? payload.messages : [] + await requireChatStore().upsertMessages( + agentRef, + chatId, + messages as import('./types.js').ChatMessage[] + ) + } + ) + ipcMain.handle( 'chat:replaceMessages', async ( @@ -1593,7 +1723,25 @@ export function registerIpcHandlers(service: AppService): void { assertTrustedSender(event) const agentRef = sanitizeString(payload?.agentRef) if (!agentRef) throw new Error('agentRef is required') - return requireChatStore().getIndex(agentRef) + const store = requireChatStore() + // The catalog read must never depend on the deleted-chat artifact cleanup: + // a cleanup failure (scope changed mid-flight, no session, an unreadable + // chats directory) would otherwise reject the read and leave the renderer + // without the tombstones it filters deleted chats with. The failure is + // reported with its code and the cleanup stays queued for the next read. + // Only this agent's queue is retried (the renderer reads every agent in a + // fan-out); the store walks every agent once per bind on its own. + try { + const authority = service.getChatDeletionFenceAuthority() + await store.retryPendingDeleteCleanups(authority.authorityScope, agentRef) + } catch (error) { + const code = (error as NodeJS.ErrnoException | undefined)?.code + console.warn( + `[chat:getIndex] Deleted-chat cleanup retry failed for agent "${agentRef}"${code ? ` (${code})` : ''}; returning the index with its tombstones`, + error + ) + } + return store.getIndex(agentRef) }) ipcMain.handle('chat:dismissOnboarding', async (event, payload: { agentRef: string }) => { diff --git a/desktop-app/src/preload.ts b/desktop-app/src/preload.ts index 79a047e4fe..d9ce05167d 100644 --- a/desktop-app/src/preload.ts +++ b/desktop-app/src/preload.ts @@ -578,14 +578,19 @@ const clerum = Object.freeze({ list: (agentRef: string) => ipcRenderer.invoke('chat:list', { agentRef }), create: (agentRef: string, chatId: string) => ipcRenderer.invoke('chat:create', { agentRef, chatId }), - rename: (agentRef: string, chatId: string, title: string) => - ipcRenderer.invoke('chat:rename', { agentRef, chatId, title }), - delete: (agentRef: string, chatId: string) => - ipcRenderer.invoke('chat:delete', { agentRef, chatId }), + rename: (agentRef: string, chatId: string, title: string, bindingGeneration: number) => + ipcRenderer.invoke('chat:rename', { agentRef, chatId, title, bindingGeneration }), + getBindingGeneration: () => ipcRenderer.invoke('chat:bindingGeneration'), + captureDeleteFence: (expectedAuthorityScope: import('./types.js').ChatAuthorityScope) => + ipcRenderer.invoke('chat:captureDeleteFence', { expectedAuthorityScope }), + delete: (agentRef: string, chatId: string, fence: import('./types.js').ChatDeleteFence) => + ipcRenderer.invoke('chat:delete', { version: 3, agentRef, chatId, fence }), loadMessages: (agentRef: string, chatId: string, limit?: number, offset?: number) => ipcRenderer.invoke('chat:loadMessages', { agentRef, chatId, limit, offset }), appendMessages: (agentRef: string, chatId: string, messages: unknown[]) => ipcRenderer.invoke('chat:appendMessages', { agentRef, chatId, messages }), + upsertMessages: (agentRef: string, chatId: string, messages: unknown[]) => + ipcRenderer.invoke('chat:upsertMessages', { agentRef, chatId, messages }), replaceMessages: ( agentRef: string, chatId: string, diff --git a/desktop-app/src/renderer.d.ts b/desktop-app/src/renderer.d.ts index 1b9ba4585f..23e5ecaffb 100644 --- a/desktop-app/src/renderer.d.ts +++ b/desktop-app/src/renderer.d.ts @@ -8,6 +8,9 @@ import { AccessCatalog, AgentWithMcpServers, ApprovalDecisionResult, + ChatAuthorityScope, + ChatDeleteFence, + ChatDeleteResult, ChatIndex, ChatMessage, ChatMetadata, @@ -695,8 +698,19 @@ declare global { chat: { list: (agentRef: string) => Promise create: (agentRef: string, chatId: string) => Promise - rename: (agentRef: string, chatId: string, title: string) => Promise - delete: (agentRef: string, chatId: string) => Promise + rename: ( + agentRef: string, + chatId: string, + title: string, + bindingGeneration: number + ) => Promise + getBindingGeneration: () => Promise + captureDeleteFence: (expectedAuthorityScope: ChatAuthorityScope) => Promise + delete: ( + agentRef: string, + chatId: string, + fence: ChatDeleteFence + ) => Promise loadMessages: ( agentRef: string, chatId: string, @@ -704,6 +718,7 @@ declare global { offset?: number ) => Promise appendMessages: (agentRef: string, chatId: string, messages: ChatMessage[]) => Promise + upsertMessages: (agentRef: string, chatId: string, messages: ChatMessage[]) => Promise replaceMessages: ( agentRef: string, chatId: string, diff --git a/desktop-app/src/rpcProxyClient.ts b/desktop-app/src/rpcProxyClient.ts index 3f7a45e948..45c60dba09 100644 --- a/desktop-app/src/rpcProxyClient.ts +++ b/desktop-app/src/rpcProxyClient.ts @@ -23,6 +23,7 @@ import { SessionsListResult, SetHostModelResult, } from './types.js' +import { boundedErrorExcerpt, hostAccessDenialMessage } from './upstreamErrors.js' export type { SandboxUiApp } from './types.js' @@ -167,6 +168,12 @@ function parseToolSteps(value: unknown, label: string): MessageToolStep[] { }) } +/** + * Longest sessions cursor the `rpc:listSessions` IPC handler accepts + * (`sanitizeSessionsListQuery` in `ipc.ts`; mcp-host enforces the same bound). + */ +const SESSIONS_CURSOR_MAX_LENGTH = 2048 + // Exported for tests: renderer/parser tests derive their fixtures from the real // producer (pr-discipline T1) instead of hand-mocking the parsed shape. export function parseSessionsListResult(value: unknown): SessionsListResult { @@ -249,12 +256,24 @@ export function parseSessionsListResult(value: unknown): SessionsListResult { } }) if (record.items.length > 0 && items.length === 0 && firstItemError) throw firstItemError + const nextCursor = + record.nextCursor != null + ? wireString(record.nextCursor, 'sessions response.nextCursor') + : undefined + // R3-L10: the `rpc:listSessions` IPC handler refuses a longer cursor before + // any request, so handing one to the renderer would only produce a status-less + // error on "Load more". End the page chain here instead. The cursor is opaque + // upstream state: log its length only, never its value. + const cursorTooLong = nextCursor !== undefined && nextCursor.length > SESSIONS_CURSOR_MAX_LENGTH + if (cursorTooLong) { + console.warn( + `[RpcProxyClient] Dropped a sessions nextCursor of ${nextCursor.length} characters (limit ${SESSIONS_CURSOR_MAX_LENGTH})` + ) + } return { items, ...(droppedItemCount > 0 ? { droppedItemCount } : {}), - ...(record.nextCursor != null - ? { nextCursor: wireString(record.nextCursor, 'sessions response.nextCursor') } - : {}), + ...(nextCursor !== undefined && !cursorTooLong ? { nextCursor } : {}), } } @@ -425,7 +444,9 @@ export class SandboxUiSessionError extends Error { readonly status: number readonly body: string constructor(status: number, body: string) { - super(`sandbox-ui session mint failed (${status}): ${body || ''}`) + super( + `sandbox-ui session mint failed (${status}): ${boundedErrorExcerpt(body) || ''}` + ) this.status = status this.body = body this.name = 'SandboxUiSessionError' @@ -456,6 +477,16 @@ function errorCodeFromBody(body: string): string | null { } } +/** + * The single producer of Host-access denial errors for the raw-`fetch` methods: + * the message strings and the `code` rule live in `upstreamErrors.ts` and are + * shared with `httpClient.requestJson`. + */ +function hostAccessDenialError(status: number, body: string): ApiError | null { + const message = hostAccessDenialMessage(status, body) + return message ? new ApiError(message, status, body) : null +} + export class RpcProxyClient { async health(): Promise<{ status: string }> { return requestJson<{ status: string }>('GET', url('/health')) @@ -574,7 +605,11 @@ export class RpcProxyClient { return { status } } const body = await readErrorBody(response) - throw new ApiError(`Prewarm failed (${response.status}): ${body}`, response.status, body) + throw new ApiError( + `Prewarm failed (${response.status}): ${boundedErrorExcerpt(body)}`, + response.status, + body + ) } async getHostHealth(rpcAccessToken: string, hostRef: string): Promise { @@ -685,7 +720,7 @@ export class RpcProxyClient { if (!response.ok) { const body = await response.text() throw new Error( - `sandbox-ui authorize-url request failed (${response.status}): ${body || ''}` + `sandbox-ui authorize-url request failed (${response.status}): ${boundedErrorExcerpt(body) || ''}` ) } const json = (await response.json()) as { authorizeUrl?: unknown } @@ -732,7 +767,7 @@ export class RpcProxyClient { if (!response.ok) { const body = await response.text() throw new ApiError( - `mcp-oauth authorize-url request failed (${response.status}): ${body || ''}`, + `mcp-oauth authorize-url request failed (${response.status}): ${boundedErrorExcerpt(body) || ''}`, response.status, body ) @@ -765,7 +800,9 @@ export class RpcProxyClient { }) if (!response.ok) { const body = await response.text() - throw new Error(`Host stream failed (${response.status}): ${body || response.statusText}`) + throw new Error( + `Host stream failed (${response.status}): ${boundedErrorExcerpt(body) || response.statusText}` + ) } if (!response.body) { throw new Error('Host stream missing response body') @@ -887,7 +924,11 @@ export class RpcProxyClient { const body = await response.text() // ApiError (not a bare Error) so `AppService.shouldRefreshRpcToken` can see // the 401/403 status and drive the retry-after-refresh (§4.5-7). - throw new ApiError(`Approve failed (${response.status}): ${body}`, response.status, body) + throw new ApiError( + `Approve failed (${response.status}): ${boundedErrorExcerpt(body)}`, + response.status, + body + ) } return parseApprovalDecisionResponse(response) } @@ -913,7 +954,11 @@ export class RpcProxyClient { ) if (!response.ok) { const body = await response.text() - throw new ApiError(`Deny failed (${response.status}): ${body}`, response.status, body) + throw new ApiError( + `Deny failed (${response.status}): ${boundedErrorExcerpt(body)}`, + response.status, + body + ) } return parseApprovalDecisionResponse(response) } @@ -934,7 +979,11 @@ export class RpcProxyClient { ) if (!response.ok) { const body = await response.text() - throw new ApiError(`cancelTask failed (${response.status}): ${body}`, response.status, body) + throw new ApiError( + `cancelTask failed (${response.status}): ${boundedErrorExcerpt(body)}`, + response.status, + body + ) } } @@ -954,7 +1003,7 @@ export class RpcProxyClient { if (!response.ok) { const body = await response.text() throw new ApiError( - `List artifacts failed (${response.status}): ${body}`, + `List artifacts failed (${response.status}): ${boundedErrorExcerpt(body)}`, response.status, body ) @@ -975,7 +1024,7 @@ export class RpcProxyClient { ) if (!response.ok) { const body = await response.text() - throw new Error(`Download artifact failed (${response.status}): ${body}`) + throw new Error(`Download artifact failed (${response.status}): ${boundedErrorExcerpt(body)}`) } return Buffer.from(await response.arrayBuffer()) } @@ -996,8 +1045,10 @@ export class RpcProxyClient { }) if (!response.ok) { const body = await response.text() + const hostAccessDenial = hostAccessDenialError(response.status, body) + if (hostAccessDenial) throw hostAccessDenial throw new ApiError( - `List sessions failed (${response.status}): ${body}`, + `List sessions failed (${response.status}): ${boundedErrorExcerpt(body)}`, response.status, body ) @@ -1032,14 +1083,16 @@ export class RpcProxyClient { signal: withTimeout(), }) if (response.status === 404) { - // Keep the '404' token in the message: the renderer's `isHttp404` matches on - // it to evict a stale local chat. + // Preserve the status for the renderer. A transcript 404 is ambiguous + // during Host wake and never confirms that local data may be deleted. throw new ApiError(`Session not found (404)`, 404, '') } if (!response.ok) { const body = await response.text() + const hostAccessDenial = hostAccessDenialError(response.status, body) + if (hostAccessDenial) throw hostAccessDenial throw new ApiError( - `Load session messages failed (${response.status}): ${body}`, + `Load session messages failed (${response.status}): ${boundedErrorExcerpt(body)}`, response.status, body ) @@ -1082,7 +1135,7 @@ export class RpcProxyClient { if (!response.ok) { const body = await response.text() throw new ApiError( - `Get context breakdown failed (${response.status}): ${body}`, + `Get context breakdown failed (${response.status}): ${boundedErrorExcerpt(body)}`, response.status, body ) @@ -1125,7 +1178,7 @@ export class RpcProxyClient { if (!response.ok) { const body = await readErrorBody(response) throw new ApiError( - `Get host models failed (${response.status}): ${body}`, + `Get host models failed (${response.status}): ${boundedErrorExcerpt(body)}`, response.status, body ) @@ -1185,7 +1238,7 @@ export class RpcProxyClient { ) } throw new ApiError( - `Set host model failed (${response.status}): ${body}`, + `Set host model failed (${response.status}): ${boundedErrorExcerpt(body)}`, response.status, body ) @@ -1229,6 +1282,8 @@ export class RpcProxyClient { // Body may echo the invalid title; read it for the ApiError payload but // keep the raw title out of the human-facing message. const body = await readErrorBody(response) + const hostAccessDenial = hostAccessDenialError(response.status, body) + if (hostAccessDenial) throw hostAccessDenial throw new ApiError(`Rename session failed (${response.status})`, response.status, body) } const parsed = (await response.json().catch(() => ({}))) as { title?: unknown } @@ -1272,7 +1327,7 @@ export class RpcProxyClient { if (!res.ok) { const text = await readErrorBody(res) throw new ApiError( - `Desktop session exchange failed: ${res.status} ${text || res.statusText}`, + `Desktop session exchange failed: ${res.status} ${boundedErrorExcerpt(text) || res.statusText}`, res.status, text ) diff --git a/desktop-app/src/sharedFilesClient.ts b/desktop-app/src/sharedFilesClient.ts index 3f574e47ef..c86a806c57 100644 --- a/desktop-app/src/sharedFilesClient.ts +++ b/desktop-app/src/sharedFilesClient.ts @@ -1,5 +1,6 @@ import { config } from './config.js' import { ApiError, requestJson } from './httpClient.js' +import { boundedErrorExcerpt } from './upstreamErrors.js' export type ContextSharedFilesystemSummary = { name: string @@ -96,7 +97,7 @@ export class SharedFilesClient { if (!response.ok) { const body = await response.text() throw new ApiError( - `${response.status} ${response.statusText}: ${body}`, + `${response.status} ${response.statusText}: ${boundedErrorExcerpt(body)}`, response.status, body ) diff --git a/desktop-app/src/types.ts b/desktop-app/src/types.ts index bfcf48cf4c..04679d7478 100644 --- a/desktop-app/src/types.ts +++ b/desktop-app/src/types.ts @@ -772,15 +772,16 @@ export type HostRuntimeHealth = { * 409 `not-stateless`). `skipped: 'cooldown'` means a recent attempt for the * same hostRef suppressed the HTTP call entirely; `skipped: 'in-flight'` * means that attempt's bounded re-emission loop is still running (structural - * single-loop-per-host guarantee). `error` carries the failure - * message for any other outcome — the caller treats prewarm as + * single-loop-per-host guarantee). `skipped: 'auth-changed'` means the + * authenticated owner changed or is changing before a wake can be issued. + * `error` carries the failure message for any other outcome — the caller treats prewarm as * fire-and-forget, so failures surface here (and in main-process logs), never * as a thrown error. */ export type PrewarmHostResult = { requested: boolean status?: string - skipped?: 'cooldown' | 'in-flight' + skipped?: 'cooldown' | 'in-flight' | 'auth-changed' error?: string } @@ -925,12 +926,41 @@ export interface ChatMetadata { lastTerminalAt?: string } +/** The authority identity that owns a local chat deletion. */ +export interface ChatAuthorityScope { + environmentKey: string + userId: string + teamId: string | null +} + +/** Main-issued fence captured when a delete is queued and rechecked on confirm. */ +export interface ChatDeleteFence { + version: 1 + authorityScope: ChatAuthorityScope + bindingGeneration: number + sessionGeneration: number +} + +/** Durable tombstone and cleanup identity for one scoped local deletion. */ +export interface ChatDeleteTombstone { + chatId: string + authorityScope: ChatAuthorityScope +} + export interface ChatIndex { /** v2 remains readable by pre-paging builds; v3 is accepted and normalized for compatibility. */ version: 1 | 2 | 3 lastActiveChatId: string | null onboardingDismissed: boolean chats: ChatMetadata[] + /** New deletions are isolated to their complete authority identity. */ + deletedChatTombstones?: ChatDeleteTombstone[] + /** Durable artifact-cleanup work, retried only in the owning authority scope. */ + pendingChatCleanup?: ChatDeleteTombstone[] +} + +export interface ChatDeleteResult { + cleanupPending: boolean } /** Server-reported session lifecycle (D.1). `idle` once no task is in flight. */ diff --git a/desktop-app/src/upstreamErrors.ts b/desktop-app/src/upstreamErrors.ts new file mode 100644 index 0000000000..e738391a4f --- /dev/null +++ b/desktop-app/src/upstreamErrors.ts @@ -0,0 +1,72 @@ +/** + * Shared vocabulary for the errors Desktop main builds from upstream (rpc-proxy, + * control-api, mcp-host) HTTP responses. + * + * Two concerns live here so every call site agrees on them: + * + * 1. The Host-access denial messages the renderer matches on + * (`ui/src/lib/format.ts isConfirmedHostAccessRevoked`). The strings are + * defined ONCE; `httpClient.requestJson` and the raw-`fetch` methods of + * `rpcProxyClient` all build them through {@link hostAccessDenialMessage}. + * 2. A bounded, single-line excerpt of an upstream body for error messages + * ({@link boundedErrorExcerpt}), so a hostile or misbehaving upstream cannot + * push an unbounded payload through `Error.message` into logs, IPC and UI. + */ + +/** Machine `code` rpc-proxy sets when the user's access to the Host was removed. */ +export const HOST_ACCESS_REVOKED_CODE = 'host_access_revoked' +/** Machine `code` rpc-proxy sets for every other denial it authors itself. */ +export const HOST_ACCESS_DENIED_CODE = 'host_access_denied' + +/** The only message that lets the renderer treat a 403 as confirmed revocation. */ +export const HOST_ACCESS_REVOKED_MESSAGE = `403 Forbidden: ${HOST_ACCESS_REVOKED_CODE}` +/** A 403 rpc-proxy authored that does NOT confirm revocation (renderer: generic 403). */ +export const HOST_ACCESS_DENIED_MESSAGE = `403 Forbidden: ${HOST_ACCESS_DENIED_CODE}` + +/** Upper bound (characters) of the upstream body excerpt embedded in a message. */ +export const ERROR_EXCERPT_MAX_CHARS = 512 + +const RESERVED_HOST_ACCESS_TOKEN = /host_access_(revoked|denied)/gi + +/** + * A short single-line excerpt of an upstream body for an error message. + * + * Whitespace is collapsed, the result is cut at {@link ERROR_EXCERPT_MAX_CHARS} + * characters (an ellipsis marks the cut), and the reserved Host-access tokens are + * defused (`host_access_revoked` -> `host-access-revoked`): only + * {@link hostAccessDenialMessage} may emit them, so a body that merely contains + * the words can never forge a confirmed revocation. + */ +export function boundedErrorExcerpt(text: string): string { + const collapsed = text + .replace(/\s+/g, ' ') + .trim() + .replace(RESERVED_HOST_ACCESS_TOKEN, 'host-access-$1') + return collapsed.length > ERROR_EXCERPT_MAX_CHARS + ? `${collapsed.slice(0, ERROR_EXCERPT_MAX_CHARS)}…` + : collapsed +} + +/** The `code` string of a JSON error body, or null for non-JSON / code-less bodies. */ +function machineCodeFromBody(body: string): string | null { + try { + const parsed = JSON.parse(body) as { code?: unknown } + return typeof parsed?.code === 'string' ? parsed.code : null + } catch { + return null + } +} + +/** + * The fixed Host-access message for a 403 response, or null when the response is + * not one rpc-proxy authored with a Host-access `code`. Decided ONLY from the + * parsed JSON `code` field: an `error` string, a text body or an interposed + * proxy's page never qualifies. + */ +export function hostAccessDenialMessage(status: number, body: string): string | null { + if (status !== 403) return null + const code = machineCodeFromBody(body) + if (code === HOST_ACCESS_REVOKED_CODE) return HOST_ACCESS_REVOKED_MESSAGE + if (code === HOST_ACCESS_DENIED_CODE) return HOST_ACCESS_DENIED_MESSAGE + return null +} diff --git a/desktop-app/test/e2e-playwright/global-setup.ts b/desktop-app/test/e2e-playwright/global-setup.ts index 21cb983476..07479223ad 100644 --- a/desktop-app/test/e2e-playwright/global-setup.ts +++ b/desktop-app/test/e2e-playwright/global-setup.ts @@ -142,8 +142,11 @@ export function validateBaseUrls({ } } -function kubectlCurrentContext(): string { - return execFileSync('kubectl', ['config', 'current-context'], { +function readGlobalKubectlCurrentContext(expectedContext: string): string { + // Deliberately strict: `config current-context` reads the global kubeconfig + // context even when --context selects another context for object commands. + // E2E must refuse to start unless that global context already matches. + return execFileSync('kubectl', ['--context', expectedContext, 'config', 'current-context'], { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], }).trim() @@ -185,7 +188,7 @@ async function globalSetup(): Promise { let current: string try { - current = kubectlCurrentContext() + current = readGlobalKubectlCurrentContext(expected) } catch (err) { throw new Error(`[E2E-GUARD] Failed to read kubectl current-context: ${(err as Error).message}`) } diff --git a/desktop-app/test/e2e-playwright/human-stateless-continuity.spec.ts b/desktop-app/test/e2e-playwright/human-stateless-continuity.spec.ts new file mode 100644 index 0000000000..e8ec0b6977 --- /dev/null +++ b/desktop-app/test/e2e-playwright/human-stateless-continuity.spec.ts @@ -0,0 +1,786 @@ +/** + * Recorded human journey: two Hosts (stateful + stateless), Desktop restart, + * cache, and two verified stateless wake episodes. It mutates the cluster + * (suspends and wakes Hosts) and sends four real model turns. + * + * Environment (no repository doc covers these; this header is the reference): + * - `HUMAN_E2E_STATELESS_CONTINUITY=1` opts in. Without exactly `1` the whole + * file is skipped (see the `test.skip` below), so the default lane never + * mutates a cluster. + * - `HUMAN_E2E_KUBE_CONTEXT=` is required and must equal the + * global-setup context (`E2E_K8S_CONTEXT` / `KUBECONTEXT` / `K8S_CONTEXT`, + * default `clerum-test`), so the journey cannot mutate a different cluster + * than the one the suite was set up against. + * - `QA_RECORDER_CONFIRM_CHAT` acknowledges the model cost. + * - Optional Host overrides: `HUMAN_E2E_STATEFUL_HOST`, `HUMAN_E2E_STATELESS_HOST`, + * `HUMAN_E2E_STATEFUL_HOST_RESOURCE`, `HUMAN_E2E_STATELESS_HOST_RESOURCE`. + */ +import { type ElectronApplication, type Locator, type Page, expect, test } from '@playwright/test' +import { type ChildProcess, execFile } from 'node:child_process' +import { writeFileSync } from 'node:fs' +import { promisify } from 'node:util' +import { + EXTERNAL_REST_API_BASE_URL, + RPC_PROXY_BASE_URL, + assertAllowedTarget, + desktopCredentials, + finalizeRecording, + launchDesktopApp, + login, + openAgentsPage, + requireRecorderConfirm, + screenshotAndLog, +} from './qa-recorder-helpers' + +const execFileAsync = promisify(execFile) + +const KUBE_CONTEXT = process.env.HUMAN_E2E_KUBE_CONTEXT?.trim() || '' +const STATEFUL_HOST = process.env.HUMAN_E2E_STATEFUL_HOST?.trim() || 'chatLLM' +const STATELESS_HOST = process.env.HUMAN_E2E_STATELESS_HOST?.trim() || 'chatllm-stateless' +const STATEFUL_HOST_RESOURCE = process.env.HUMAN_E2E_STATEFUL_HOST_RESOURCE?.trim() || 'chatllm' +const STATELESS_HOST_RESOURCE = + process.env.HUMAN_E2E_STATELESS_HOST_RESOURCE?.trim() || 'chatllm-stateless' +const HCC_DEPLOYMENT = 'host-context-controller' +const HCC_NAMESPACE = 'control-plane' +const STATELESS_NAMESPACE = 'mcp-host' +const STATELESS_DEPLOYMENT = 'chatllm-stateless' +const STATEFUL_DEPLOYMENT = 'chatllm' +const SUSPEND_TIMEOUT_MS = 240_000 + +// E2E_GUARDIAN_IPC_FLOW: Desktop sends RPC through Electron's main process. +// Renderer page.waitForResponse cannot observe those requests. The visible +// assistant response, known conversation marker, and Host CRD/pod state are +// the business signals for this recorded journey. + +type HostSnapshot = { + lifecycleState: string + lifecycleReason: string + replicas: number + readyReplicas: number + podNames: string[] + imageIds: string[] + pullPolicyRejection: string | null +} + +type JourneyMetrics = { + hcc_idle_minutes: number + hcc_idle_floor_minutes: number + hcc_drain_grace_ms: number + hcc_heartbeat_poll_ms: number + stateful_pod_before: string[] + stateless_pod_before: string[] + stateless_initial_lifecycle_state: string + stateless_initial_lifecycle_reason: string + stateless_image_ids_before: string[] + stateless_image_ids_after_reopen: string[] + stateless_image_ids_after_cold_send: string[] + stateless_pull_policy_warning: string | null + first_close_ms: number | null + first_suspend_ms: number + second_launch_to_window_ms: number + second_launch_to_authenticated_ms: number + second_launch_to_stateless_ready_ms: number + second_launch_to_cached_transcript_ms: number | null + cache_visible_before_ready_observed: boolean | null + draft_preserved_while_switching_hosts: boolean + draft_preserved_across_restart: boolean + second_suspend_ms: number + cold_send_to_waking_visible_ms: number | null + cold_send_pre_click_ready_replicas: number + cold_send_first_outcome: 'response' | 'waking' | null + cold_send_to_user_message_visible_ms: number | null + cold_send_to_task_acknowledged_ms: number | null + cold_send_to_ready_observed_ms: number | null + cold_send_response_preceded_stepper: boolean + cold_send_to_response_ms: number + cold_send_to_composer_idle_ms: number + cold_send_retries: number + initial_stateless_send_retries: number + stateful_follow_up_to_user_message_visible_ms: number | null + stateful_follow_up_to_response_ms: number | null + stateless_pod_after_cold_send: string[] + stateful_pod_after: string[] + markers: { + stateful_initial: string + stateless_initial: string + stateless_follow_up: string + stateful_follow_up: string + } +} + +async function kubectl(args: string[], timeoutMs = 20_000): Promise { + const { stdout } = await execFileAsync('kubectl', ['--context', KUBE_CONTEXT, ...args], { + timeout: timeoutMs, + maxBuffer: 8 * 1024 * 1024, + }) + return stdout.toString() +} + +async function jsonKubectl(resource: string, namespace: string): Promise { + return JSON.parse(await kubectl(['-n', namespace, 'get', resource, '-o', 'json'])) +} + +async function readHostSnapshot(deployment: string, hostName: string): Promise { + const [deploymentJson, hostJson, podsJson] = await Promise.all([ + jsonKubectl(`deployment.apps/${deployment}`, STATELESS_NAMESPACE), + jsonKubectl(`host/${hostName}`, STATELESS_NAMESPACE), + kubectl(['-n', STATELESS_NAMESPACE, 'get', 'pods', '-l', `app=${hostName}`, '-o', 'json']), + ]) + const pods = JSON.parse(podsJson).items ?? [] + const spec = deploymentJson.spec ?? {} + const status = deploymentJson.status ?? {} + const lifecycle = hostJson.status?.lifecycle ?? {} + const pullPolicyRejection = (hostJson.status?.conditions ?? []).find( + (condition: { type?: string; status?: string }) => + condition.type === 'StatelessPullPolicyRejected' && condition.status === 'True' + ) + return { + lifecycleState: String(lifecycle.state ?? ''), + lifecycleReason: String(lifecycle.reason ?? ''), + replicas: Number(spec.replicas ?? -1), + readyReplicas: Number(status.readyReplicas ?? 0), + podNames: pods + .map((pod: { metadata?: { name?: string } }) => pod.metadata?.name) + .filter(Boolean) + .sort(), + imageIds: [ + ...new Set( + pods.flatMap((pod: { status?: { containerStatuses?: Array<{ imageID?: string }> } }) => + (pod.status?.containerStatuses ?? []) + .map(container => container.imageID) + .filter((id): id is string => typeof id === 'string' && id.length > 0) + ) + ), + ].sort(), + pullPolicyRejection: pullPolicyRejection?.reason ?? null, + } +} + +async function waitForStatelessSuspended(timeoutMs: number): Promise<{ + snapshot: HostSnapshot + elapsedMs: number +}> { + const started = Date.now() + let last = await readHostSnapshot(STATELESS_DEPLOYMENT, STATELESS_HOST_RESOURCE) + while (Date.now() - started < timeoutMs) { + last = await readHostSnapshot(STATELESS_DEPLOYMENT, STATELESS_HOST_RESOURCE) + if ( + last.lifecycleState === 'suspended' && + last.replicas === 0 && + last.readyReplicas === 0 && + last.podNames.length === 0 + ) { + expect(last.lifecycleReason, 'stateless suspension must be caused by idle timeout').toBe( + 'idle' + ) + return { snapshot: last, elapsedMs: Date.now() - started } + } + await new Promise(resolve => setTimeout(resolve, 5_000)) + } + throw new Error( + `Stateless host did not reach verified replicas-0 suspension: ${JSON.stringify(last)}` + ) +} + +async function waitForStatelessReady(timeoutMs: number): Promise<{ + snapshot: HostSnapshot + elapsedMs: number +}> { + const started = Date.now() + let last = await readHostSnapshot(STATELESS_DEPLOYMENT, STATELESS_HOST_RESOURCE) + while (Date.now() - started < timeoutMs) { + last = await readHostSnapshot(STATELESS_DEPLOYMENT, STATELESS_HOST_RESOURCE) + if (last.replicas >= 1 && last.readyReplicas >= 1 && last.podNames.length > 0) { + return { snapshot: last, elapsedMs: Date.now() - started } + } + await new Promise(resolve => setTimeout(resolve, 2_000)) + } + throw new Error(`Stateless host did not become Ready: ${JSON.stringify(last)}`) +} + +async function readHccCadences(): Promise<{ + idleMinutes: number + idleFloorMinutes: number + drainGraceMs: number + heartbeatPollMs: number +}> { + const deployment = await jsonKubectl(`deployment.apps/${HCC_DEPLOYMENT}`, HCC_NAMESPACE) + const env = deployment.spec?.template?.spec?.containers?.[0]?.env ?? [] + const value = (key: string, fallback: number): number => { + const found = env.find((item: { name?: string }) => item.name === key) + return typeof found?.value === 'string' ? Number(found.value) : fallback + } + return { + idleMinutes: value('CONTEXT_MAPPER_STATELESS_IDLE_MINUTES', 30), + idleFloorMinutes: value('CONTEXT_MAPPER_STATELESS_IDLE_FLOOR_MINUTES', 15), + drainGraceMs: value('CONTEXT_MAPPER_STATELESS_DRAIN_GRACE_MS', 60_000), + heartbeatPollMs: value('CONTEXT_MAPPER_HEARTBEAT_POLL_MS', 10_000), + } +} + +async function desktopProfilePath(app: ElectronApplication): Promise { + return app.evaluate(({ app: electronApp }) => electronApp.getPath('userData')) +} + +async function newChatFromAgentsPage(page: Page, hostName: string): Promise { + await openAgentsPage(page) + await page.getByRole('button', { name: `More actions for ${hostName}`, exact: true }).click() + const menu = page.getByRole('menu') + await expect(menu).toBeVisible({ timeout: 15_000 }) + const newChat = menu + .getByRole('button', { name: /^New chat$/ }) + .or(menu.getByRole('menuitem', { name: /^New chat$/ })) + await expect(newChat).toBeVisible() + await newChat.click() + const composer = page.getByRole('textbox', { name: 'Agent message composer' }) + await expect(composer).toBeVisible({ timeout: 30_000 }) + await expect(page.getByRole('button', { name: /^Switch chat agent$/ })).toContainText(hostName, { + timeout: 30_000, + }) +} + +async function openExactSession(page: Page, title: string, hostName: string): Promise { + await page.getByRole('button', { name: `Open ${title}`, exact: true }).click() + const composer = page.getByRole('textbox', { name: 'Agent message composer' }) + await expect(composer).toBeVisible({ timeout: 30_000 }) + // Existing conversations intentionally replace the new-chat agent selector + // with the breadcrumb identity. The composer placeholder keeps exposing the + // active binding; it uses the Host id, while catalog display names may vary + // in case (chatllm vs chatLLM), so compare case-insensitively. + await expect(composer).toHaveAttribute( + 'placeholder', + `Message ${hostName}... (Enter to send, Shift+Enter for newline)`, + { timeout: 30_000, ignoreCase: true } + ) +} + +async function renameSessionByMarker(page: Page, marker: string, title: string): Promise { + await expect(page.getByTestId('agent-response').filter({ hasText: marker })).toBeVisible() + const row = page.locator('.nav-latest-session.active') + await expect(row).toBeVisible({ timeout: 30_000 }) + const options = row.getByRole('button', { name: /^Session options for / }) + await options.click() + await page.getByRole('menuitem', { name: 'Rename', exact: true }).click() + const input = page.getByRole('textbox', { name: 'Rename session', exact: true }) + await expect(input).toBeVisible() + await input.fill(title) + await input.press('Enter') + await expect(page.getByRole('button', { name: `Open ${title}`, exact: true })).toBeVisible({ + timeout: 30_000, + }) +} + +async function humanType(locator: Locator, text: string): Promise { + await locator.fill('') + await locator.pressSequentially(text, { delay: 90 }) +} + +async function sendMarker( + page: Page, + marker: string, + prompt: string +): Promise<{ toUserMessageMs: number; toResponseMs: number }> { + const composer = page.getByRole('textbox', { name: 'Agent message composer' }) + const response = page.getByTestId('agent-response').filter({ hasText: marker }) + await humanType(composer, prompt) + await expect(page.getByTestId('send-button')).toBeEnabled() + const sendStarted = Date.now() + await page.getByTestId('send-button').click() + await expect(page.getByTestId('message-list')).toContainText(marker, { timeout: 30_000 }) + const toUserMessageMs = Date.now() - sendStarted + await expect(response).toBeVisible({ timeout: 150_000 }) + await expect(response).toContainText(marker, { timeout: 150_000 }) + const toResponseMs = Date.now() - sendStarted + await expect(composer).toHaveValue('', { timeout: 150_000 }) + await expect(page.getByTestId('send-button')).toHaveAttribute('aria-label', 'Send message', { + timeout: 150_000, + }) + await expect(page.getByTestId('send-button')).toBeDisabled({ timeout: 150_000 }) + return { toUserMessageMs, toResponseMs } +} + +async function authenticatedWithoutRelogin(page: Page): Promise { + await expect(page.locator('.boot-overlay')).toBeHidden({ timeout: 30_000 }) + const emailInput = page.locator('#email-input') + const authenticatedShell = page.getByTestId('nav-chat') + await expect(authenticatedShell).toBeVisible({ timeout: 30_000 }) + await expect(emailInput).toBeHidden({ timeout: 30_000 }) +} + +async function resolveColdSend( + page: Page, + responseMarker: string, + contextMarker: string, + startedAt?: number +): Promise<{ + wakingVisibleMs: number | null + retries: number + firstOutcome: 'response' | 'waking' +}> { + const started = startedAt ?? Date.now() + const response = page.getByTestId('agent-response').filter({ hasText: responseMarker }) + const waking = page.getByTestId('waking-state') + let firstOutcome: 'response' | 'waking' | null = null + await expect + .poll( + async () => { + if (await response.isVisible()) return (firstOutcome = 'response') + if (await waking.isVisible()) return (firstOutcome = 'waking') + return null + }, + { timeout: 270_000 } + ) + .not.toBeNull() + let wakingVisibleMs: number | null = null + let retries = 0 + if (firstOutcome === null) { + throw new Error('Cold send ended without an observable first outcome') + } + if (firstOutcome === 'waking') { + wakingVisibleMs = Date.now() - started + // The retry is a user action. Wait for the observed Host to become Ready + // before taking it; a fast click can fail while the pod is still waking. + await waitForStatelessReady(270_000) + const retry = waking.getByRole('button', { name: /retry last send/i }) + await expect(retry).toBeEnabled({ timeout: 150_000 }) + await retry.click() + retries += 1 + } + await expect(response).toBeVisible({ timeout: 270_000 }) + await expect(response).toContainText(contextMarker, { timeout: 150_000 }) + return { + wakingVisibleMs, + retries, + firstOutcome, + } +} + +async function closeElectron( + app: ElectronApplication | undefined, + capturedChild?: ChildProcess +): Promise { + // finalizeRecording closes Electron to flush the video; the Playwright + // application handle is disposed after that, so callers capture the child + // before finalization and this helper tolerates an already-closed app. + let child = capturedChild + if (!child) { + try { + child = app?.process() + } catch { + child = undefined + } + } + const pid = child?.pid ?? null + await app?.close().catch(() => undefined) + if (!child) return null + const deadline = Date.now() + 30_000 + while (Date.now() < deadline && child.exitCode === null && child.signalCode === null) { + await new Promise(resolve => setTimeout(resolve, 250)) + } + expect( + child.exitCode !== null || child.signalCode !== null, + `Electron process ${pid ?? 'unknown'} must exit; window close alone is not app exit on macOS` + ).toBe(true) + return pid +} + +test.skip( + process.env.HUMAN_E2E_STATELESS_CONTINUITY !== '1', + 'Set HUMAN_E2E_STATELESS_CONTINUITY=1 for this recorded, cluster-mutating journey' +) + +test('human journey — two Hosts, restart, cache, and two verified stateless wake episodes', async ({}, testInfo) => { + test.setTimeout(1_800_000) + + if (!KUBE_CONTEXT) { + throw new Error('HUMAN_E2E_KUBE_CONTEXT is required for the recorded stateless journey.') + } + const setupContext = + process.env.E2E_K8S_CONTEXT || + process.env.KUBECONTEXT || + process.env.K8S_CONTEXT || + 'clerum-test' + if (KUBE_CONTEXT !== setupContext) { + throw new Error( + `HUMAN_E2E_KUBE_CONTEXT must match the global-setup context: expected ${setupContext}, got ${KUBE_CONTEXT}.` + ) + } + requireRecorderConfirm( + 'QA_RECORDER_CONFIRM_CHAT', + 'This journey sends four real model turns and may incur model cost.' + ) + + await assertAllowedTarget('EXTERNAL_REST_API_BASE_URL', EXTERNAL_REST_API_BASE_URL) + await assertAllowedTarget('RPC_PROXY_BASE_URL', RPC_PROXY_BASE_URL) + + const runId = `${Date.now()}-${Math.random().toString(16).slice(2, 8)}` + const statefulTitle = `HUMAN_STATEFUL_${runId}` + const statelessTitle = `HUMAN_STATELESS_${runId}` + const statefulMarker = `HUMAN_A_${runId}` + const statelessMarker = `HUMAN_B_${runId}` + const statelessFollowMarker = `HUMAN_B2_${runId}` + const statefulFollowMarker = `HUMAN_A2_${runId}` + + let app: ElectronApplication | undefined + let page: Page | undefined + // The handles are cleared while Electron is closed between steps; a step that + // uses one while it is cleared must fail loudly, not act on `undefined`. + const currentApp = (): ElectronApplication => { + if (!app) throw new Error('Electron app is not running at this step') + return app + } + const currentPage = (): Page => { + if (!page) throw new Error('Desktop page is not open at this step') + return page + } + let firstProfilePath = '' + + const metrics: JourneyMetrics = { + hcc_idle_minutes: 0, + hcc_idle_floor_minutes: 0, + hcc_drain_grace_ms: 0, + hcc_heartbeat_poll_ms: 0, + stateful_pod_before: [], + stateless_pod_before: [], + stateless_initial_lifecycle_state: '', + stateless_initial_lifecycle_reason: '', + stateless_image_ids_before: [], + stateless_image_ids_after_reopen: [], + stateless_image_ids_after_cold_send: [], + stateless_pull_policy_warning: null, + first_close_ms: null, + first_suspend_ms: 0, + second_launch_to_window_ms: 0, + second_launch_to_authenticated_ms: 0, + second_launch_to_stateless_ready_ms: 0, + second_launch_to_cached_transcript_ms: null, + cache_visible_before_ready_observed: null, + draft_preserved_while_switching_hosts: false, + draft_preserved_across_restart: false, + second_suspend_ms: 0, + cold_send_to_waking_visible_ms: null, + cold_send_pre_click_ready_replicas: 0, + cold_send_first_outcome: null, + cold_send_to_user_message_visible_ms: null, + cold_send_to_task_acknowledged_ms: null, + cold_send_to_ready_observed_ms: null, + cold_send_response_preceded_stepper: false, + cold_send_to_response_ms: 0, + cold_send_to_composer_idle_ms: 0, + cold_send_retries: 0, + initial_stateless_send_retries: 0, + stateful_follow_up_to_user_message_visible_ms: null, + stateful_follow_up_to_response_ms: null, + stateless_pod_after_cold_send: [], + stateful_pod_after: [], + markers: { + stateful_initial: statefulMarker, + stateless_initial: statelessMarker, + stateless_follow_up: statelessFollowMarker, + stateful_follow_up: statefulFollowMarker, + }, + } + + try { + await test.step('verify both Hosts and the branch-owned runtime baseline', async () => { + const [statefulBefore, statelessBefore, cadences] = await Promise.all([ + readHostSnapshot(STATEFUL_DEPLOYMENT, STATEFUL_HOST_RESOURCE), + readHostSnapshot(STATELESS_DEPLOYMENT, STATELESS_HOST_RESOURCE), + readHccCadences(), + ]) + expect( + statefulBefore.readyReplicas, + 'stateful control Host must start Ready' + ).toBeGreaterThan(0) + if (statelessBefore.lifecycleState === 'suspended') { + expect(statelessBefore.lifecycleReason).toBe('idle') + expect(statelessBefore.replicas).toBe(0) + expect(statelessBefore.readyReplicas).toBe(0) + expect(statelessBefore.podNames).toHaveLength(0) + } else { + expect( + statelessBefore.readyReplicas, + 'active stateless Host must start Ready' + ).toBeGreaterThan(0) + } + const estimatedSuspendMs = + Math.max(cadences.idleMinutes, cadences.idleFloorMinutes) * 60_000 + + cadences.drainGraceMs + + cadences.heartbeatPollMs * 2 + expect( + Object.values(cadences).every(value => Number.isFinite(value) && value > 0) && + estimatedSuspendMs < SUSPEND_TIMEOUT_MS, + 'HCC idle/floor/drain/poll cadences must permit a bounded natural suspension; configure the owned profile before recording' + ).toBe(true) + metrics.hcc_idle_minutes = cadences.idleMinutes + metrics.hcc_idle_floor_minutes = cadences.idleFloorMinutes + metrics.hcc_drain_grace_ms = cadences.drainGraceMs + metrics.hcc_heartbeat_poll_ms = cadences.heartbeatPollMs + metrics.stateful_pod_before = statefulBefore.podNames + metrics.stateless_initial_lifecycle_state = statelessBefore.lifecycleState + metrics.stateless_initial_lifecycle_reason = statelessBefore.lifecycleReason + metrics.stateless_pull_policy_warning = statelessBefore.pullPolicyRejection + }) + + await test.step('first Desktop session: create, name, and exercise both Host conversations', async () => { + const launched = await launchDesktopApp(testInfo) + app = launched.app + page = launched.page + firstProfilePath = await desktopProfilePath(app) + await login(page, desktopCredentials()) + await openAgentsPage(page) + await expect( + page.getByRole('button', { name: `More actions for ${STATEFUL_HOST}`, exact: true }) + ).toBeVisible({ timeout: 30_000 }) + await expect( + page.getByRole('button', { name: `More actions for ${STATELESS_HOST}`, exact: true }) + ).toBeVisible({ timeout: 30_000 }) + await screenshotAndLog(page, testInfo, '01-authenticated-fleet') + + await newChatFromAgentsPage(page, STATELESS_HOST) + const statelessComposer = page.getByRole('textbox', { name: 'Agent message composer' }) + await humanType( + statelessComposer, + `Reply with exactly: ${statelessMarker}. Remember this token for a later question.` + ) + const statelessSend = page.getByTestId('send-button') + await expect(statelessSend).toBeEnabled() + await statelessSend.click() + const firstOutcome = await resolveColdSend(page, statelessMarker, statelessMarker) + metrics.initial_stateless_send_retries = firstOutcome.retries + await expect(statelessComposer).toHaveValue('', { timeout: 150_000 }) + await expect(statelessSend).toHaveAttribute('aria-label', 'Send message', { + timeout: 150_000, + }) + await expect(statelessSend).toBeDisabled({ timeout: 150_000 }) + const firstReady = await waitForStatelessReady(270_000) + expect( + firstReady.snapshot.imageIds, + 'record the actual stateless container imageID after the first UI response' + ).not.toHaveLength(0) + metrics.stateless_pod_before = firstReady.snapshot.podNames + metrics.stateless_image_ids_before = firstReady.snapshot.imageIds + await renameSessionByMarker(page, statelessMarker, statelessTitle) + await screenshotAndLog(page, testInfo, '02-stateless-initial-turn') + + await newChatFromAgentsPage(page, STATEFUL_HOST) + await sendMarker( + page, + statefulMarker, + `Reply with exactly: ${statefulMarker}. Remember this token for a later question.` + ) + await renameSessionByMarker(page, statefulMarker, statefulTitle) + await screenshotAndLog(page, testInfo, '03-stateful-initial-turn') + + const composer = page.getByRole('textbox', { name: 'Agent message composer' }) + await openExactSession(page, statelessTitle, STATELESS_HOST) + await humanType(composer, `Unsent stateless draft ${runId}`) + await openExactSession(page, statefulTitle, STATEFUL_HOST) + await expect(composer).toHaveValue('') + await openExactSession(page, statelessTitle, STATELESS_HOST) + await expect(composer).toHaveValue(`Unsent stateless draft ${runId}`) + metrics.draft_preserved_while_switching_hosts = true + await screenshotAndLog(page, testInfo, '04-draft-preserved-across-host-switch') + }) + + await test.step('exit Electron and prove a real replicas-0 suspension', async () => { + const closeStarted = Date.now() + const electronChild = currentApp().process() + await finalizeRecording(app, page) + await closeElectron(app, electronChild) + metrics.first_close_ms = Date.now() - closeStarted + app = undefined + page = undefined + + const suspended = await waitForStatelessSuspended(SUSPEND_TIMEOUT_MS) + metrics.first_suspend_ms = suspended.elapsedMs + expect(suspended.snapshot.lifecycleState).toBe('suspended') + expect(suspended.snapshot.replicas).toBe(0) + expect(suspended.snapshot.podNames).toHaveLength(0) + }) + + await test.step('reopen the same Desktop profile while the stateless Host is suspended', async () => { + const launchStarted = Date.now() + const launched = await launchDesktopApp(testInfo) + metrics.second_launch_to_window_ms = Date.now() - launchStarted + app = launched.app + page = launched.page + expect(await desktopProfilePath(app)).toBe(firstProfilePath) + await authenticatedWithoutRelogin(page) + metrics.second_launch_to_authenticated_ms = Date.now() - launchStarted + + // Observe the catalog-driven prewarm wake from authentication while the + // retained conversation opens from local cache before Ready is observed. + let readyObserved = false + const readyObservation = waitForStatelessReady(270_000).then(result => { + readyObserved = true + return { result, observedAt: Date.now() } + }) + await openExactSession(page, statelessTitle, STATELESS_HOST) + await expect(page.getByTestId('message-list')).toContainText(statelessMarker, { + timeout: 30_000, + }) + metrics.second_launch_to_cached_transcript_ms = Date.now() - launchStarted + metrics.cache_visible_before_ready_observed = !readyObserved + const ready = await readyObservation + metrics.second_launch_to_stateless_ready_ms = ready.observedAt - launchStarted + metrics.stateless_image_ids_after_reopen = ready.result.snapshot.imageIds + expect(ready.result.snapshot.readyReplicas).toBeGreaterThan(0) + expect(ready.result.snapshot.podNames.join(',')).not.toBe( + metrics.stateless_pod_before.join(',') + ) + expect(ready.result.snapshot.imageIds).toEqual(metrics.stateless_image_ids_before) + + const composer = page.getByRole('textbox', { name: 'Agent message composer' }) + // Drafts are currently renderer-memory scoped. Record this observation; + // PR A requires draft continuity during transient errors in a live app. + metrics.draft_preserved_across_restart = + (await composer.inputValue()) === `Unsent stateless draft ${runId}` + await screenshotAndLog(page, testInfo, '05-reopened-stateless-transcript') + + await openExactSession(page, statefulTitle, STATEFUL_HOST) + await expect(page.getByTestId('message-list')).toContainText(statefulMarker, { + timeout: 30_000, + }) + await screenshotAndLog(page, testInfo, '06-reopened-stateful-transcript') + }) + + await test.step('prepare and send from a second verified cold stateless state', async () => { + await openExactSession(currentPage(), statelessTitle, STATELESS_HOST) + const composer = currentPage().getByRole('textbox', { name: 'Agent message composer' }) + await composer.fill('') + const secondSuspended = await waitForStatelessSuspended(SUSPEND_TIMEOUT_MS) + metrics.second_suspend_ms = secondSuspended.elapsedMs + expect(secondSuspended.snapshot.lifecycleState).toBe('suspended') + expect(secondSuspended.snapshot.replicas).toBe(0) + expect(secondSuspended.snapshot.podNames).toHaveLength(0) + + const followPrompt = `Earlier I gave you a token. Reply with exactly that token followed by ${statelessFollowMarker}.` + await humanType(composer, followPrompt) + await expect(currentPage().getByTestId('send-button')).toBeEnabled() + // The catalog prewarm after reopen is measured separately by + // second_launch_to_stateless_ready_ms. This fresh snapshot proves that + // the second send is itself a cold wake intent, not a continuation of + // that earlier prewarm window. + const preClick = await readHostSnapshot(STATELESS_DEPLOYMENT, STATELESS_HOST_RESOURCE) + metrics.cold_send_pre_click_ready_replicas = preClick.readyReplicas + expect(preClick.lifecycleState).toBe('suspended') + expect(preClick.replicas).toBe(0) + expect(preClick.readyReplicas).toBe(0) + expect(preClick.podNames).toHaveLength(0) + + // Arm every observer before the trusted click: Ready polling starts on + // the still-suspended Host, and the stepper count baseline excludes + // completed steppers from earlier turns that keep the same test id. + let readyObserved = false + const readyObservation = waitForStatelessReady(270_000).then(result => { + readyObserved = true + return { result, observedAt: Date.now() } + }) + const taskStepper = currentPage().getByTestId('progress-stepper') + const stepperBaseline = await taskStepper.count() + const sendStarted = Date.now() + await currentPage().getByTestId('send-button').click() + await expect(currentPage().getByTestId('message-list')).toContainText(statelessFollowMarker, { + timeout: 30_000, + }) + metrics.cold_send_to_user_message_visible_ms = Date.now() - sendStarted + + const followResponse = currentPage() + .getByTestId('agent-response') + .filter({ hasText: statelessFollowMarker }) + const stepperWatch = expect + .poll( + async () => { + if ( + metrics.cold_send_to_task_acknowledged_ms === null && + (await taskStepper.count()) > stepperBaseline + ) { + metrics.cold_send_to_task_acknowledged_ms = Date.now() - sendStarted + } + if (metrics.cold_send_to_task_acknowledged_ms === null) { + metrics.cold_send_response_preceded_stepper = await followResponse + .isVisible() + .catch(() => false) + } + return ( + metrics.cold_send_to_task_acknowledged_ms !== null || + (await followResponse.isVisible().catch(() => false)) + ) + }, + { + timeout: 270_000, + message: 'Cold send must expose either a task acknowledgement or a final response', + } + ) + .toBe(true) + const outcome = await resolveColdSend( + currentPage(), + statelessFollowMarker, + statelessMarker, + sendStarted + ) + metrics.cold_send_to_waking_visible_ms = outcome.wakingVisibleMs + metrics.cold_send_first_outcome = outcome.firstOutcome + metrics.cold_send_to_response_ms = Date.now() - sendStarted + metrics.cold_send_retries = outcome.retries + await expect(composer).toHaveValue('', { timeout: 150_000 }) + await expect(currentPage().getByTestId('send-button')).toHaveAttribute( + 'aria-label', + 'Send message', + { + timeout: 150_000, + } + ) + await expect(currentPage().getByTestId('send-button')).toBeDisabled({ timeout: 150_000 }) + metrics.cold_send_to_composer_idle_ms = Date.now() - sendStarted + await stepperWatch + + const readyAfterSend = await readyObservation + metrics.cold_send_to_ready_observed_ms = readyAfterSend.observedAt - sendStarted + metrics.stateless_pod_after_cold_send = readyAfterSend.result.snapshot.podNames + metrics.stateless_image_ids_after_cold_send = readyAfterSend.result.snapshot.imageIds + expect(readyAfterSend.result.snapshot.readyReplicas).toBeGreaterThan(0) + expect(readyAfterSend.result.snapshot.imageIds).toEqual(metrics.stateless_image_ids_before) + await expect( + currentPage().getByTestId('agent-response').filter({ hasText: statelessFollowMarker }) + ).toContainText(statelessFollowMarker, { timeout: 30_000 }) + await screenshotAndLog( + currentPage(), + testInfo, + '07-stateless-context-continuity-after-cold-send' + ) + }) + + await test.step('stateful control remains continuous and responsive', async () => { + await openExactSession(currentPage(), statefulTitle, STATEFUL_HOST) + await expect(currentPage().getByTestId('message-list')).toContainText(statefulMarker, { + timeout: 30_000, + }) + const statefulTimings = await sendMarker( + currentPage(), + statefulFollowMarker, + `What was the earlier token in this conversation? Reply with that token followed by ${statefulFollowMarker}.` + ) + metrics.stateful_follow_up_to_user_message_visible_ms = statefulTimings.toUserMessageMs + metrics.stateful_follow_up_to_response_ms = statefulTimings.toResponseMs + await expect( + currentPage().getByTestId('agent-response').filter({ hasText: statefulFollowMarker }) + ).toContainText(statefulMarker) + const statefulAfter = await readHostSnapshot(STATEFUL_DEPLOYMENT, STATEFUL_HOST_RESOURCE) + metrics.stateful_pod_after = statefulAfter.podNames + expect(statefulAfter.readyReplicas).toBeGreaterThan(0) + expect(statefulAfter.podNames.join(',')).toBe(metrics.stateful_pod_before.join(',')) + await screenshotAndLog(currentPage(), testInfo, '08-stateful-control-continuity') + }) + } finally { + await finalizeRecording(app, page).catch(() => undefined) + const metricsPath = testInfo.outputPath('human-stateless-continuity-metrics.json') + writeFileSync(metricsPath, JSON.stringify(metrics, null, 2), 'utf8') + await testInfo.attach('human-stateless-continuity-metrics', { + body: JSON.stringify(metrics, null, 2), + contentType: 'application/json', + }) + console.log(`[HumanStatelessContinuity] ${JSON.stringify(metrics)}`) + } +}) diff --git a/desktop-app/test/e2e-playwright/pr849-controlled-parity.spec.ts b/desktop-app/test/e2e-playwright/pr849-controlled-parity.spec.ts new file mode 100644 index 0000000000..8532c477f5 --- /dev/null +++ b/desktop-app/test/e2e-playwright/pr849-controlled-parity.spec.ts @@ -0,0 +1,1226 @@ +/// +// E2E_GUARDIAN_IPC_FLOW: Desktop sends RPC and GFS calls through Electron's +// main-process IPC, so renderer page.waitForRequest cannot observe this journey. +import { type ElectronApplication, type Page, expect, test } from '@playwright/test' +import { execFile, execFileSync } from 'node:child_process' +import { createHash } from 'node:crypto' +import path from 'node:path' +import { + type AgentGfsFixtures, + assertGfsInfraHealthy, + seedAgentGfsFixtures, +} from './helpers/gfsFixtures' +import { + EXTERNAL_REST_API_BASE_URL, + RPC_PROXY_BASE_URL, + assertAllowedTarget, + desktopCredentials, + launchDesktopApp, + login, + openAgentsPage, +} from './qa-recorder-helpers' + +const STATELESS_HOST = process.env.E2E_STATELESS_HOST_REF || 'chatllm-stateless' +const STATEFUL_HOST = process.env.E2E_STATEFUL_HOST_REF || 'chatllm' +const STATEFUL_HOST_DISPLAY = process.env.E2E_STATEFUL_HOST_DISPLAY || 'chatLLM' +const EXPECTED_STATELESS_MODEL_PROVIDER = process.env.E2E_EXPECTED_STATELESS_MODEL_PROVIDER || '' +const EXPECTED_STATELESS_MODEL_NAME = process.env.E2E_EXPECTED_STATELESS_MODEL_NAME || '' +const REQUIRE_IDENTITY_FILES = process.env.E2E_REQUIRE_IDENTITY_FILES === '1' +const KUBE_CONTEXT = + process.env.E2E_K8S_CONTEXT || process.env.KUBECONTEXT || process.env.K8S_CONTEXT || '' +const HCC_DEPLOYMENT = 'host-context-controller' +const RUNTIME_TOKEN_REVISION_ANNOTATION = 'clerum.io/runtime-token-revision' +const RUNTIME_TOKEN_SECRET_REVISION_ANNOTATION = 'clerum.io/runtime-token-secret-revision' +const RUNTIME_TOKEN_BOOTSTRAP_STATE_ANNOTATION = 'clerum.io/runtime-token-bootstrap-state' +const RUNTIME_TOKEN_ROLLOUT_REQUIRED_ANNOTATION = 'clerum.io/runtime-token-rollout-required' +const RUNTIME_TOKEN_ISSUED_AT_ANNOTATION = 'clerum.io/runtime-token-issued-at' +const BASELINE_HCC_ENV = { + CONTEXT_MAPPER_STATELESS_IDLE_MINUTES: '30', + CONTEXT_MAPPER_STATELESS_IDLE_FLOOR_MINUTES: '15', + CONTEXT_MAPPER_STATELESS_DRAIN_GRACE_MS: '60000', + CONTEXT_MAPPER_HEARTBEAT_POLL_MS: '10000', +} +const ACCELERATED_HCC_ENV = { + CONTEXT_MAPPER_STATELESS_IDLE_MINUTES: '1', + CONTEXT_MAPPER_STATELESS_IDLE_FLOOR_MINUTES: '1', + CONTEXT_MAPPER_STATELESS_DRAIN_GRACE_MS: '20000', + CONTEXT_MAPPER_HEARTBEAT_POLL_MS: '5000', +} +const IDENTITY_FILES = ['IDENTITY.md', 'SOUL.md', 'AGENTS.md', 'USER.md'] as const +const EMPTY_FILE_SHA256 = createHash('sha256').update('').digest('hex') +type IdentityFileName = (typeof IDENTITY_FILES)[number] +const HCC_RUNTIME_PROBE_ANNOTATION = 'clerum.io/pr849-hcc-runtime-probe' + +type Json = Record +type HostSnapshot = { + hostUid: string + lifecycle: { state: string; reason: string } + replicas: number + readyReplicas: number + pods: string[] + podUids: string[] + imageIds: string[] + templateHash: string + rawTemplateHash: string + runtimeTokenRevision: string + parameterHash: string + claimName: string + pvcUid: string + mountPath: string + sqliteDir: string | null + declaredIdentityFiles: string[] +} + +function kubectl(args: readonly string[], timeout = 20_000): string { + if (!KUBE_CONTEXT) throw new Error('E2E_K8S_CONTEXT is required') + return execFileSync('kubectl', ['--context', KUBE_CONTEXT, ...args], { + encoding: 'utf8', + timeout, + maxBuffer: 12 * 1024 * 1024, + }) +} + +function json(args: readonly string[]): Json { + return JSON.parse(kubectl(args)) as Json +} + +function kubectlAsync(args: readonly string[], timeout = 20_000): Promise { + return new Promise((resolve, reject) => { + execFile( + 'kubectl', + ['--context', KUBE_CONTEXT, ...args], + { encoding: 'utf8', timeout, maxBuffer: 12 * 1024 * 1024 }, + (error, stdout, stderr) => { + if (error) { + reject(new Error(`${error.message}: ${stderr}`)) + return + } + resolve(stdout) + } + ) + }) +} + +async function jsonAsync(args: readonly string[]): Promise { + return JSON.parse(await kubectlAsync(args)) as Json +} + +function hash(value: unknown): string { + return createHash('sha256').update(JSON.stringify(value)).digest('hex') +} + +function functionalWorkloadTemplate(template: Json | undefined): Json | undefined { + if (!template?.metadata?.annotations) return template + const functional = structuredClone(template) + const annotations = { ...(functional.metadata?.annotations ?? {}) } + delete annotations[RUNTIME_TOKEN_REVISION_ANNOTATION] + functional.metadata.annotations = annotations + return functional +} + +async function hostSnapshot( + deploymentName: string, + hostName: string, + requireStatelessSqlite: boolean +): Promise { + const [deployment, host, podList] = await Promise.all([ + jsonAsync(['-n', 'mcp-host', 'get', `deployment/${deploymentName}`, '-o', 'json']), + jsonAsync(['-n', 'mcp-host', 'get', `host/${hostName}`, '-o', 'json']), + jsonAsync(['-n', 'mcp-host', 'get', 'pods', '-l', `app=${hostName}`, '-o', 'json']), + ]) + const container = deployment.spec?.template?.spec?.containers?.find( + (item: Json) => item.name === 'mcp-host' + ) + const workspace = container?.volumeMounts?.find((item: Json) => item.name === 'workspace') + const volume = deployment.spec?.template?.spec?.volumes?.find( + (item: Json) => item.name === workspace?.name + ) + const claimName = volume?.persistentVolumeClaim?.claimName + const sqliteDir = container?.env?.find( + (item: Json) => item.name === 'CLERUM_SESSION_DB_DIR' + )?.value + if (!workspace?.mountPath || !claimName) { + throw new Error(`${hostName} lost its workspace PVC declaration`) + } + if (requireStatelessSqlite) { + const sqliteMount = container?.volumeMounts?.find( + (item: Json) => item.name === workspace.name && item.mountPath === sqliteDir + ) + if (!sqliteDir || sqliteMount?.subPath !== 'state') { + throw new Error(`${hostName} lost its dedicated SQLite subPath mount`) + } + } + const claim = await jsonAsync(['-n', 'mcp-host', 'get', `pvc/${claimName}`, '-o', 'json']) + const livePods = (podList.items ?? []).filter((item: Json) => !item.metadata?.deletionTimestamp) + + return { + hostUid: String(host.metadata?.uid ?? ''), + lifecycle: { + state: String(host.status?.lifecycle?.state ?? ''), + reason: String(host.status?.lifecycle?.reason ?? ''), + }, + replicas: Number(deployment.spec?.replicas ?? -1), + readyReplicas: Number(deployment.status?.readyReplicas ?? 0), + pods: livePods + .map((item: Json) => item.metadata?.name) + .filter(Boolean) + .sort(), + podUids: livePods + .map((item: Json) => item.metadata?.uid) + .filter(Boolean) + .sort(), + imageIds: livePods + .flatMap((item: Json) => + (item.status?.containerStatuses ?? []).map((status: Json) => status.imageID) + ) + .filter(Boolean) + .sort(), + templateHash: hash(functionalWorkloadTemplate(deployment.spec?.template)), + rawTemplateHash: hash(deployment.spec?.template ?? {}), + runtimeTokenRevision: String( + deployment.spec?.template?.metadata?.annotations?.[RUNTIME_TOKEN_REVISION_ANNOTATION] ?? '' + ), + parameterHash: hash({ + personalization: host.spec?.personalization, + model: host.spec?.model, + contextRef: host.spec?.contextRef, + workflowControl: host.spec?.workflowControl, + approval: host.spec?.approval, + }), + claimName, + pvcUid: String(claim.metadata?.uid ?? ''), + mountPath: workspace.mountPath, + sqliteDir: sqliteDir ?? null, + declaredIdentityFiles: host.spec?.personalization == null ? [] : [...IDENTITY_FILES], + } +} + +function requireInheritedMutationLease(): void { + const profile = process.env.T2_PROFILE ?? process.env.MINIKUBE_PROFILE ?? '' + const leasedContext = process.env.T2_CONTEXT ?? process.env.CONTROL_API_REAL_PG_CONTEXT ?? profile + if (!profile || KUBE_CONTEXT !== leasedContext || leasedContext !== profile) { + throw new Error( + `Kubernetes context must exactly match the inherited mutation lease: kubectl=${KUBE_CONTEXT}, lease=${leasedContext}, profile=${profile}` + ) + } + execFileSync( + 'bash', + [path.resolve(__dirname, '../../../scripts/minikube/require-t2-mutation-lock.sh')], + { env: process.env, encoding: 'utf8', timeout: 20_000 } + ) +} + +function hccEnv(): Record { + const deployment = json([ + '-n', + 'control-plane', + 'get', + `deployment/${HCC_DEPLOYMENT}`, + '-o', + 'json', + ]) + const env = + deployment.spec?.template?.spec?.containers + ?.find((item: Json) => item.name === 'host-context-controller') + ?.env?.filter((item: Json) => item.name in BASELINE_HCC_ENV) ?? [] + return Object.fromEntries(env.map((item: Json) => [item.name, item.value])) +} + +function setHccEnv(values: Record): void { + kubectl( + [ + '-n', + 'control-plane', + 'set', + 'env', + `deployment/${HCC_DEPLOYMENT}`, + ...Object.entries(values).map(([key, value]) => `${key}=${value}`), + ], + 30_000 + ) + kubectl( + ['-n', 'control-plane', 'rollout', 'status', `deployment/${HCC_DEPLOYMENT}`, '--timeout=180s'], + 200_000 + ) + expect(hccEnv()).toEqual(values) +} + +function identityFileSnapshot( + podName: string +): Record { + const script = [ + 'set -eu', + ...IDENTITY_FILES.map(file => { + const path = `/workspace/${file}` + return `if test -f "${path}"; then sha256sum "${path}"; else printf 'ABSENT ${file}\\n'; fi` + }), + ].join('\n') + const output = kubectl(['-n', 'mcp-host', 'exec', podName, '--', 'sh', '-c', script]) + const snapshot: Record = { + 'IDENTITY.md': { exists: false, sha256: null }, + 'SOUL.md': { exists: false, sha256: null }, + 'AGENTS.md': { exists: false, sha256: null }, + 'USER.md': { exists: false, sha256: null }, + } + const observedFiles = new Set() + for (const line of output.trim().split('\n')) { + const absentMatch = /^ABSENT (IDENTITY\.md|SOUL\.md|AGENTS\.md|USER\.md)$/.exec(line.trim()) + const absentFile = absentMatch?.[1] as IdentityFileName | undefined + if (absentFile) { + if (observedFiles.has(absentFile)) { + throw new Error(`Duplicate identity output for ${absentFile} on ${podName}`) + } + observedFiles.add(absentFile) + snapshot[absentFile] = { exists: false, sha256: null } + continue + } + const presentMatch = + /^([0-9a-f]{64})\s+\/workspace\/(IDENTITY\.md|SOUL\.md|AGENTS\.md|USER\.md)$/.exec( + line.trim() + ) + const digest = presentMatch?.[1] + const fileName = presentMatch?.[2] as IdentityFileName | undefined + if (!digest || !fileName) { + throw new Error(`Unexpected identity hash output for ${podName}`) + } + if (observedFiles.has(fileName)) { + throw new Error(`Duplicate identity output for ${fileName} on ${podName}`) + } + observedFiles.add(fileName) + snapshot[fileName] = { exists: true, sha256: digest } + } + expect([...observedFiles].sort()).toEqual([...IDENTITY_FILES].sort()) + return snapshot +} + +type IdentityFileSnapshot = ReturnType + +function proveStatelessRuntimeModel( + podName: string, + sinceTime: string +): { toolStartEvents: number; matchingEvents: number; otherModelEvents: number } { + const logs = kubectl(['-n', 'mcp-host', 'logs', podName, '--since-time', sinceTime]) + let toolStartEvents = 0 + let matchingEvents = 0 + let otherModelEvents = 0 + + for (const line of logs.trim().split('\n')) { + if (!line.trim()) continue + let event: Json + try { + event = JSON.parse(line) as Json + } catch { + continue + } + if (event.msg !== 'LLM tool completion started') continue + toolStartEvents += 1 + if ( + event.provider === EXPECTED_STATELESS_MODEL_PROVIDER && + event.model === EXPECTED_STATELESS_MODEL_NAME + ) { + matchingEvents += 1 + } else { + otherModelEvents += 1 + } + } + + if (matchingEvents === 0 || otherModelEvents > 0) { + throw new Error( + `Expected ${EXPECTED_STATELESS_MODEL_PROVIDER}/${EXPECTED_STATELESS_MODEL_NAME} ` + + `to serve every tool-capable LLM call; events=${toolStartEvents}, ` + + `matching=${matchingEvents}, other=${otherModelEvents}` + ) + } + return { toolStartEvents, matchingEvents, otherModelEvents } +} + +async function waitForStateless( + desired: 'suspended' | 'ready', + timeoutMs: number, + shouldStop?: () => boolean +): Promise<{ snapshot: HostSnapshot; elapsedMs: number }> { + const started = Date.now() + for (;;) { + if (shouldStop?.()) throw new Error(`Stopped waiting for ${STATELESS_HOST}=${desired}`) + const snapshot = await hostSnapshot(STATELESS_HOST, STATELESS_HOST, true) + if (shouldStop?.()) throw new Error(`Stopped waiting for ${STATELESS_HOST}=${desired}`) + const matched = + desired === 'suspended' + ? snapshot.lifecycle.state === 'suspended' && + snapshot.lifecycle.reason === 'idle' && + snapshot.replicas === 0 && + snapshot.readyReplicas === 0 && + snapshot.pods.length === 0 + : snapshot.replicas === 1 && snapshot.readyReplicas === 1 && snapshot.pods.length === 1 + if (matched) return { snapshot, elapsedMs: Date.now() - started } + if (Date.now() - started >= timeoutMs) { + throw new Error( + `Timed out waiting for ${STATELESS_HOST}=${desired}: ${JSON.stringify(snapshot)}` + ) + } + await new Promise(resolve => setTimeout(resolve, 2_000)) + } +} + +async function closeElectron(app: ElectronApplication | undefined): Promise { + const child = app?.process() + await app?.close().catch(() => undefined) + if (!child) return + const deadline = Date.now() + 30_000 + while (Date.now() < deadline && child.exitCode === null && child.signalCode === null) { + await new Promise(resolve => setTimeout(resolve, 200)) + } + expect(child.exitCode !== null || child.signalCode !== null).toBe(true) +} + +async function waitForSingleCurrentPod(deploymentName: string): Promise { + const deadline = Date.now() + 180_000 + for (;;) { + const [deployment, podList] = await Promise.all([ + jsonAsync(['-n', 'mcp-host', 'get', `deployment/${deploymentName}`, '-o', 'json']), + jsonAsync(['-n', 'mcp-host', 'get', 'pods', '-l', `app=${deploymentName}`, '-o', 'json']), + ]) + const generation = Number(deployment.metadata?.generation ?? 0) + const observedGeneration = Number(deployment.status?.observedGeneration ?? 0) + const desired = Number(deployment.spec?.replicas ?? -1) + const updated = Number(deployment.status?.updatedReplicas ?? 0) + const ready = Number(deployment.status?.readyReplicas ?? 0) + const available = Number(deployment.status?.availableReplicas ?? 0) + const livePods = (podList.items ?? []).filter((item: Json) => !item.metadata?.deletionTimestamp) + const pod = livePods[0] + const containerStatus = (pod?.status?.containerStatuses ?? []).find( + (status: Json) => status.name === 'mcp-host' + ) + const podReadyCondition = (pod?.status?.conditions ?? []).find( + (condition: Json) => condition.type === 'Ready' + ) + const currentPodReady = + livePods.length === 1 && + pod?.status?.phase === 'Running' && + containerStatus?.ready === true && + podReadyCondition?.status === 'True' + if ( + observedGeneration >= generation && + desired === 1 && + updated === 1 && + ready === 1 && + available === 1 && + currentPodReady + ) { + return + } + if (Date.now() >= deadline) { + throw new Error(`Timed out waiting for one current Ready pod for ${deploymentName}`) + } + await new Promise(resolve => setTimeout(resolve, 2_000)) + } +} + +function readCredentialAnnotation(hostName: string, annotation: string): string { + const value = kubectl([ + '-n', + 'mcp-host', + 'get', + 'secret', + `host-${hostName}-mcp-host-runtime-tokens`, + '-o', + `go-template={{index .metadata.annotations "${annotation}"}}`, + ]).trim() + return value === '' || value === '' ? '' : value +} + +function credentialMetadata(hostName: string): { + revision: string + bootstrapState: string + rolloutRequired: string + issuedAt: string +} { + const metadata = { + revision: readCredentialAnnotation(hostName, RUNTIME_TOKEN_SECRET_REVISION_ANNOTATION), + bootstrapState: readCredentialAnnotation(hostName, RUNTIME_TOKEN_BOOTSTRAP_STATE_ANNOTATION), + rolloutRequired: readCredentialAnnotation(hostName, RUNTIME_TOKEN_ROLLOUT_REQUIRED_ANNOTATION), + issuedAt: readCredentialAnnotation(hostName, RUNTIME_TOKEN_ISSUED_AT_ANNOTATION), + } + if (Object.values(metadata).some(value => value === '')) { + throw new Error(`Incomplete runtime credential metadata for ${hostName}`) + } + return metadata +} + +function deploymentRevisionMetadata(deployment: Json): { + revision: string + restartedAt: string +} { + const annotations = deployment.spec?.template?.metadata?.annotations ?? {} + return { + revision: String(annotations[RUNTIME_TOKEN_REVISION_ANNOTATION] ?? ''), + restartedAt: String(annotations['kubectl.kubernetes.io/restartedAt'] ?? ''), + } +} + +function runtimeStateAllowsPaidTurn( + runtime: ReturnType, + deployment: ReturnType +): boolean { + if (runtime.revision === '' || deployment.revision === '') return false + if (runtime.rolloutRequired !== 'false') return false + + const issuedAt = Date.parse(runtime.issuedAt) + if (!Number.isFinite(issuedAt)) return false + if (deployment.restartedAt !== '') { + const restartedAt = Date.parse(deployment.restartedAt) + if (!Number.isFinite(restartedAt) || restartedAt > issuedAt) return false + } + + if (runtime.bootstrapState === 'consumed') { + return deployment.revision === runtime.revision + } + if (runtime.bootstrapState !== 'fresh') return false + return true +} + +async function settleHccRuntimeIdentity(hostName: string): Promise { + const sinceTime = new Date().toISOString() + kubectl([ + '-n', + 'mcp-host', + 'annotate', + `host/${hostName}`, + `${HCC_RUNTIME_PROBE_ANNOTATION}=${sinceTime}`, + '--overwrite', + ]) + const deadline = Date.now() + 180_000 + for (;;) { + const [deployment, runtime] = await Promise.all([ + jsonAsync(['-n', 'mcp-host', 'get', `deployment/${hostName}`, '-o', 'json']), + (async () => credentialMetadata(hostName))(), + ]) + const deploymentRuntime = deploymentRevisionMetadata(deployment) + if (runtimeStateAllowsPaidTurn(runtime, deploymentRuntime)) { + break + } + if (Date.now() >= deadline) { + throw new Error( + `Timed out waiting for settled HCC runtime state for ${hostName}: ` + + `revision=${runtime.revision || 'missing'}, ` + + `deploymentRevision=${deploymentRuntime.revision || 'missing'}, ` + + `bootstrap=${runtime.bootstrapState || 'missing'}, ` + + `rolloutRequired=${runtime.rolloutRequired || 'missing'}, ` + + `issuedAt=${runtime.issuedAt || 'missing'}, ` + + `restartedAt=${deploymentRuntime.restartedAt || 'missing'}` + ) + } + await new Promise(resolve => setTimeout(resolve, 2_000)) + } + await waitForSingleCurrentPod(hostName) +} + +async function newChat(page: Page, hostName: string): Promise { + await openAgentsPage(page) + await page.getByRole('button', { name: `More actions for ${hostName}`, exact: true }).click() + const menu = page.getByRole('menu') + const newChatAction = menu + .getByRole('button', { name: /^New chat$/ }) + .or(menu.getByRole('menuitem', { name: /^New chat$/ })) + await expect(newChatAction).toBeVisible({ timeout: 15_000 }) + await newChatAction.click() + await expect(page.getByRole('textbox', { name: 'Agent message composer' })).toBeVisible({ + timeout: 30_000, + }) + await expect(page.getByRole('button', { name: /^Switch chat agent$/ })).toContainText(hostName, { + timeout: 30_000, + }) +} + +async function renameByMarker(page: Page, marker: string, title: string): Promise { + await expect(page.getByTestId('agent-response').filter({ hasText: marker })).toBeVisible() + const row = page.locator('.nav-latest-session.active') + await expect(row).toBeVisible({ timeout: 30_000 }) + await row.getByRole('button', { name: /^Session options for / }).click() + await page.getByRole('menuitem', { name: 'Rename', exact: true }).click() + const input = page.getByRole('textbox', { name: 'Rename session', exact: true }) + await input.fill(title) + await input.press('Enter') + await expect(page.getByRole('button', { name: `Open ${title}`, exact: true })).toBeVisible({ + timeout: 30_000, + }) +} + +async function openSession(page: Page, title: string, hostName: string): Promise { + await page.getByRole('button', { name: `Open ${title}`, exact: true }).click() + await expect(page.getByRole('textbox', { name: 'Agent message composer' })).toBeVisible({ + timeout: 30_000, + }) + await expect( + page.getByRole('navigation', { name: 'Chat breadcrumb' }).getByText(hostName, { exact: true }) + ).toBeVisible({ timeout: 30_000 }) +} + +async function assertEffectiveStatelessModel(page: Page): Promise { + const selector = page.getByTestId('model-selector-up') + await expect(selector).toHaveAttribute('data-host-ref', STATELESS_HOST, { timeout: 30_000 }) + await expect(selector).toHaveAttribute('data-provider', EXPECTED_STATELESS_MODEL_PROVIDER, { + timeout: 30_000, + }) + await expect(selector).toHaveAttribute('data-model', EXPECTED_STATELESS_MODEL_NAME, { + timeout: 30_000, + }) + await expect(page.getByTestId('selected-chat-model')).toHaveAttribute( + 'data-model-id', + EXPECTED_STATELESS_MODEL_NAME, + { timeout: 30_000 } + ) +} + +async function probeHostModels( + page: Page, + hostRef: string +): Promise<{ error: string | null; modelCount: number | null }> { + return page.evaluate(async targetHostRef => { + try { + const result = await window.clerum.rpc.getHostModels(targetHostRef, '') + const models = (result as unknown as { models?: unknown[] } | null)?.models + return { error: null, modelCount: Array.isArray(models) ? models.length : null } + } catch (error) { + return { + error: error instanceof Error ? error.message : String(error), + modelCount: null, + } + } + }, hostRef) +} + +async function terminateRunOwnedShellTask( + page: Page, + marker: string +): Promise<'cancelled' | 'response'> { + const response = page.getByTestId('agent-response').filter({ hasText: marker }) + const cancel = page.getByTestId('progress-cancel-btn') + await expect + .poll( + async () => { + if (await response.isVisible().catch(() => false)) return 'response' + if (await cancel.isVisible().catch(() => false)) return 'cancel' + return 'waiting' + }, + { timeout: 60_000, intervals: [250, 500, 1_000, 2_000, 5_000] } + ) + .toMatch(/^(?:response|cancel)$/) + + if (await response.isVisible().catch(() => false)) return 'response' + + await expect(cancel).toBeVisible({ timeout: 10_000 }) + await cancel.click() + await expect(page.locator('.stepper-cancelled-badge')).toBeVisible({ timeout: 30_000 }) + return 'cancelled' +} + +async function sendAndExpect( + page: Page, + prompt: string, + marker: string, + timeout = 180_000 +): Promise { + const composer = page.getByRole('textbox', { name: 'Agent message composer' }) + await composer.fill(prompt) + await expect(page.getByTestId('send-button')).toBeEnabled() + await page.getByTestId('send-button').click() + const response = page.getByTestId('agent-response').filter({ hasText: marker }) + await expect(response).toBeVisible({ timeout }) + await expect(composer).toHaveValue('', { timeout: 30_000 }) + await expect(page.getByTestId('send-button')).toBeDisabled({ timeout: 30_000 }) + await expect(page.getByTestId('send-button')).toHaveAttribute('aria-label', 'Send message', { + timeout: 30_000, + }) +} + +async function waitAuthenticatedWithoutLogin(page: Page): Promise { + await expect(page.locator('.boot-overlay')).toBeHidden({ timeout: 30_000 }) + await expect(page.getByTestId('nav-chat')).toBeVisible({ timeout: 30_000 }) + await expect(page.locator('#email-input')).toBeHidden({ timeout: 30_000 }) +} + +test.skip( + process.env.PR849_CONTROLLED_PARITY !== '1', + 'Set PR849_CONTROLLED_PARITY=1 for this real paid Desktop journey' +) + +test('PR849 controlled parity across pending work, GFS, host switching, and cold wake', async ({}, testInfo) => { + test.setTimeout(35 * 60_000) + if (process.env.QA_RECORDER_CONFIRM_CHAT !== '1') { + throw new Error('Set QA_RECORDER_CONFIRM_CHAT=1; this journey sends real model turns.') + } + if (!KUBE_CONTEXT || !/^clerum-[a-z0-9][a-z0-9-]*-[0-9a-f]{8}$/.test(KUBE_CONTEXT)) { + throw new Error(`Refusing non-branch-owned context: ${KUBE_CONTEXT || ''}`) + } + if (!EXPECTED_STATELESS_MODEL_PROVIDER || !EXPECTED_STATELESS_MODEL_NAME) { + throw new Error( + 'Set E2E_EXPECTED_STATELESS_MODEL_PROVIDER and E2E_EXPECTED_STATELESS_MODEL_NAME so the effective session model cannot drift silently' + ) + } + if ( + process.env.E2E_ALLOW_STATELESS_CADENCE_ACCELERATION !== '1' || + process.env.E2E_GFS_AGENT_A !== STATELESS_HOST + ) { + throw new Error( + `Set E2E_ALLOW_STATELESS_CADENCE_ACCELERATION=1 and E2E_GFS_AGENT_A=${STATELESS_HOST}` + ) + } + await assertAllowedTarget('EXTERNAL_REST_API_BASE_URL', EXTERNAL_REST_API_BASE_URL) + await assertAllowedTarget('RPC_PROXY_BASE_URL', RPC_PROXY_BASE_URL) + requireInheritedMutationLease() + + const runId = `pr849-${Date.now()}-${Math.random().toString(16).slice(2, 8)}` + const statelessTitle = `PR849 S ${runId}` + const statefulTitle = `PR849 F ${runId}` + const statelessMarker = `PR849_S_${runId}` + const statefulMarker = `PR849_F_${runId}` + const statelessCode = `S-${Math.random().toString(16).slice(2, 9).toUpperCase()}` + const statefulCode = `F-${Math.random().toString(16).slice(2, 9).toUpperCase()}` + const statelessDraft = `stateless draft ${runId}` + const statefulDraft = `stateful draft ${runId}` + + let app: ElectronApplication | undefined + let fixtures: AgentGfsFixtures | undefined + let lifecycleMutationActive = false + let beforeStateless: HostSnapshot | undefined + let beforeStateful: HostSnapshot | undefined + let afterStateless: HostSnapshot | undefined + let afterStateful: HostSnapshot | undefined + let cancelReadinessObserver = false + let readinessObserver: Promise | undefined + let firstPage: Page | undefined + let beforeIdentityFileSnapshot: IdentityFileSnapshot | undefined + let afterIdentityFileSnapshot: IdentityFileSnapshot | undefined + let shellTaskActive = false + let shellTaskPage: Page | undefined + let shellTaskTitle = '' + let shellTaskMarker = '' + const runtimeProbeHosts = new Set() + const cleanupErrors: string[] = [] + let primaryError: unknown + const metrics: Json = { + runId, + githubMcpConfigured: false, + approvalPendingAcrossHostSwitch: false, + draftPreservedAcrossHostSwitch: false, + controlledSuspendMs: 0, + hccRecoveryPending: false, + secondLaunchToAuthenticatedMs: 0, + secondLaunchToReadyMs: 0, + secondLaunchToCachedTranscriptMs: 0, + followUpToResponseMs: 0, + effectiveIdentityFileHashesAsserted: false, + identityFileCoverage: 'unknown', + statelessRuntimeModelLogAsserted: false, + initialRuntimeModelProof: null, + postWakeRuntimeModelProof: null, + effectiveStatelessModelAsserted: false, + statelessModelRpcAsserted: false, + statefulModelRpcAsserted: false, + postWakeStatelessModelRpcAsserted: false, + runOwnedShellCleanup: 'not-required', + statelessSessionId: '', + statefulSessionId: '', + reopenedSessionId: '', + statelessRawTemplateHashBefore: '', + statelessRawTemplateHashAfter: '', + statelessRuntimeTokenRevisionBefore: '', + statelessRuntimeTokenRevisionAfter: '', + } + + try { + await test.step('verify policy, seed GFS, and wake through Desktop catalog', async () => { + expect(hccEnv()).toEqual(BASELINE_HCC_ENV) + runtimeProbeHosts.add(STATEFUL_HOST) + await settleHccRuntimeIdentity(STATEFUL_HOST) + beforeStateful = await hostSnapshot(STATEFUL_HOST, STATEFUL_HOST, false) + expect(beforeStateful.readyReplicas).toBe(1) + assertGfsInfraHealthy() + fixtures = seedAgentGfsFixtures(desktopCredentials().email) + expect(fixtures.agent.name).toBe(STATELESS_HOST) + + const launchStarted = Date.now() + const first = await launchDesktopApp(testInfo, 'pr849-controlled-parity') + app = first.app + firstPage = first.page + await login(firstPage, desktopCredentials()) + metrics.firstLaunchToAuthenticatedMs = Date.now() - launchStarted + await openAgentsPage(firstPage) + await expect( + firstPage.getByRole('button', { + name: `More actions for ${STATELESS_HOST}`, + exact: true, + }) + ).toBeVisible({ timeout: 30_000 }) + runtimeProbeHosts.add(STATELESS_HOST) + await settleHccRuntimeIdentity(STATELESS_HOST) + beforeStateless = (await waitForStateless('ready', 270_000)).snapshot + expect(beforeStateless.readyReplicas).toBe(1) + beforeIdentityFileSnapshot = identityFileSnapshot(beforeStateless.pods[0]!) + + const mcpServers = json(['-n', 'mcp-server', 'get', 'mcpservers', '-o', 'json']) + metrics.githubMcpConfigured = (mcpServers.items ?? []).some((item: Json) => + /github/i.test(String(item.metadata?.name ?? '')) + ) + }) + + await test.step('run pending stateless work and switch to the stateful Host', async () => { + const page = firstPage! + + await newChat(page, STATELESS_HOST) + await assertEffectiveStatelessModel(page) + metrics.effectiveStatelessModelAsserted = true + const statelessModels = await probeHostModels(page, STATELESS_HOST) + expect( + statelessModels.error, + `stateless model RPC failed before the first paid turn: ${statelessModels.error}` + ).toBeNull() + expect(statelessModels.modelCount ?? 0).toBeGreaterThan(0) + metrics.statelessModelRpcAsserted = true + const initialModelLogSince = new Date().toISOString() + await sendAndExpect( + page, + `Prepare the continuity case. Remember exactly code=${statelessCode}, owner=Lucia, limit=27. Reply exactly ${statelessMarker}. Do not use a tool yet.`, + statelessMarker + ) + metrics.statelessSessionId = await page.evaluate( + hostRef => window.clerum.chat.getLastActive(hostRef), + STATELESS_HOST + ) + expect(metrics.statelessSessionId).toMatch(/^[0-9a-f-]{36}$/i) + await renameByMarker(page, statelessMarker, statelessTitle) + + const gfsPath = `/${fixtures!.granted.name}/${fixtures!.granted.fileName}` + const sentinel = `E2E GFS file fixture: ${fixtures!.granted.name}` + const gfsAck = `PR849_GFS_${runId}` + const gfsPrompt = + `Read the file at path "${gfsPath}" in GFS drive main. Its resourceId is ` + + `${fixtures!.granted.fileResourceId}. Call clerum__gfs_read directly with drive main and ` + + `that resourceId, then quote its contents verbatim. Begin your final reply with exactly ` + + `${gfsAck}. Do not answer from memory.` + const composer = page.getByRole('textbox', { name: 'Agent message composer' }) + await sendAndExpect(page, gfsPrompt, gfsAck) + + const gfsResponse = page.getByTestId('agent-response').filter({ hasText: gfsAck }) + await expect(gfsResponse).toBeVisible({ timeout: 240_000 }) + await expect(gfsResponse).not.toContainText(/not_mounted|gfsc 503|fetch failed/i) + const expand = gfsResponse.getByTestId('progress-expand-btn') + await expect(expand).toBeVisible({ timeout: 30_000 }) + await expand.click() + const readSteps = gfsResponse + .locator('.stepper-step') + .filter({ has: page.locator('.stepper-step-fn', { hasText: 'gfs_read' }) }) + let openedGfsReadStep = false + for (const readStep of await readSteps.all()) { + await readStep.click() + openedGfsReadStep = true + } + expect(openedGfsReadStep, 'the turn must contain a gfs_read step').toBe(true) + await expect + .poll( + () => + readSteps.evaluateAll( + (steps, expected) => + steps.some(step => { + if ( + !step.querySelector('.stepper-step-icon.state-completed') || + step.querySelector('.stepper-step-icon.state-error') + ) { + return false + } + let sibling = step.nextElementSibling + while ( + sibling && + !sibling.classList.contains('stepper-step') && + !sibling.classList.contains('stepper-iteration-divider') + ) { + const output = sibling.matches('[data-testid="step-output-panel"]') + ? sibling.querySelector('.stepper-step-output-code') + : sibling.querySelector( + '[data-testid="step-output-panel"] .stepper-step-output-code' + ) + if (output?.textContent?.includes(expected) === true) return true + sibling = sibling.nextElementSibling + } + return false + }), + sentinel + ), + { timeout: 30_000 } + ) + .toBe(true) + metrics.initialRuntimeModelProof = proveStatelessRuntimeModel( + beforeStateless!.pods[0]!, + initialModelLogSince + ) + metrics.statelessRuntimeModelLogAsserted = true + + const shellMarker = `PR849_SHELL_${runId}` + const shellCommand = `printf '${shellMarker}'` + const shellPrompt = + `Use the shell tool to run exactly this command: ${shellCommand}. ` + + `Then reply with exactly ${shellMarker}.` + await composer.fill(shellPrompt) + await page.getByTestId('send-button').click() + shellTaskActive = true + shellTaskPage = page + shellTaskTitle = statelessTitle + shellTaskMarker = shellMarker + await expect(page.getByTestId('message-list')).toContainText(shellCommand, { + timeout: 30_000, + }) + const approvalStepper = page + .getByTestId('progress-stepper') + .filter({ hasText: /Shell.*requires approval/i }) + const approval = approvalStepper.getByTestId('approval-approve-btn') + await expect(approval).toBeVisible({ timeout: 180_000 }) + await expect(approvalStepper).toContainText(/Shell.*requires approval/i) + + await newChat(page, STATEFUL_HOST_DISPLAY) + const statefulSelector = page.getByTestId('model-selector-up') + await expect(statefulSelector).toHaveAttribute('data-provider', 'openai', { + timeout: 30_000, + }) + await expect(statefulSelector).toHaveAttribute('data-model', 'gpt-5.4-mini', { + timeout: 30_000, + }) + const statefulModels = await probeHostModels(page, STATEFUL_HOST) + expect( + statefulModels.error, + `stateful model RPC failed before the control turn: ${statefulModels.error}` + ).toBeNull() + expect(statefulModels.modelCount ?? 0).toBeGreaterThan(0) + metrics.statefulModelRpcAsserted = true + await expect(page.getByTestId('message-list')).not.toContainText(gfsPath) + await expect(page.getByTestId('message-list')).not.toContainText(shellCommand) + await sendAndExpect( + page, + `Prepare the control case. Remember exactly code=${statefulCode}, owner=Mateo, limit=14. Reply exactly ${statefulMarker}.`, + statefulMarker + ) + metrics.statefulSessionId = await page.evaluate( + hostRef => window.clerum.chat.getLastActive(hostRef), + STATEFUL_HOST + ) + expect(metrics.statefulSessionId).not.toBe(metrics.statelessSessionId) + await renameByMarker(page, statefulMarker, statefulTitle) + + await openSession(page, statelessTitle, STATELESS_HOST) + await expect(page.getByTestId('message-list')).toContainText(gfsPath) + await expect(page.getByTestId('message-list')).toContainText(shellCommand) + await expect(approval).toBeVisible({ + timeout: 30_000, + }) + metrics.approvalPendingAcrossHostSwitch = true + await approval.click() + const shellResponse = page.getByTestId('agent-response').filter({ hasText: shellMarker }) + await expect(shellResponse).toBeVisible({ timeout: 240_000 }) + await expect(approval).toBeHidden({ timeout: 30_000 }) + await expect(composer).toHaveValue('', { timeout: 30_000 }) + await expect(page.getByTestId('send-button')).toBeDisabled({ timeout: 30_000 }) + + const shellExpand = shellResponse.getByTestId('progress-expand-btn') + await expect(shellExpand).toBeVisible({ timeout: 30_000 }) + await shellExpand.click() + const shellSteps = shellResponse + .locator('.stepper-step') + .filter({ has: page.locator('.stepper-step-fn', { hasText: 'shell_exec' }) }) + let openedShellStep = false + for (const shellStep of await shellSteps.all()) { + await shellStep.click() + openedShellStep = true + } + expect(openedShellStep, 'the turn must contain a shell_exec step').toBe(true) + await expect + .poll( + () => + shellSteps.evaluateAll( + (steps, expected) => + steps.some(step => { + if ( + !step.querySelector('.stepper-step-icon.state-completed') || + step.querySelector('.stepper-step-icon.state-error') + ) { + return false + } + let sibling = step.nextElementSibling + let inputMatches = false + let outputMatches = false + while ( + sibling && + !sibling.classList.contains('stepper-step') && + !sibling.classList.contains('stepper-iteration-divider') + ) { + const input = sibling.matches('[data-testid="step-input-preview"]') + ? sibling + : sibling.querySelector('[data-testid="step-input-preview"]') + const output = sibling.matches('[data-testid="step-output-panel"]') + ? sibling.querySelector('.stepper-step-output-code') + : sibling.querySelector( + '[data-testid="step-output-panel"] .stepper-step-output-code' + ) + if (input?.textContent?.includes(expected.command) === true) { + inputMatches = true + } + if (output?.textContent?.includes(expected.marker) === true) { + outputMatches = true + } + sibling = sibling.nextElementSibling + } + return inputMatches && outputMatches + }), + { command: shellCommand, marker: shellMarker } + ), + { timeout: 30_000 } + ) + .toBe(true) + shellTaskActive = false + shellTaskPage = undefined + + await composer.fill(statelessDraft) + await openSession(page, statefulTitle, STATEFUL_HOST_DISPLAY) + await expect(composer).toHaveValue('') + await composer.fill(statefulDraft) + await openSession(page, statelessTitle, STATELESS_HOST) + await expect(composer).toHaveValue(statelessDraft) + await openSession(page, statefulTitle, STATEFUL_HOST_DISPLAY) + await expect(composer).toHaveValue(statefulDraft) + metrics.draftPreservedAcrossHostSwitch = true + }) + + await test.step('suspend through HCC and restore the 30-minute product policy', async () => { + await closeElectron(app) + app = undefined + lifecycleMutationActive = true + metrics.hccRecoveryPending = true + setHccEnv(ACCELERATED_HCC_ENV) + const suspended = await waitForStateless('suspended', 300_000) + metrics.controlledSuspendMs = suspended.elapsedMs + expect(suspended.snapshot.pods).toHaveLength(0) + setHccEnv(BASELINE_HCC_ENV) + lifecycleMutationActive = false + metrics.hccRecoveryPending = false + metrics.hccBaselineRestored = true + }) + + await test.step('reopen the same isolated profile and continue after wake', async () => { + const relaunchBefore = await hostSnapshot(STATELESS_HOST, STATELESS_HOST, true) + expect(relaunchBefore.lifecycle.state).toBe('suspended') + expect(relaunchBefore.pods).toHaveLength(0) + let readyObservedAt: number | null = null + let readinessFailure: unknown + const launchStarted = Date.now() + const secondLaunch = launchDesktopApp(testInfo, 'pr849-controlled-parity') + readinessObserver = waitForStateless('ready', 270_000, () => cancelReadinessObserver) + .then(snapshot => { + readyObservedAt = Date.now() + afterStateless = snapshot.snapshot + return snapshot + }) + .catch(error => { + readinessFailure = error + return undefined + }) + const second = await secondLaunch + app = second.app + const page = second.page + await waitAuthenticatedWithoutLogin(page) + metrics.secondLaunchToAuthenticatedMs = Date.now() - launchStarted + await openAgentsPage(page) + await expect( + page.getByRole('button', { name: `More actions for ${STATELESS_HOST}`, exact: true }) + ).toBeVisible({ timeout: 30_000 }) + + await openSession(page, statelessTitle, STATELESS_HOST) + await expect(page.getByTestId('message-list')).toContainText(statelessCode) + const retainedMarker = page.getByTestId('agent-response').filter({ hasText: statelessMarker }) + const retainedGfs = page + .getByTestId('agent-response') + .filter({ hasText: `PR849_GFS_${runId}` }) + await expect(retainedMarker).toBeVisible({ timeout: 90_000 }) + await expect(retainedGfs).toBeVisible({ timeout: 90_000 }) + metrics.secondLaunchToCachedTranscriptMs = Date.now() - launchStarted + const cachedVisibleAt = Date.now() + await readinessObserver + if (readinessFailure) throw readinessFailure + if (!readyObservedAt || !afterStateless) { + throw new Error(`${STATELESS_HOST} readiness observer completed without a Ready snapshot`) + } + afterStateful = await hostSnapshot(STATEFUL_HOST, STATEFUL_HOST, false) + metrics.secondLaunchToReadyMs = readyObservedAt - launchStarted + metrics.cacheVisibleBeforeReadyObserved = + readyObservedAt !== null && cachedVisibleAt < readyObservedAt + await assertEffectiveStatelessModel(page) + metrics.effectiveStatelessModelAsserted = true + const postWakeModels = await probeHostModels(page, STATELESS_HOST) + expect( + postWakeModels.error, + `stateless model RPC failed after wake: ${postWakeModels.error}` + ).toBeNull() + expect(postWakeModels.modelCount ?? 0).toBeGreaterThan(0) + metrics.postWakeStatelessModelRpcAsserted = true + metrics.reopenedSessionId = await page.evaluate( + hostRef => window.clerum.chat.getLastActive(hostRef), + STATELESS_HOST + ) + expect(metrics.reopenedSessionId).toBe(metrics.statelessSessionId) + + const followStarted = Date.now() + const postWakeModelLogSince = new Date().toISOString() + await sendAndExpect( + page, + 'Continue the earlier continuity case. From the first turn only, return exactly three labeled fields in order: code=..., owner=..., limit=... Do not repeat control-case values.', + statelessCode, + 240_000 + ) + metrics.followUpToResponseMs = Date.now() - followStarted + const follow = page.getByTestId('agent-response').filter({ hasText: statelessCode }) + await expect(follow).toContainText(new RegExp(`code\\s*[:=]\\s*${statelessCode}`, 'i')) + await expect(follow).toContainText(/owner\s*[:=]\s*Lucia/i) + await expect(follow).toContainText(/limit\s*[:=]\s*27/i) + await expect(follow).not.toContainText(statefulCode) + await expect(follow).not.toContainText('Mateo') + metrics.postWakeRuntimeModelProof = proveStatelessRuntimeModel( + afterStateless!.pods[0]!, + postWakeModelLogSince + ) + metrics.statelessRuntimeModelLogAsserted = true + + await openSession(page, statefulTitle, STATEFUL_HOST_DISPLAY) + await expect( + page.getByTestId('agent-response').filter({ hasText: statefulMarker }) + ).toBeVisible() + await expect(page.getByTestId('message-list')).not.toContainText(statelessCode) + }) + + const checks = { + hostUidStable: afterStateless!.hostUid === beforeStateless!.hostUid, + templateStable: afterStateless!.templateHash === beforeStateless!.templateHash, + rawTemplateChanged: afterStateless!.rawTemplateHash !== beforeStateless!.rawTemplateHash, + runtimeTokenRevisionChanged: + afterStateless!.runtimeTokenRevision !== beforeStateless!.runtimeTokenRevision, + declaredParametersStable: afterStateless!.parameterHash === beforeStateless!.parameterHash, + pvcStable: afterStateless!.pvcUid === beforeStateless!.pvcUid, + sqliteStable: + afterStateless!.claimName === beforeStateless!.claimName && + afterStateless!.mountPath === beforeStateless!.mountPath && + afterStateless!.sqliteDir === beforeStateless!.sqliteDir, + imageStable: afterStateless!.imageIds.join(',') === beforeStateless!.imageIds.join(','), + statelessPodChanged: afterStateless!.podUids.join(',') !== beforeStateless!.podUids.join(','), + statefulPodStable: afterStateful!.podUids.join(',') === beforeStateful!.podUids.join(','), + } + afterIdentityFileSnapshot = identityFileSnapshot(afterStateless!.pods[0]!) + expect(afterIdentityFileSnapshot).toEqual(beforeIdentityFileSnapshot) + const identityAndSoulNonEmpty = (['IDENTITY.md', 'SOUL.md'] as const).every( + (file: IdentityFileName) => + beforeIdentityFileSnapshot?.[file].exists === true && + beforeIdentityFileSnapshot[file].sha256 !== EMPTY_FILE_SHA256 + ) + const anyIdentityFilePresent = Object.values(beforeIdentityFileSnapshot ?? {}).some( + state => state.exists + ) + metrics.identityFileCoverage = !anyIdentityFilePresent + ? 'empty-config-only' + : identityAndSoulNonEmpty + ? 'nonempty-config' + : 'empty-or-partial-config' + if (REQUIRE_IDENTITY_FILES) { + expect(metrics.identityFileCoverage).toBe('nonempty-config') + } + metrics.effectiveIdentityFileHashesAsserted = true + Object.assign(metrics, checks) + expect(checks).toEqual({ + hostUidStable: true, + templateStable: true, + rawTemplateChanged: true, + runtimeTokenRevisionChanged: true, + declaredParametersStable: true, + pvcStable: true, + sqliteStable: true, + imageStable: true, + statelessPodChanged: true, + statefulPodStable: true, + }) + metrics.statelessRawTemplateHashBefore = beforeStateless!.rawTemplateHash + metrics.statelessRawTemplateHashAfter = afterStateless!.rawTemplateHash + metrics.statelessRuntimeTokenRevisionBefore = beforeStateless!.runtimeTokenRevision + metrics.statelessRuntimeTokenRevisionAfter = afterStateless!.runtimeTokenRevision + } catch (error) { + primaryError = error + throw error + } finally { + cancelReadinessObserver = true + await readinessObserver + for (const hostName of runtimeProbeHosts) { + try { + kubectl([ + '-n', + 'mcp-host', + 'annotate', + `host/${hostName}`, + `${HCC_RUNTIME_PROBE_ANNOTATION}-`, + ]) + } catch (error) { + cleanupErrors.push( + `HCC runtime probe cleanup ${hostName}: ${ + error instanceof Error ? error.message : String(error) + }` + ) + } + } + if (shellTaskActive && shellTaskPage && app) { + try { + await openSession(shellTaskPage, shellTaskTitle, STATELESS_HOST) + metrics.runOwnedShellCleanup = await terminateRunOwnedShellTask( + shellTaskPage, + shellTaskMarker + ) + shellTaskActive = false + } catch (error) { + cleanupErrors.push( + `Run-owned Shell task termination: ${ + error instanceof Error ? error.message : String(error) + }` + ) + } + } + try { + await closeElectron(app) + } catch (error) { + cleanupErrors.push( + `Electron close: ${error instanceof Error ? error.message : String(error)}` + ) + } + if (lifecycleMutationActive) { + try { + setHccEnv(BASELINE_HCC_ENV) + lifecycleMutationActive = false + metrics.hccRecoveryPending = false + metrics.hccBaselineRestored = true + } catch (error) { + cleanupErrors.push( + `HCC baseline restoration: ${error instanceof Error ? error.message : String(error)}` + ) + } + } + try { + fixtures?.cleanup() + } catch { + cleanupErrors.push('GFS fixture cleanup failed') + } + metrics.cleanupErrors = cleanupErrors + await testInfo.attach('pr849-controlled-parity-metrics', { + body: JSON.stringify(metrics, null, 2), + contentType: 'application/json', + }) + console.log(`[PR849ControlledParity] ${JSON.stringify(metrics)}`) + if (cleanupErrors.length > 0) { + await testInfo.attach('pr849-controlled-parity-cleanup-errors', { + body: cleanupErrors.join('\n'), + contentType: 'text/plain', + }) + if (!primaryError) throw new Error(cleanupErrors.join('; ')) + } + } +}) diff --git a/desktop-app/test/ipc.test.ts b/desktop-app/test/ipc.test.ts index 7802f9f10a..98940595bf 100644 --- a/desktop-app/test/ipc.test.ts +++ b/desktop-app/test/ipc.test.ts @@ -1,6 +1,13 @@ import { beforeEach, describe, expect, it, vi } from 'vitest' import { promises as fs } from 'node:fs' import path from 'node:path' +import { + __setChatStoreBaseDirForTests, + bindChatStoreForUser, + getChatStoreBindingGeneration, + requireChatStore, + unbindChatStore, +} from '../src/chatStoreBinding.js' import { registerIpcHandlers, sanitizeChatLoadWindow } from '../src/ipc.js' import type { HostStatusStreamEvent } from '../src/types.js' @@ -104,6 +111,7 @@ describe('ipc host status stream handlers', () => { listArtifacts: vi.fn(), downloadArtifact: vi.fn(), listSessions: vi.fn(), + getChatDeletionFenceAuthority: vi.fn(), loadSessionMessages: vi.fn(), renameSession: vi.fn(), listWorkflowRuns: vi.fn(), @@ -132,10 +140,76 @@ describe('ipc host status stream handlers', () => { vi.clearAllMocks() await fs.rm(testState.downloadsDir, { recursive: true, force: true }) await fs.mkdir(testState.downloadsDir, { recursive: true }) + unbindChatStore() + __setChatStoreBaseDirForTests(path.join(testState.downloadsDir, 'chat-cache')) + service.getChatDeletionFenceAuthority.mockReturnValue({ + authorityScope: { environmentKey: 'env-1', userId: 'user-1', teamId: 'team-a' }, + sessionGeneration: 7, + }) service.stopHostStatusStream.mockReturnValue(true) registerIpcHandlers(service as never) }) + it('rejects legacy chat deletion without a binding fence', async () => { + const handler = testState.handlers.get('chat:delete') + expect(handler).toBeDefined() + const { event } = makeTrustedEvent() + await expect( + Promise.resolve(handler?.(event, { agentRef: 'agent-x', chatId: 'chat-1' })) + ).rejects.toThrow('Scoped chat deletion is required') + }) + + it('captures and rechecks the main-owned authority and binding fence', async () => { + await bindChatStoreForUser('user-1', 'env-1', { teamId: 'team-a' }) + const { event } = makeTrustedEvent() + const capture = testState.handlers.get('chat:captureDeleteFence') + const remove = testState.handlers.get('chat:delete') + const expectedAuthorityScope = { + environmentKey: 'env-1', + userId: 'user-1', + teamId: 'team-a', + } + const fence = await Promise.resolve(capture?.(event, { expectedAuthorityScope })) + expect(fence).toMatchObject({ + version: 1, + authorityScope: expectedAuthorityScope, + bindingGeneration: getChatStoreBindingGeneration(), + sessionGeneration: 7, + }) + + // Returning to the same team does not revive the old confirmation: the + // auth/session epoch changed during the intervening team switch. + service.getChatDeletionFenceAuthority.mockReturnValue({ + authorityScope: expectedAuthorityScope, + sessionGeneration: 9, + }) + await expect( + Promise.resolve(remove?.(event, { version: 3, agentRef: 'agent-x', chatId: 'c1', fence })) + ).rejects.toThrow('Chat deletion authority changed before confirmation') + }) + + it('persists a scoped tombstone before returning from the fenced delete IPC', async () => { + await bindChatStoreForUser('user-1', 'env-1', { teamId: 'team-a' }) + const { event } = makeTrustedEvent() + const capture = testState.handlers.get('chat:captureDeleteFence') + const remove = testState.handlers.get('chat:delete') + const authorityScope = { + environmentKey: 'env-1', + userId: 'user-1', + teamId: 'team-a', + } + const fence = await Promise.resolve( + capture?.(event, { expectedAuthorityScope: authorityScope }) + ) + const result = await Promise.resolve( + remove?.(event, { version: 3, agentRef: 'agent-x', chatId: 'c1', fence }) + ) + expect(result).toEqual({ cleanupPending: false }) + const index = await requireChatStore().getIndex('agent-x') + expect(index.deletedChatTombstones).toContainEqual({ chatId: 'c1', authorityScope }) + expect(index.pendingChatCleanup).toEqual([]) + }) + it('rejects untrusted sender for stream start', async () => { const handler = testState.handlers.get('rpc:hostStatusStreamStart') expect(handler).toBeDefined() diff --git a/desktop-app/test/rpcProxyClient.test.ts b/desktop-app/test/rpcProxyClient.test.ts index 5990c4e957..2a0b8a3e89 100644 --- a/desktop-app/test/rpcProxyClient.test.ts +++ b/desktop-app/test/rpcProxyClient.test.ts @@ -207,6 +207,58 @@ describe('RpcProxyClient — openTaskProgressStream()', () => { }) describe('RpcProxyClient — renameSession() (spec 15 Fase B)', () => { + it.each([ + { + status: 403, + body: JSON.stringify({ + error: 'Forbidden: user cannot access this host', + code: 'host_access_revoked', + }), + marked: true, + }, + { + status: 403, + body: JSON.stringify({ + error: 'Forbidden: user cannot access this host', + code: 'host_access_denied', + }), + marked: false, + }, + { + status: 403, + body: JSON.stringify({ error: 'Forbidden: user cannot access this host' }), + marked: false, + }, + { status: 403, body: JSON.stringify({ error: 'host_access_revoked' }), marked: false }, + { + status: 403, + body: JSON.stringify({ error: 'Forbidden: missing rename scope' }), + marked: false, + }, + { status: 403, body: 'Forbidden: user cannot access this host', marked: false }, + { + status: 503, + body: JSON.stringify({ + error: 'Forbidden: user cannot access this host', + code: 'host_access_revoked', + }), + marked: false, + }, + ])( + 'marks Host-wide revocation only for a 403 whose body carries code host_access_revoked ($status, $marked)', + async ({ status, body, marked }) => { + vi.stubGlobal('fetch', vi.fn().mockResolvedValue(new Response(body, { status }))) + const client = new RpcProxyClient() + const error = await client + .renameSession('rpc-token', 'chatllm', 'chatllm', 'chat-1', 'name') + .catch((caught: Error) => caught) + expect(error).toBeInstanceOf(Error) + // Liveness witness: the request reached the HTTP-error path and reported its status. + expect((error as Error).message).toContain(String(status)) + expect((error as Error).message.includes('host_access_revoked')).toBe(marked) + } + ) + it('PATCHes the name route with the title and returns the (re-sanitized) server title', async () => { const fetchMock = vi .fn() diff --git a/desktop-app/ui/src/App.tsx b/desktop-app/ui/src/App.tsx index 14be2aca84..61b024077f 100644 --- a/desktop-app/ui/src/App.tsx +++ b/desktop-app/ui/src/App.tsx @@ -318,6 +318,15 @@ export function App() { // The universal tab store lives in the controller (single writer; `vm.navItem` // is its projection). App composes the strip/reveal/reconcile over it. const workspaceTabs = vm.workspaceTabs + const visibleWorkspaceTabs = React.useMemo( + () => + workspaceTabs.tabs.map(tab => + tab.kind === 'chat' && tab.chat?.agentRef && vm.isHostAccessBlocked(tab.chat.agentRef) + ? { ...tab, title: 'Conversation unavailable' } + : tab + ), + [workspaceTabs.tabs, vm.hostAuthorityRevision, vm.isHostAccessBlocked] + ) const setWorkspaceTabs = vm.setWorkspaceTabs const nextChatTabId = vm.nextWorkspaceTabId const workspaceTabsRef = React.useRef(workspaceTabs) @@ -2204,8 +2213,12 @@ export function App() { const notifValue = React.useMemo( () => ({ - notifications: vm.notifications, - unreadNotificationCount: vm.unreadNotificationCount, + notifications: vm.notifications.filter( + notification => !vm.isHostAccessBlocked(notification.agentName) + ), + unreadNotificationCount: vm.notifications.filter( + notification => !vm.isHostAccessBlocked(notification.agentName) && !notification.read + ).length, notificationActionById: vm.notificationActionById, pendingApprovals: vm.pendingApprovals, pendingApprovalsLoading: vm.pendingApprovalsLoading, @@ -2233,13 +2246,14 @@ export function App() { vm.markNotificationsRead, vm.notificationActionById, vm.notifications, + vm.isHostAccessBlocked, + vm.hostAuthorityRevision, vm.pendingApprovalActionId, vm.pendingApprovals, vm.pendingApprovalsLoading, vm.removeNotification, vm.resolveApprovalNotification, vm.toasts, - vm.unreadNotificationCount, ] ) @@ -2437,8 +2451,8 @@ export function App() { // Its "current" chat is the controller's (selectedAgent, activeChatId) — NOT // the store's active tab, which is the app tab while the drawer is open. const chatWorkspaceTabs = React.useMemo( - () => workspaceTabs.tabs.filter((tab): tab is WorkspaceTab => tab.kind === 'chat'), - [workspaceTabs.tabs] + () => visibleWorkspaceTabs.filter((tab): tab is WorkspaceTab => tab.kind === 'chat'), + [visibleWorkspaceTabs] ) const drawerActiveChatTabId = chatWorkspaceTabs.find( @@ -2540,7 +2554,7 @@ export function App() { tab. Hidden only when the workspace is empty. */} {workspaceTabs.tabs.length > 0 && ( ({ // Captured from context so tests can drive the "open conversation" gesture the // notification tray fires. openNotification: null as null | ((notification: AppNotification) => Promise), + notifications: [] as AppNotification[], + unreadNotificationCount: 0, })) vi.mock('@hooks/useAppController', () => ({ useAppController: vi.fn() })) @@ -100,7 +102,10 @@ vi.mock('@hooks/useAgentChatActionsValue', () => ({ useAgentChatActionsValue: () vi.mock('@components/AppHeader', () => ({ AppHeader: (props: NonNullable) => { appHeaderHarness.props = props - appHeaderHarness.openNotification = useNotificationsContext().handleOpenNotification + const notificationsContext = useNotificationsContext() + appHeaderHarness.openNotification = notificationsContext.handleOpenNotification + appHeaderHarness.notifications = notificationsContext.notifications + appHeaderHarness.unreadNotificationCount = notificationsContext.unreadNotificationCount return null }, })) @@ -341,6 +346,7 @@ function makeController(overrides: Partial = {}): AppController { authTransitioning: false, handleEnsureTeamContext: vi.fn(async () => false), getCurrentTeamId: vi.fn(() => 'team-a'), + isHostAccessBlocked: vi.fn(() => false), handleSelectChatAgent, handleOpenNotification, handleNavSelect, @@ -405,6 +411,8 @@ describe('App chat drawer — reopen preserves the last-viewed chat', () => { sandboxUiPageHarness.props = null appHeaderHarness.props = null appHeaderHarness.openNotification = null + appHeaderHarness.notifications = [] + appHeaderHarness.unreadNotificationCount = 0 currentController = makeController() vi.mocked(useAppController).mockImplementation(() => useReactiveController(currentController)) Object.defineProperty(window, 'clerum', { @@ -477,6 +485,40 @@ describe('App chat drawer — reopen preserves the last-viewed chat', () => { expect(screen.getByRole('button', { name: 'Open chats' }).textContent).toContain('Second chat') }) + it('hides already-delivered notifications after a Host is revoked', () => { + const notification = { + id: 'approval-1', + kind: 'approval_required', + agentName: 'alpha', + text: 'Approval required', + timestamp: 1, + read: false, + } as AppNotification + let hostBlocked = false + let hostAuthorityRevision = 0 + const isHostAccessBlocked = vi.fn(() => hostBlocked) + currentController = makeController({ + notifications: [notification], + isHostAccessBlocked, + hostAuthorityRevision, + } as Partial) + + const { rerender } = render() + + expect(appHeaderHarness.notifications).toEqual([notification]) + expect(appHeaderHarness.unreadNotificationCount).toBe(1) + + act(() => { + hostBlocked = true + hostAuthorityRevision += 1 + currentController.hostAuthorityRevision = hostAuthorityRevision + rerender() + }) + + expect(appHeaderHarness.notifications).toEqual([]) + expect(appHeaderHarness.unreadNotificationCount).toBe(0) + }) + // The header's "Open chat in full screen" CTA must EJECT the drawer's ACTIVE // conversation — the one the user switched to inside the drawer, not the launch // origin — to the full-screen chat route. That is the non-drawer branch of diff --git a/desktop-app/ui/src/__tests__/App.deepLinks.test.tsx b/desktop-app/ui/src/__tests__/App.deepLinks.test.tsx index a521a80088..64dda1896e 100644 --- a/desktop-app/ui/src/__tests__/App.deepLinks.test.tsx +++ b/desktop-app/ui/src/__tests__/App.deepLinks.test.tsx @@ -320,6 +320,7 @@ function makeController(overrides: Partial = {}): AppController { authTransitioning: false, handleEnsureTeamContext: ensureTeamContext, getCurrentTeamId: vi.fn(() => liveTeamId), + isHostAccessBlocked: vi.fn(() => false), handleSelectChatAgent, handleNavSelect, handleLogout: vi.fn(), diff --git a/desktop-app/ui/src/components/SidebarNav/index.tsx b/desktop-app/ui/src/components/SidebarNav/index.tsx index 575bf40cb9..4e372dc111 100644 --- a/desktop-app/ui/src/components/SidebarNav/index.tsx +++ b/desktop-app/ui/src/components/SidebarNav/index.tsx @@ -1,5 +1,8 @@ import React, { useEffect, useMemo, useRef, useState } from 'react' -import { useAgentChatActionsContext } from '@contexts/AgentChatActionsContext' +import { + type QueuedChatDelete, + useAgentChatActionsContext, +} from '@contexts/AgentChatActionsContext' import { makeTaskKey } from '@contexts/AgentTaskTrackerContext/types' import { useAuthContext } from '@contexts/AuthContext' import { useChatListContext } from '@contexts/ChatListContext' @@ -53,6 +56,7 @@ export function SidebarNav({ const { activeChatId, latestChatSessions, latestChatSessionsLoading, sessionStateByChatKey } = useChatListContext() const { + captureChatDeleteFence, handleRenameChatForAgent: onRenameChatForAgent, handleDeleteChatForAgent: onDeleteChatForAgent, } = useAgentChatActionsContext() @@ -71,6 +75,7 @@ export function SidebarNav({ agentRef: string chatId: string title: string + deletion: QueuedChatDelete } | null>(null) const lastToggleRequestIdRef = useRef(0) const desktopAppInfo = useDesktopAppInfo() @@ -88,6 +93,24 @@ export function SidebarNav({ [onSettingsMenuOpenChange] ) + const requestDeleteSession = React.useCallback( + async (session: { agentRef: string; id: string; title: string }) => { + setSessionMenuId(null) + try { + const deletion = await captureChatDeleteFence(session.agentRef) + setPendingDeleteSession({ + agentRef: session.agentRef, + chatId: session.id, + title: session.title, + deletion, + }) + } catch { + // captureChatDeleteFence reports the failure through the shared chat toast. + } + }, + [captureChatDeleteFence, pushToast] + ) + useClickOutside(settingsMenuRef, settingsMenuOpen, () => { updateSettingsMenuOpen(false) }) @@ -503,14 +526,7 @@ export function SidebarNav({ role="menuitem" color="danger" className="danger" - onClick={() => { - setSessionMenuId(null) - setPendingDeleteSession({ - agentRef: session.agentRef, - chatId: session.id, - title: session.title, - }) - }} + onClick={() => void requestDeleteSession(session)} > Delete @@ -749,7 +765,11 @@ export function SidebarNav({ tone="danger" onCancel={() => setPendingDeleteSession(null)} onConfirm={() => { - void onDeleteChatForAgent(pendingDeleteSession.agentRef, pendingDeleteSession.chatId) + void onDeleteChatForAgent( + pendingDeleteSession.agentRef, + pendingDeleteSession.chatId, + pendingDeleteSession.deletion + ) setPendingDeleteSession(null) }} /> diff --git a/desktop-app/ui/src/components/__tests__/SidebarNav.test.tsx b/desktop-app/ui/src/components/__tests__/SidebarNav.test.tsx index ce720c7e13..8b07b7d711 100644 --- a/desktop-app/ui/src/components/__tests__/SidebarNav.test.tsx +++ b/desktop-app/ui/src/components/__tests__/SidebarNav.test.tsx @@ -24,6 +24,9 @@ vi.mock('@contexts/ChatListContext', () => ({ })) vi.mock('@contexts/AgentChatActionsContext', () => ({ useAgentChatActionsContext: () => ({ + captureChatDeleteFence: vi.fn(async () => { + throw new Error('no delete target') + }), handleRenameChatForAgent: vi.fn(), handleDeleteChatForAgent: vi.fn(), }), diff --git a/desktop-app/ui/src/components/agents/ChatThread.tsx b/desktop-app/ui/src/components/agents/ChatThread.tsx index c272bc721f..67a5b75f4e 100644 --- a/desktop-app/ui/src/components/agents/ChatThread.tsx +++ b/desktop-app/ui/src/components/agents/ChatThread.tsx @@ -7,7 +7,10 @@ import { useRef, useState, } from 'react' -import { useAgentChatActionsContext } from '@contexts/AgentChatActionsContext' +import { + type QueuedChatDelete, + useAgentChatActionsContext, +} from '@contexts/AgentChatActionsContext' import { useChatListContext } from '@contexts/ChatListContext' import { useChatThreadStateContext } from '@contexts/ChatThreadStateContext' import { useMcpRuntimeContext } from '@contexts/McpRuntimeContext' @@ -239,6 +242,7 @@ export function ChatThread({ showAgentLabel = false, onScrollPositionChange }: C chatEndRef, handleSelectChat: onSelectChat, handleRenameChat: onRenameChat, + captureChatDeleteFence, handleDeleteChat: onDeleteChat, } = useAgentChatActionsContext() const { cancelTask: onCancelTask } = useMcpRuntimeContext() @@ -249,8 +253,10 @@ export function ChatThread({ showAgentLabel = false, onScrollPositionChange }: C const [renamingSessionId, setRenamingSessionId] = useState(null) const [sessionRenameValue, setSessionRenameValue] = useState('') const [pendingDeleteSession, setPendingDeleteSession] = useState<{ + agentRef: string id: string title: string + deletion: QueuedChatDelete } | null>(null) const dedicatedSessionsRef = useRef(null) const sessionRenameInputRef = useRef(null) @@ -406,16 +412,29 @@ export function ChatThread({ showAgentLabel = false, onScrollPositionChange }: C cancelSessionRename() }, [cancelSessionRename, onRenameChat, renamingSessionId, sessionRenameValue]) - const deleteSession = useCallback((chatId: string, chatTitle: string) => { - setSessionMenuChatId(null) - setPendingDeleteSession({ id: chatId, title: chatTitle }) - }, []) + const deleteSession = useCallback( + async (chatId: string, chatTitle: string) => { + setSessionMenuChatId(null) + if (!selectedAgent) return + try { + const deletion = await captureChatDeleteFence(selectedAgent) + setPendingDeleteSession({ agentRef: selectedAgent, id: chatId, title: chatTitle, deletion }) + } catch { + // Capture reports the scope/authority failure through the normal chat toast path. + } + }, + [captureChatDeleteFence, selectedAgent] + ) const confirmDeleteSession = useCallback(() => { if (!pendingDeleteSession) return - void onDeleteChat(pendingDeleteSession.id) + if (selectedAgent !== pendingDeleteSession.agentRef) { + setPendingDeleteSession(null) + return + } + void onDeleteChat(pendingDeleteSession.id, pendingDeleteSession.deletion) setPendingDeleteSession(null) - }, [onDeleteChat, pendingDeleteSession]) + }, [onDeleteChat, pendingDeleteSession, selectedAgent]) const handleCopyMessage = useCallback(async (messageKey: string, content: string) => { try { diff --git a/desktop-app/ui/src/components/agents/ComposerPanel.tsx b/desktop-app/ui/src/components/agents/ComposerPanel.tsx index 318169a323..f455c01d21 100644 --- a/desktop-app/ui/src/components/agents/ComposerPanel.tsx +++ b/desktop-app/ui/src/components/agents/ComposerPanel.tsx @@ -162,7 +162,7 @@ export function ComposerPanel({ inline = false, agentSelector }: ComposerPanelPr // Per-chat draft from the module-scoped store. Typing only re-renders this // component (it's the sole subscriber); the value is restored automatically when // activeChatId changes and survives the inline→docked composer swap. - const [draft, setDraft] = useComposerDraft(activeChatId) + const [draft, setDraft] = useComposerDraft(activeChatId, selectedAgent ?? undefined) const composerInputRef = useRef(null) const composerInputWidthRef = useRef(null) const composerFileInputRef = useRef(null) diff --git a/desktop-app/ui/src/components/agents/__tests__/ChatThread.renderSemantics.test.tsx b/desktop-app/ui/src/components/agents/__tests__/ChatThread.renderSemantics.test.tsx index 838dd7971d..7305b9b1bd 100644 --- a/desktop-app/ui/src/components/agents/__tests__/ChatThread.renderSemantics.test.tsx +++ b/desktop-app/ui/src/components/agents/__tests__/ChatThread.renderSemantics.test.tsx @@ -16,6 +16,9 @@ const actionsValue = { chatEndRef: { current: null }, handleSelectChat: vi.fn(), handleRenameChat: vi.fn(), + captureChatDeleteFence: vi.fn(async () => { + throw new Error('no delete target') + }), handleDeleteChat: vi.fn(), } const runtimeValue = { cancelTask: vi.fn() } diff --git a/desktop-app/ui/src/components/agents/__tests__/chatThread.connectFailure.test.tsx b/desktop-app/ui/src/components/agents/__tests__/chatThread.connectFailure.test.tsx index c863860722..0573203575 100644 --- a/desktop-app/ui/src/components/agents/__tests__/chatThread.connectFailure.test.tsx +++ b/desktop-app/ui/src/components/agents/__tests__/chatThread.connectFailure.test.tsx @@ -19,6 +19,9 @@ const actionsValue = { chatEndRef: { current: null }, handleSelectChat: vi.fn(), handleRenameChat: vi.fn(), + captureChatDeleteFence: vi.fn(async () => { + throw new Error('no delete target') + }), handleDeleteChat: vi.fn(), } const runtimeValue = { cancelTask: vi.fn() } diff --git a/desktop-app/ui/src/components/agents/__tests__/chatThread.toolSteps.test.tsx b/desktop-app/ui/src/components/agents/__tests__/chatThread.toolSteps.test.tsx index 569adc9b35..0013f2e1c0 100644 --- a/desktop-app/ui/src/components/agents/__tests__/chatThread.toolSteps.test.tsx +++ b/desktop-app/ui/src/components/agents/__tests__/chatThread.toolSteps.test.tsx @@ -26,6 +26,9 @@ const actionsValue = { chatEndRef: { current: null }, handleSelectChat: vi.fn(), handleRenameChat: vi.fn(), + captureChatDeleteFence: vi.fn(async () => { + throw new Error('no delete target') + }), handleDeleteChat: vi.fn(), } const mcpRuntimeValue = { cancelTask: vi.fn() } diff --git a/desktop-app/ui/src/contexts/AgentChatActionsContext/index.ts b/desktop-app/ui/src/contexts/AgentChatActionsContext/index.ts index 39d0cfe9d1..50ee10b34b 100644 --- a/desktop-app/ui/src/contexts/AgentChatActionsContext/index.ts +++ b/desktop-app/ui/src/contexts/AgentChatActionsContext/index.ts @@ -1,3 +1,7 @@ export { AgentChatActionsContext, AgentChatActionsProvider } from './context' export { useAgentChatActionsContext } from './useAgentChatActionsContext' -export type { AgentChatActionsContextValue, AgentChatActionsProviderProps } from './types' +export type { + AgentChatActionsContextValue, + AgentChatActionsProviderProps, + QueuedChatDelete, +} from './types' diff --git a/desktop-app/ui/src/contexts/AgentChatActionsContext/types.ts b/desktop-app/ui/src/contexts/AgentChatActionsContext/types.ts index 81beab82ab..be847729ff 100644 --- a/desktop-app/ui/src/contexts/AgentChatActionsContext/types.ts +++ b/desktop-app/ui/src/contexts/AgentChatActionsContext/types.ts @@ -1,6 +1,13 @@ import type { ReactNode, RefObject } from 'react' +import type { ChatDeleteFence } from '../../../../src/types' import type { ComposerImageAttachment, ComposerReferenceAttachment } from '../../uiTypes' +export interface QueuedChatDelete { + agentRef: string + fence: ChatDeleteFence + hostAuthorityEpoch: number +} + /** * Stable, cross-cutting chat actions. These are `useCallback` handlers plus the * scroll ref — split out from chat state so action-only consumers (e.g. FleetBoard) @@ -12,8 +19,13 @@ export interface AgentChatActionsContextValue { handleCreateChat: () => Promise handleRenameChat: (chatId: string, newTitle: string) => Promise handleRenameChatForAgent: (agentRef: string, chatId: string, newTitle: string) => Promise - handleDeleteChat: (chatId: string) => Promise - handleDeleteChatForAgent: (agentRef: string, chatId: string) => Promise + captureChatDeleteFence: (agentRef: string) => Promise + handleDeleteChat: (chatId: string, deletion: QueuedChatDelete) => Promise + handleDeleteChatForAgent: ( + agentRef: string, + chatId: string, + deletion: QueuedChatDelete + ) => Promise handleSelectChat: (chatId: string) => Promise clearComposerSendError: () => void handleAddComposerImageAttachments: (attachments: ComposerImageAttachment[]) => void diff --git a/desktop-app/ui/src/contexts/AgentTaskTrackerContext/__tests__/taskTracker.test.ts b/desktop-app/ui/src/contexts/AgentTaskTrackerContext/__tests__/taskTracker.test.ts index bd21597adf..79dc253d2c 100644 --- a/desktop-app/ui/src/contexts/AgentTaskTrackerContext/__tests__/taskTracker.test.ts +++ b/desktop-app/ui/src/contexts/AgentTaskTrackerContext/__tests__/taskTracker.test.ts @@ -1,6 +1,13 @@ // @vitest-environment jsdom import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { classifyTier } from '@hooks/useTaskTier' +import { + ipcGenericForbidden, + ipcHostAccessDenied, + ipcHostAccessRevoked, + ipcHttpError, + ipcServerErrorMentioning403, +} from '../../../hooks/domain/__tests__/__fixtures__/ipcErrors' import { TaskTracker } from '../taskTracker' import { type TaskKey, type TaskState, makeTaskKey } from '../types' @@ -177,6 +184,64 @@ describe('TaskTracker', () => { expect(state.terminalResult).toMatchObject({ kind: 'error', source: 'result_fetch' }) }) + // R2-B1 / NEW-dui-3: the result fetch fails with the real RpcProxyClient + // rejection wrapped the way Electron IPC wraps it. + it.each([ + [ + 'confirmed revocation', + () => ipcHostAccessRevoked('rpc:getTaskResult'), + 'authority', + 'revoked', + ], + [ + 'host_access_denied', + () => ipcHostAccessDenied('rpc:getTaskResult'), + 'authority', + 'uncertain', + ], + ['generic 403', () => ipcGenericForbidden('rpc:getTaskResult'), 'authority', 'uncertain'], + [ + '401', + () => ipcHttpError('rpc:getTaskResult', 401, 'Unauthorized', { error: 'expired' }), + 'authority', + 'uncertain', + ], + [ + '503 whose body mentions 403', + () => ipcServerErrorMentioning403('rpc:getTaskResult'), + 'result_fetch', + undefined, + ], + [ + 'waking Host named support-401', + // getTaskResult does not map a waking Host (only the send path does), so + // the renderer sees rpc-proxy's raw structured 503. + () => + ipcHttpError('rpc:getTaskResult', 503, 'Service Unavailable', { + code: 'host_waking', + hostRef: 'support-401', + retryAfterMs: 2000, + }), + 'result_fetch', + undefined, + ], + ])( + 'IPC-wrapped %s result-fetch failure classifies authority', + async (_label, build, source, authority) => { + rpc.getTaskResult.mockRejectedValueOnce(await build()) + const onTerminal = vi.fn() + tracker.setCallbacks({ onTerminal }) + tracker.start(KEY, 'task-1', 'um-1') + await rpc.emit({ type: 'terminal', data: { taskId: 'task-1', status: 'completed' } }) + + expect(rpc.getTaskResult).toHaveBeenCalledWith('agent-x', 'task-1', ['agent-x']) + const [, state] = onTerminal.mock.calls[0]! + expect(state.status).toBe('failed') + expect(state.terminalResult).toMatchObject({ kind: 'error', source }) + expect(state.terminalResult.authority).toBe(authority) + } + ) + it('terminal completed does NOT corrupt a task that replaced it during the result fetch (H2)', async () => { const onTerminal = vi.fn(async () => undefined) tracker.setCallbacks({ onTerminal }) diff --git a/desktop-app/ui/src/contexts/AgentTaskTrackerContext/taskTracker.ts b/desktop-app/ui/src/contexts/AgentTaskTrackerContext/taskTracker.ts index 7d61459f1e..61ad238802 100644 --- a/desktop-app/ui/src/contexts/AgentTaskTrackerContext/taskTracker.ts +++ b/desktop-app/ui/src/contexts/AgentTaskTrackerContext/taskTracker.ts @@ -1,6 +1,10 @@ import { classifyTier } from '@hooks/useTaskTier' import { buildResponseFileAttachments } from '@lib/chatMessageAttachments' -import { extractAssistantReply } from '@lib/format' +import { + extractAssistantReply, + isAuthorizationError, + isConfirmedHostAccessRevoked, +} from '@lib/format' import type { ProgressStep } from '@/uiTypes' import type { ChatMessageAttachment, @@ -629,7 +633,21 @@ export class TaskTracker implements AgentTaskTracker { const { agentRef } = parseTaskKey(key) const result = await window.clerum.rpc.getTaskResult(agentRef, failedTaskId, [agentRef]) attachments = buildResponseFileAttachments(result) - } catch { + } catch (error) { + if (isAuthorizationError(error)) { + if (this.states.get(key)?.taskId !== failedTaskId) return + this.mutate(key, s => { + s.status = 'failed' + s.terminalResult = { + kind: 'error', + source: 'authority', + message: 'Host authorization rejected', + authority: isConfirmedHostAccessRevoked(error) ? 'revoked' : 'uncertain', + } + }) + await this.fireTerminal(key) + return + } // Preserve the authoritative terminal error if artifact retrieval fails. } if (this.states.get(key)?.taskId !== failedTaskId) return @@ -694,7 +712,7 @@ export class TaskTracker implements AgentTaskTracker { ...(attachments.length ? { attachments } : {}), } }) - } catch { + } catch (error) { // Same post-await identity guard on the failure branch — a replacement // during the await must not have its state overwritten with this error. if (this.states.get(key)?.taskId !== completedTaskId) return @@ -702,8 +720,17 @@ export class TaskTracker implements AgentTaskTracker { s.status = 'failed' s.terminalResult = { kind: 'error', - source: 'result_fetch', - message: 'Failed to retrieve task result after completion', + source: isAuthorizationError(error) ? 'authority' : 'result_fetch', + message: isAuthorizationError(error) + ? 'Host authorization rejected' + : 'Failed to retrieve task result after completion', + ...(isAuthorizationError(error) + ? { + authority: isConfirmedHostAccessRevoked(error) + ? ('revoked' as const) + : ('uncertain' as const), + } + : {}), } }) } diff --git a/desktop-app/ui/src/contexts/AgentTaskTrackerContext/types.ts b/desktop-app/ui/src/contexts/AgentTaskTrackerContext/types.ts index 9bc88667f8..5232a41aca 100644 --- a/desktop-app/ui/src/contexts/AgentTaskTrackerContext/types.ts +++ b/desktop-app/ui/src/contexts/AgentTaskTrackerContext/types.ts @@ -57,7 +57,9 @@ export type TaskTerminalResult = | { kind: 'reply'; content: string; attachments?: ChatMessageAttachment[] } | { kind: 'error' - source: 'failed' | 'result_fetch' | 'stream' + source: 'failed' | 'result_fetch' | 'stream' | 'authority' + /** A final task-result read denial; no raw response body crosses this boundary. */ + authority?: 'revoked' | 'uncertain' attachments?: ChatMessageAttachment[] message: string code?: string diff --git a/desktop-app/ui/src/hooks/__tests__/useAppController.hostHoldCatalogCache.test.tsx b/desktop-app/ui/src/hooks/__tests__/useAppController.hostHoldCatalogCache.test.tsx new file mode 100644 index 0000000000..ea6930712d --- /dev/null +++ b/desktop-app/ui/src/hooks/__tests__/useAppController.hostHoldCatalogCache.test.tsx @@ -0,0 +1,99 @@ +// @vitest-environment jsdom +/** + * F4: the renderer shares one in-flight `listSessions` request per + * scope/Host/query for five seconds. A request issued before a Host hold is + * answered with the authority that held before the hold; handing it to a reader + * that starts after `verifyHostAccess` released the hold would let that stale + * answer re-hold a Host the user was just cleared for. Holding a Host therefore + * invalidates its cached catalog requests. + * + * The held Host is deliberately not the selected one: a revoked hold of the + * selected agent already clears the whole cache through the chat controller's + * teardown, which would hide the defect. + */ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { act, waitFor } from '@testing-library/react' +import type { SessionsListResult } from '../../../../src/types' +import { + installAppControllerClerum, + renderAppController, +} from '../domain/__tests__/__fixtures__/appControllerHarness' +import { + deferred, + localIndex, + serverSessions, +} from '../domain/__tests__/__fixtures__/catalogFixtures' +import { ipcHostAccessRevoked } from '../domain/__tests__/__fixtures__/ipcErrors' +import { uninstallMockClerum } from '../domain/__tests__/__fixtures__/mockClerum' + +;(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true + +type ListSessionsQuery = { agent?: string; limit?: number; cursor?: string } + +describe('useAppController Host hold and the session catalog cache', () => { + let unmount: (() => void) | null = null + + afterEach(() => { + unmount?.() + unmount = null + vi.restoreAllMocks() + uninstallMockClerum() + }) + + it('does not hand a pre-hold catalog request to a reader after the hold is released', async () => { + const { clerum } = installAppControllerClerum({ agentNames: ['agent-x', 'agent-y'] }) + const staleCatalog = deferred() + let heldHostPageReads = 0 + clerum.chat.getIndex.mockImplementation(async (agentRef: string) => + agentRef === 'agent-x' ? localIndex([{ id: 'c1', title: 'old' }]) : localIndex([]) + ) + clerum.rpc.listSessions.mockImplementation( + async (hostRef: string, _teamId: string | undefined, query?: ListSessionsQuery) => { + if (query?.limit === 1 || hostRef !== 'agent-x') return serverSessions([]) + heldHostPageReads += 1 + if (heldHostPageReads === 1) return staleCatalog.promise + return serverSessions([{ agent: 'agent-x', chatId: 'c1', title: 'old' }]) + } + ) + clerum.rpc.renameSession.mockRejectedValue(await ipcHostAccessRevoked('rpc:renameSession')) + + const app = renderAppController() + unmount = app.unmount + await waitFor(() => expect(app.result.current.isAuthenticated).toBe(true)) + await waitFor(() => expect(app.result.current.initialExperienceLoading).toBe(false)) + act(() => { + app.result.current.handleSelectChatAgent('agent-y') + }) + // The cross-agent preview's catalog read for agent-x is in flight and + // stays so across the hold. + await waitFor(() => expect(heldHostPageReads).toBe(1)) + + await act(async () => { + await app.result.current.handleRenameChatForAgent('agent-x', 'c1', 'new') + }) + // Witness: the rename's confirmed revocation held agent-x. + expect(clerum.rpc.renameSession).toHaveBeenCalledWith('agent-x', 'agent-x', 'c1', 'new') + expect(app.result.current.isHostAccessBlocked('agent-x')).toBe(true) + + act(() => { + app.result.current.handleSelectChatAgent('agent-x') + }) + await waitFor(() => expect(app.result.current.isHostAccessBlocked('agent-x')).toBe(false)) + // The post-release reader issued its own catalog request instead of + // sharing the pre-hold one. + await waitFor(() => expect(heldHostPageReads).toBe(2)) + await waitFor(() => + expect(app.result.current.chatList.some(chat => chat.id === 'c1')).toBe(true) + ) + + await act(async () => { + staleCatalog.reject(await ipcHostAccessRevoked('rpc:listSessions')) + await staleCatalog.promise.catch(() => undefined) + }) + await act(async () => { + await new Promise(resolve => setTimeout(resolve, 20)) + }) + + expect(app.result.current.isHostAccessBlocked('agent-x')).toBe(false) + }) +}) diff --git a/desktop-app/ui/src/hooks/__tests__/useAppController.teamContext.test.tsx b/desktop-app/ui/src/hooks/__tests__/useAppController.teamContext.test.tsx index d0c26fc191..de996a46ea 100644 --- a/desktop-app/ui/src/hooks/__tests__/useAppController.teamContext.test.tsx +++ b/desktop-app/ui/src/hooks/__tests__/useAppController.teamContext.test.tsx @@ -58,4 +58,71 @@ describe('useAppController team context', () => { expect(switchedBack).toBe(true) expect(handle.switchTeam).toHaveBeenCalledWith(HARNESS_ME.teamId) }) + + // R1-L10: the delete fence binds to the authenticated principal's team only. + // The team directory is display data; a directory team must never stand in + // for an absent `me.teamId`, or a deletion would be fenced to a team the + // session never proved membership of. + it.each([ + ['an absent me.teamId stays null despite a directory team', '', null], + ['a padded me.teamId is trimmed', ' team-1 ', 'team-1'], + ])('fences chat deletion to %s', async (_label, meTeamId, expectedTeamId) => { + const { clerum } = installAppControllerClerum({ + me: { teamId: meTeamId }, + teamDirectory: { items: [], currentTeamId: 'team-dir' }, + }) + const app = renderAppController() + unmount = app.unmount + + await waitFor(() => expect(app.result.current.isAuthenticated).toBe(true)) + await waitFor(() => expect(app.result.current.initialExperienceLoading).toBe(false)) + // Witness: the directory team is visible to the rest of the app, so a + // fallback to it would have a value to use. + expect(app.result.current.currentTeamId).toBe(meTeamId || 'team-dir') + + await act(async () => { + await app.result.current.captureChatDeleteFence('agent-x') + }) + + expect(clerum.chat.captureDeleteFence).toHaveBeenCalledTimes(1) + expect(clerum.chat.captureDeleteFence).toHaveBeenCalledWith({ + environmentKey: expect.any(String), + userId: HARNESS_ME.id, + teamId: expectedTeamId, + }) + }) + + // The cache scope (`currentTeamId`) and the delete fence read `me.teamId` + // from one point: while it is present both carry it, and both follow it when + // the session switches team, whatever the (unchanged) directory reports. + it('derives the chat cache scope and the delete fence from the same me.teamId', async () => { + const { clerum, handle } = installAppControllerClerum({ + teamDirectory: { items: [], currentTeamId: 'team-dir' }, + }) + const app = renderAppController() + unmount = app.unmount + + await waitFor(() => expect(app.result.current.isAuthenticated).toBe(true)) + await waitFor(() => expect(app.result.current.initialExperienceLoading).toBe(false)) + + const fencedTeamId = async () => { + clerum.chat.captureDeleteFence.mockClear() + await act(async () => { + await app.result.current.captureChatDeleteFence('agent-x') + }) + expect(clerum.chat.captureDeleteFence).toHaveBeenCalledTimes(1) + return clerum.chat.captureDeleteFence.mock.calls[0]?.[0]?.teamId + } + + expect(app.result.current.currentTeamId).toBe(HARNESS_ME.teamId) + expect(await fencedTeamId()).toBe(HARNESS_ME.teamId) + + await act(async () => { + await app.result.current.handleEnsureTeamContext({ teamId: 'team-2' }) + }) + // Witness: the switch really went through and moved me.teamId. + expect(handle.switchTeam).toHaveBeenCalledWith('team-2') + await waitFor(() => expect(app.result.current.currentTeamId).toBe('team-2')) + expect(await fencedTeamId()).toBe('team-2') + }) }) diff --git a/desktop-app/ui/src/hooks/__tests__/useChatStore.test.tsx b/desktop-app/ui/src/hooks/__tests__/useChatStore.test.tsx index 9503dfad98..ae1c80ac9c 100644 --- a/desktop-app/ui/src/hooks/__tests__/useChatStore.test.tsx +++ b/desktop-app/ui/src/hooks/__tests__/useChatStore.test.tsx @@ -48,6 +48,76 @@ describe('useChatStore remote request cache', () => { expect(listSessions).toHaveBeenCalledTimes(2) }) + // F4: holding a Host drops its cached catalog requests (every query) and + // leaves every other Host's entries shared. The scope itself contains ':', + // so the Host is matched on the entry, not parsed out of the key. + it('invalidates only the held Host catalog entries', async () => { + const listSessions = vi.fn(async () => ({ items: [] })) + Object.defineProperty(window, 'clerum', { + configurable: true, + value: { rpc: { listSessions } }, + }) + const { result } = renderHook(() => useChatStore()) + const readAll = () => + Promise.all([ + result.current.listSessions('host', { agent: 'host', limit: 50 }), + result.current.listSessions('host', { agent: 'host', limit: 50, cursor: 'page-2' }), + result.current.listSessions('host-b', { agent: 'host-b', limit: 50 }), + ]) + const callsFor = (hostRef: string) => + listSessions.mock.calls.filter(call => (call as unknown[])[0] === hostRef).length + + result.current.setRemoteCacheScope('authenticated:user-f4:team-f4') + await readAll() + await readAll() + // Witness: the cache is live (three distinct keys, one upstream call each). + expect(callsFor('host')).toBe(2) + expect(callsFor('host-b')).toBe(1) + + result.current.invalidateSessionCatalog('host') + await readAll() + + expect(callsFor('host')).toBe(4) + expect(callsFor('host-b')).toBe(1) + }) + + // A request issued before the invalidation that rejects afterwards must not + // evict the newer request cached under the same key. + it('keeps the newer cached request when an invalidated one rejects late', async () => { + let rejectStale: (error: Error) => void = () => { + throw new Error('the stale request was never issued') + } + const listSessions = vi + .fn() + .mockImplementationOnce( + () => + new Promise((_resolve, reject) => { + rejectStale = reject + }) + ) + .mockImplementation(async () => ({ items: [] })) + Object.defineProperty(window, 'clerum', { + configurable: true, + value: { rpc: { listSessions } }, + }) + const { result } = renderHook(() => useChatStore()) + result.current.setRemoteCacheScope('authenticated:user-race:team-race') + const query = { agent: 'host', limit: 50 } + + const stale = result.current.listSessions('host', query) + result.current.invalidateSessionCatalog('host') + const fresh = result.current.listSessions('host', query) + await fresh + expect(listSessions).toHaveBeenCalledTimes(2) + + rejectStale(new Error('host access revoked')) + await expect(stale).rejects.toThrow('host access revoked') + + // Witness: the fresh request is still the cached one for this key. + await expect(result.current.listSessions('host', query)).resolves.toEqual({ items: [] }) + expect(listSessions).toHaveBeenCalledTimes(2) + }) + // #654 M12 — the host-model catalog has exactly one cache, in // `hostModelSelectionStore`, which also owns revision ordering. A second TTL // layer here could only serve a response read BEFORE a write, which is the diff --git a/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/appControllerHarness.tsx b/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/appControllerHarness.tsx index 2c9d24a965..2e3fa88631 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/appControllerHarness.tsx +++ b/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/appControllerHarness.tsx @@ -108,6 +108,8 @@ export interface AppControllerClerumOptions { desktopReleaseStatus?: typeof DEFAULT_DESKTOP_RELEASE_STATUS /** Payload for `team.directory` / `team.initialDirectory` / `team.list`. */ teamDirectory?: { items: unknown[]; currentTeamId: string } + /** Overrides on the authenticated `me` the session state reports. */ + me?: Partial } export interface AppControllerClerumHandle { @@ -146,7 +148,7 @@ export function extendMockClerumForAppController( const bridge = clerum as unknown as Record let authenticated = options.startAuthenticated ?? true - let sessionMe = { ...HARNESS_ME } + let sessionMe = { ...HARNESS_ME, ...options.me } const teamDirectoryPayload = options.teamDirectory ?? { items: [], currentTeamId: HARNESS_ME.teamId, diff --git a/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/catalogFixtures.ts b/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/catalogFixtures.ts new file mode 100644 index 0000000000..8d0a9d6c41 --- /dev/null +++ b/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/catalogFixtures.ts @@ -0,0 +1,55 @@ +import { parseSessionsListResult } from '../../../../../../src/rpcProxyClient' +import type { ChatIndex, SessionsListResult } from '../../../../../../src/types' + +export const CATALOG_NOW = '2026-09-12T00:00:00.000Z' + +/** + * A server catalog page built by the REAL wire parser, so fixtures are derived + * from the producer instead of hand-built parsed shapes. + */ +export function serverSessions( + items: Array<{ agent: string; chatId: string; title?: string }>, + nextCursor?: string +): SessionsListResult { + return parseSessionsListResult({ + items: items.map(i => ({ + agent: i.agent, + chatId: i.chatId, + turnCount: 1, + lastActivityAt: CATALOG_NOW, + ...(i.title !== undefined ? { title: i.title } : {}), + })), + ...(nextCursor !== undefined ? { nextCursor } : {}), + }) +} + +/** A local chat index whose tombstones carry the harness's authority scope. */ +export function localIndex( + chats: Array<{ id: string; title: string }>, + deletedChatIds: string[] = [] +): ChatIndex { + const authorityScope = { environmentKey: 'env-test', userId: 'unknown-user', teamId: 'team-1' } + return { + version: 1, + lastActiveChatId: null, + onboardingDismissed: false, + chats: chats.map(c => ({ + id: c.id, + title: c.title, + createdAt: CATALOG_NOW, + updatedAt: CATALOG_NOW, + messageCount: 0, + })), + deletedChatTombstones: deletedChatIds.map(chatId => ({ chatId, authorityScope })), + } +} + +export function deferred() { + let resolve!: (value: T) => void + let reject!: (reason: unknown) => void + const promise = new Promise((finish, fail) => { + resolve = finish + reject = fail + }) + return { promise, resolve, reject } +} diff --git a/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/controllerHarness.tsx b/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/controllerHarness.tsx index 56ec8e5655..830d83e399 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/controllerHarness.tsx +++ b/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/controllerHarness.tsx @@ -1,18 +1,30 @@ import type { ReactNode } from 'react' import { vi } from 'vitest' import { AgentTaskTrackerProvider } from '@contexts/AgentTaskTrackerContext' -import { renderHook } from '@testing-library/react' +import { act, renderHook } from '@testing-library/react' +import type { HostAuthorityHoldKind } from '../../../../lib/hostAuthorityStore' import { useAgentChatController } from '../../useAgentChatController' +import { type HarnessHostAuthority, useHarnessHostAuthority } from './hostAuthorityHarness' type ControllerParams = Parameters[0] export interface RenderControllerResult { result: ReturnType< - typeof renderHook, ControllerParams> + typeof renderHook, Partial> >['result'] rerender: (props?: Partial) => void unmount: () => void - params: ControllerParams + params: Partial + /** + * The mount's Host authority, from the production store factory. Only + * meaningful when the test did not override the authority params. + */ + hostAuthority: { + hold: (agentRef: string, kind: HostAuthorityHoldKind) => void + release: (agentRef: string) => void + getEpoch: (agentRef: string) => number + isBlocked: (agentRef: string) => boolean + } spies: { pushToast: ReturnType pushNotification: ReturnType @@ -30,6 +42,9 @@ export interface RenderControllerResult { * (`useAgentChatController.ts:492`) does NOT auto-switchToChat on mount — * tests that exercise `switchToChat` / `sendAgentMessage` control the chat * explicitly. Pass `navItem: 'chat'` to exercise the auto-select path. + * + * Host authority defaults to `useHarnessHostAuthority`, the production store + * factory, so epoch comparisons in the controller are exercised for real. */ export function renderController( overrides: Partial = {} @@ -43,10 +58,12 @@ export function renderController( canDeliverChatResponseNotification: vi.fn(() => true), } - const params = { + const params: Partial = { selectedAgent: 'agent-x', agentNames: ['agent-x'], currentTeamId: 'team-1', + chatAuthorityTeamId: 'team-1', + currentEnvironmentKey: 'env-test', currentTeamName: 'Team 1', isAuthenticated: true, loadMenuData: true, @@ -60,18 +77,38 @@ export function renderController( openAgentConversationFromNotification: spies.openAgentConversationFromNotification, decideApprovalFromNotification: spies.decideApprovalFromNotification, ...overrides, - } as ControllerParams + } + + let authority: HarnessHostAuthority | null = null + const currentAuthority = (): HarnessHostAuthority => { + if (!authority) throw new Error('controller is not mounted') + return authority + } // Post-D.3 the controller reads the task tracker from context and registers // its own onTerminal/onSuspended callbacks, so the provider must wrap it. No // callbacks are injected here — the controller owns them (it has pushToast, // pushNotification, chatStore and the visibility refs). - const utils = renderHook((p: ControllerParams) => useAgentChatController(p), { - initialProps: params, - wrapper: ({ children }: { children: ReactNode }) => ( - {children} - ), - }) + const utils = renderHook( + (p: Partial) => { + const hostAuthority = useHarnessHostAuthority() + authority = hostAuthority + return useAgentChatController({ + onHostAccessRevoked: hostAuthority.onHostAccessRevoked, + onHostAuthorityUncertain: hostAuthority.onHostAuthorityUncertain, + isHostAccessBlocked: hostAuthority.isHostAccessBlocked, + getHostAuthorityEpoch: hostAuthority.getHostAuthorityEpoch, + hostAuthorityRevision: hostAuthority.revision, + ...p, + } as ControllerParams) + }, + { + initialProps: params, + wrapper: ({ children }: { children: ReactNode }) => ( + {children} + ), + } + ) return { result: utils.result, @@ -79,6 +116,20 @@ export function renderController( utils.rerender({ ...params, ...(props ?? {}) }), unmount: utils.unmount, params, + hostAuthority: { + hold: (agentRef, kind) => + act(() => { + const hostAuthority = currentAuthority() + if (kind === 'revoked') hostAuthority.onHostAccessRevoked(agentRef) + else hostAuthority.onHostAuthorityUncertain(agentRef) + }), + release: agentRef => + act(() => { + currentAuthority().release(agentRef) + }), + getEpoch: agentRef => currentAuthority().store.getEpoch(agentRef), + isBlocked: agentRef => currentAuthority().store.isBlocked(agentRef), + }, spies, } } diff --git a/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/hostAuthorityHarness.ts b/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/hostAuthorityHarness.ts new file mode 100644 index 0000000000..313ce6c9a2 --- /dev/null +++ b/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/hostAuthorityHarness.ts @@ -0,0 +1,56 @@ +import { useRef, useState } from 'react' +import { + type HostAuthorityHoldKind, + type HostAuthorityStore, + createHostAuthorityStore, +} from '../../../../lib/hostAuthorityStore' + +/** + * Host authority for controller tests, built on the SAME store factory + * `useAppController` uses (R2-M5). A stub such as `getHostAuthorityEpoch = () => + * 0` makes every epoch comparison vacuous, so a regression in the production + * epoch path could never turn a controller test red. + * + * The callbacks are stable for the mount, like the production `useCallback`s, + * and a hold or release bumps `revision` exactly as `useAppController` does. + */ +export interface HarnessHostAuthority { + store: HostAuthorityStore + revision: number + onHostAccessRevoked: (agentRef: string) => void + onHostAuthorityUncertain: (agentRef: string) => void + isHostAccessBlocked: (agentRef: string) => boolean + getHostAuthorityEpoch: (agentRef: string) => number + /** Release a hold the way a successful `verifyHostAccess` does. */ + release: (agentRef: string) => void +} + +export type HostAuthorityListener = (agentRef: string, kind: HostAuthorityHoldKind) => void + +export function useHarnessHostAuthority(listener?: HostAuthorityListener): HarnessHostAuthority { + const [revision, setRevision] = useState(0) + const listenerRef = useRef(listener) + listenerRef.current = listener + const [stable] = useState(() => { + const store = createHostAuthorityStore() + const bump = () => setRevision(value => value + 1) + const hold = (agentRef: string, kind: HostAuthorityHoldKind) => { + listenerRef.current?.(agentRef, kind) + if (store.hold(agentRef, kind)) bump() + } + return { + store, + onHostAccessRevoked: (agentRef: string) => hold(agentRef, 'revoked'), + onHostAuthorityUncertain: (agentRef: string) => hold(agentRef, 'uncertain'), + isHostAccessBlocked: (agentRef: string) => store.isBlocked(agentRef), + getHostAuthorityEpoch: (agentRef: string) => store.getEpoch(agentRef), + release: (agentRef: string) => { + const heldAtEpoch = store.heldAtEpoch(agentRef) + if (heldAtEpoch === undefined) throw new Error(`${agentRef} is not held`) + if (!store.release(agentRef, heldAtEpoch)) throw new Error(`${agentRef} hold moved`) + bump() + }, + } + }) + return { ...stable, revision } +} diff --git a/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/ipcErrors.ts b/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/ipcErrors.ts new file mode 100644 index 0000000000..8392255430 --- /dev/null +++ b/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/ipcErrors.ts @@ -0,0 +1,176 @@ +import { ApiError } from '../../../../../../src/httpClient' +import { RpcProxyClient } from '../../../../../../src/rpcProxyClient' +import { + HOST_ACCESS_DENIED_CODE, + HOST_ACCESS_REVOKED_CODE, +} from '../../../../../../src/upstreamErrors' + +/** + * Renderer-side error fixtures built from the REAL main-process producers. + * + * A main-process rejection never reaches the renderer as the producer's own + * Error: `ipcRenderer.invoke` rejects with a new Error whose message is + * `Error invoking remote method '': `. Every + * classifier in `lib/format.ts` therefore parses that wrapped string, and a + * fixture written by hand without the prefix (or with a message no producer + * emits) certifies a parser that production never exercises (R2-B1). + * + * Each builder below drives the real producer (`RpcProxyClient` over a stubbed + * `fetch`, `ApiError`, the `AppService` availability mapping) and wraps the + * result exactly the way Electron does. + */ + +const RPC_TOKEN = 'rpc-token' + +/** rpc-proxy's own Host-access denial body (cross-layer contract, R1-M6). */ +const HOST_ACCESS_DENIAL_ERROR = 'Forbidden: user cannot access this host' + +export type RpcChannel = + | 'rpc:listSessions' + | 'rpc:loadSessionMessages' + | 'rpc:renameSession' + | 'rpc:invokeHostMessage' + | 'rpc:getTaskResult' + +/** The rejection `ipcRenderer.invoke` produces for a main-process throw. */ +export function wrapLikeElectronIpc(channel: string, error: Error): Error { + return new Error(`Error invoking remote method '${channel}': ${String(error)}`) +} + +function jsonResponse(status: number, statusText: string, body: unknown): Response { + return new Response(typeof body === 'string' ? body : JSON.stringify(body), { + status, + statusText, + headers: { 'content-type': 'application/json' }, + }) +} + +/** + * Run one RpcProxyClient call against a single canned response and return the + * Error it throws. Fails loudly when the producer resolves instead: a fixture + * builder that silently returned nothing would hand the test an `undefined`. + */ +async function rejectionOf( + response: Response, + call: (client: RpcProxyClient) => Promise +): Promise { + const originalFetch = globalThis.fetch + let fetchCalls = 0 + globalThis.fetch = (async () => { + fetchCalls += 1 + return response.clone() + }) as typeof fetch + try { + await call(new RpcProxyClient()) + } catch (error) { + if (fetchCalls === 0) { + throw new Error(`producer rejected before calling fetch: ${String(error)}`) + } + if (!(error instanceof Error)) throw new Error(`producer threw a non-Error: ${String(error)}`) + return error + } finally { + globalThis.fetch = originalFetch + } + throw new Error(`producer resolved for HTTP ${response.status}; expected a rejection`) +} + +function callFor(channel: RpcChannel): (client: RpcProxyClient) => Promise { + switch (channel) { + case 'rpc:listSessions': + return client => client.listSessions(RPC_TOKEN, 'agent-x', { agent: 'agent-x', limit: 1 }) + case 'rpc:loadSessionMessages': + return client => client.loadSessionMessages(RPC_TOKEN, 'agent-x', 'agent-x', 'chat-1') + case 'rpc:renameSession': + return client => client.renameSession(RPC_TOKEN, 'agent-x', 'agent-x', 'chat-1', 'Title') + case 'rpc:invokeHostMessage': + return client => + client.invokeHostMessage(RPC_TOKEN, 'agent-x', { content: 'hello' }, { async: true }) + case 'rpc:getTaskResult': + return client => client.getTaskResult(RPC_TOKEN, 'agent-x', 'task-1') + } +} + +/** + * rpc-proxy's own authorization denial for a user whose access was removed + * (`code: 'host_access_revoked'`), as the real RpcProxyClient projects it for + * `channel`, wrapped by IPC. + */ +export async function ipcHostAccessRevoked(channel: RpcChannel): Promise { + const error = await rejectionOf( + jsonResponse(403, 'Forbidden', { + error: HOST_ACCESS_DENIAL_ERROR, + code: HOST_ACCESS_REVOKED_CODE, + }), + callFor(channel) + ) + return wrapLikeElectronIpc(channel, error) +} + +/** + * rpc-proxy's own denial that is NOT a removal (`code: 'host_access_denied'`, + * e.g. a disabled Host). The renderer must treat it as an uncertain 403. + */ +export async function ipcHostAccessDenied(channel: RpcChannel): Promise { + const error = await rejectionOf( + jsonResponse(403, 'Forbidden', { + error: HOST_ACCESS_DENIAL_ERROR, + code: HOST_ACCESS_DENIED_CODE, + }), + callFor(channel) + ) + return wrapLikeElectronIpc(channel, error) +} + +/** + * A 403 that does not confirm revocation: an arbitrary upstream 403 body + * (for example a scope denial passed through from mcp-host). + */ +export async function ipcGenericForbidden(channel: RpcChannel): Promise { + const error = await rejectionOf( + jsonResponse(403, 'Forbidden', { error: 'missing send scope' }), + callFor(channel) + ) + return wrapLikeElectronIpc(channel, error) +} + +/** A plain HTTP failure through the real producer for `channel`. */ +export async function ipcHttpError( + channel: RpcChannel, + status: number, + statusText: string, + body: unknown +): Promise { + const error = await rejectionOf(jsonResponse(status, statusText, body), callFor(channel)) + return wrapLikeElectronIpc(channel, error) +} + +/** + * The "503 body mentions 403" case: a server error whose body text carries an + * authorization status. Only the response status may classify it. + */ +export function ipcServerErrorMentioning403(channel: RpcChannel): Promise { + return ipcHttpError(channel, 503, 'Service Unavailable', { + error: 'upstream returned 403 Forbidden', + }) +} + +/** + * The exact `AppService.toHostAvailabilityError` text for a waking Host whose + * name contains a status-looking number (R1-M16: `support-401 is waking`). + * `AppService` pulls in Electron main-process modules, so the mapping's text + * is reproduced here; `appService.hostWaking.test.ts` pins the producer side. + * Only the Host send path maps a waking Host to this text (R3-L15); every other + * channel surfaces the raw 503, so build those with `ipcHttpError`. + */ +export function ipcHostWaking(channel: 'rpc:invokeHostMessage', hostRef: string): Error { + const cause = new ApiError( + '503 Service Unavailable: Host is waking up', + 503, + JSON.stringify({ code: 'host_waking', hostRef, retryAfterMs: 2000 }) + ) + if (!cause.bodyText.includes('"host_waking"')) throw new Error('fixture lost its code') + return wrapLikeElectronIpc( + channel, + new Error(`host_waking: agent host "${hostRef}" is waking up — retry shortly`) + ) +} diff --git a/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/mockClerum.ts b/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/mockClerum.ts index 125bb93925..165c0edd3b 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/mockClerum.ts +++ b/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/mockClerum.ts @@ -1,4 +1,9 @@ -import { type Mock, vi } from 'vitest' +import { type Mock, inject, vi } from 'vitest' +import { mkdtempSync, rmSync } from 'node:fs' +import { join } from 'node:path' +import { ChatStore } from '../../../../../../src/chatStore' +import type { ChatDeleteFence, ChatMessage } from '../../../../../../src/types' +import type {} from './mockClerumStoreRoot' /** * D.0 characterization fixture — installs a fake `window.clerum` bridge. @@ -8,8 +13,19 @@ import { type Mock, vi } from 'vitest' * `useAppController.test.tsx`). `useChatStore` is a thin wrapper over * `window.clerum.chat.*` / `window.clerum.rpc.*`, so this exercises the real * hook code path and keeps the suite consistent with the rest of the repo. + * + * R1-M11: the transcript writes and reads (`create`, `delete`, `loadMessages`, + * `appendMessages`, `upsertMessages`) delegate to a real `ChatStore` on a + * per-install directory under the run's temporary root, so suites observe the + * merge, delete tombstone and paging behavior the IPC handlers run in + * production. They stay `vi.fn` wrappers, so call assertions and per-test + * overrides keep working. `uninstallMockClerum` removes the directory; the + * global setup in `mockClerumStoreRoot.ts` removes the root after the run, + * including any directory a late store write recreated. */ +const storeDirs = new Set() + type Handler = (event: unknown) => void // `ReturnType` resolves to `Mock`, and // that union does not satisfy `(...args: any) => any`. Callers here write @@ -21,9 +37,12 @@ interface ChatMock { list: Fn create: Fn rename: Fn + getBindingGeneration: Fn + captureDeleteFence: Fn delete: Fn loadMessages: Fn appendMessages: Fn + upsertMessages: Fn replaceMessages: Fn markUnreadTerminal: Fn clearUnreadTerminal: Fn @@ -58,27 +77,49 @@ export interface MockClerum { hasProgressHandler: (taskId: string) => boolean /** True once an activity handler exists for `hostRef`. */ hasActivityHandler: (hostRef: string) => boolean + /** What the real store holds for one chat, read straight from it. */ + persistedMessages: (agentRef: string, chatId: string) => Promise>> } export function installMockClerum(): MockClerum { const progressHandlers = new Map() const activityHandlers = new Map() - - const isoNow = () => new Date().toISOString() + const root = inject('mockClerumStoreRoot') + if (!root) { + throw new Error( + 'mockClerum needs the mockClerumStoreRoot global setup; add it to the vitest globalSetup list' + ) + } + const storeDir = mkdtempSync(join(root, 'store-')) + storeDirs.add(storeDir) + const store = new ChatStore(storeDir) const chat = { list: vi.fn(async () => []), - create: vi.fn(async (_agentRef: string, chatId: string) => ({ - id: chatId, - title: 'New Chat', - createdAt: isoNow(), - updatedAt: isoNow(), - messageCount: 0, - })), + create: vi.fn((agentRef: string, chatId: string) => store.createChat(agentRef, chatId)), rename: vi.fn(async () => undefined), - delete: vi.fn(async () => undefined), - loadMessages: vi.fn(async () => []), - appendMessages: vi.fn(async () => undefined), + getBindingGeneration: vi.fn(async () => 1), + captureDeleteFence: vi.fn(async (authorityScope: unknown) => ({ + version: 1, + authorityScope, + bindingGeneration: 1, + sessionGeneration: 1, + })), + // `chat:delete` hands the fence's scope to the store once main has matched + // it against the current authority; the store holds that same scope. + delete: vi.fn((agentRef: string, chatId: string, fence: ChatDeleteFence) => { + store.setAuthorityScope(fence.authorityScope) + return store.deleteChat(agentRef, chatId, fence.authorityScope) + }), + loadMessages: vi.fn((agentRef: string, chatId: string, limit?: number, offset?: number) => + store.loadMessages(agentRef, chatId, limit, offset) + ), + appendMessages: vi.fn((agentRef: string, chatId: string, messages: ChatMessage[]) => + store.appendMessages(agentRef, chatId, messages) + ), + upsertMessages: vi.fn((agentRef: string, chatId: string, messages: ChatMessage[]) => + store.upsertMessages(agentRef, chatId, messages) + ), replaceMessages: vi.fn(async () => undefined), markUnreadTerminal: vi.fn(async () => undefined), clearUnreadTerminal: vi.fn(async () => undefined), @@ -138,6 +179,8 @@ export function installMockClerum(): MockClerum { }, hasProgressHandler: (taskId: string) => progressHandlers.has(taskId), hasActivityHandler: (hostRef: string) => activityHandlers.has(hostRef), + persistedMessages: async (agentRef: string, chatId: string) => + (await store.loadMessages(agentRef, chatId)) as unknown as Array>, } Object.defineProperty(window, 'clerum', { @@ -151,4 +194,6 @@ export function installMockClerum(): MockClerum { export function uninstallMockClerum(): void { delete (window as { clerum?: unknown }).clerum + for (const dir of storeDirs) rmSync(dir, { recursive: true, force: true }) + storeDirs.clear() } diff --git a/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/mockClerumStoreRoot.ts b/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/mockClerumStoreRoot.ts new file mode 100644 index 0000000000..66c3d50aab --- /dev/null +++ b/desktop-app/ui/src/hooks/domain/__tests__/__fixtures__/mockClerumStoreRoot.ts @@ -0,0 +1,27 @@ +import type { TestProject } from 'vitest/node' +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' + +declare module 'vitest' { + export interface ProvidedContext { + mockClerumStoreRoot: string + } +} + +/** + * Vitest global setup for the `mockClerum` fixture: one temporary root per run + * that holds every per-install `ChatStore` directory. + * + * A hook can still be writing through the store after a test unmounts it, and + * that write recreates a per-install directory the fixture has already + * removed. Removing the whole root here runs after every test file has + * finished, so no directory outlives the run. + */ +export default function setup(project: TestProject): () => void { + const root = mkdtempSync(join(tmpdir(), 'mockclerum-')) + project.provide('mockClerumStoreRoot', root) + return () => { + rmSync(root, { recursive: true, force: true }) + } +} diff --git a/desktop-app/ui/src/hooks/domain/__tests__/chatActions.characterization.test.tsx b/desktop-app/ui/src/hooks/domain/__tests__/chatActions.characterization.test.tsx index 5624ea5c46..20ef621e9f 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/chatActions.characterization.test.tsx +++ b/desktop-app/ui/src/hooks/domain/__tests__/chatActions.characterization.test.tsx @@ -59,13 +59,18 @@ describe('handleDeleteChat', () => { expect(result.current.sessionStateByChatKey[key]?.state).toBe('processing') await act(async () => { - await result.current.handleDeleteChat(chatId) + const deletion = await result.current.captureChatDeleteFence('agent-x') + await result.current.handleDeleteChat(chatId, deletion) }) // ack tore down the SSE subscription (no handler left) so a late terminal // can't resurrect the deleted chat. expect(clerum.hasProgressHandler('task-del')).toBe(false) - expect(clerum.chat.delete).toHaveBeenCalledWith('agent-x', chatId) + expect(clerum.chat.delete).toHaveBeenCalledWith( + 'agent-x', + chatId, + expect.objectContaining({ version: 1, bindingGeneration: 1 }) + ) // D.5 review #5: the deleted chat's session-state entry is dropped. expect(result.current.sessionStateByChatKey[key]).toBeUndefined() // It was the only/active chat → a fresh chat is created and selected. diff --git a/desktop-app/ui/src/hooks/domain/__tests__/chatListCatalogAuthority.test.tsx b/desktop-app/ui/src/hooks/domain/__tests__/chatListCatalogAuthority.test.tsx new file mode 100644 index 0000000000..2cb2382910 --- /dev/null +++ b/desktop-app/ui/src/hooks/domain/__tests__/chatListCatalogAuthority.test.tsx @@ -0,0 +1,406 @@ +// @vitest-environment jsdom +/** + * Catalog failure handling and per-Host authority in `useChatListController`, + * driven through the real controller with IPC rejections built from the real + * producers and the production Host-authority store. + */ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { act, waitFor } from '@testing-library/react' +import type { SessionsListResult } from '../../../../../src/types' +import { isHostAvailabilityError } from '../../../lib/format' +import { deferred, localIndex, serverSessions } from './__fixtures__/catalogFixtures' +import { renderController } from './__fixtures__/controllerHarness' +import { ipcHttpError } from './__fixtures__/ipcErrors' +import { type MockClerum, installMockClerum, uninstallMockClerum } from './__fixtures__/mockClerum' + +;(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true + +let clerum: MockClerum + +beforeEach(() => { + clerum = installMockClerum() +}) + +afterEach(() => { + vi.useRealTimers() + vi.restoreAllMocks() + uninstallMockClerum() +}) + +type CatalogQuery = { agent: string; limit: number; cursor?: string } | undefined + +const catalogToasts = (pushToast: ReturnType) => + pushToast.mock.calls.filter(([message]) => String(message).startsWith('Chat list for')) + +const catalogUnavailable = () => + ipcHttpError('rpc:listSessions', 503, 'Service Unavailable', { error: 'upstream unavailable' }) + +/** Let pending promise continuations run without advancing any timer. */ +async function flushMicrotasks(until: () => boolean): Promise { + for (let turn = 0; turn < 200 && !until(); turn += 1) { + await Promise.resolve() + } + if (!until()) throw new Error('condition not reached within 200 microtask turns') +} + +describe('load-more rejection for a Host that is no longer shown (NEW-dui-1)', () => { + it("keeps agent-b's load-more state when agent-a's cursor is rejected late", async () => { + const pendingA = deferred() + clerum.chat.getIndex.mockResolvedValue(localIndex([])) + clerum.rpc.listSessions.mockImplementation( + async (agentRef: string, _teamId: string | undefined, query: CatalogQuery) => { + if (query?.cursor) return pendingA.promise + return serverSessions( + [{ agent: agentRef, chatId: `${agentRef}-page-1` }], + `${agentRef}-cursor` + ) + } + ) + const controller = renderController({ + selectedAgent: 'agent-a', + agentNames: ['agent-a', 'agent-b'], + }) + await waitFor(() => expect(controller.result.current.chatListHasMoreRemoteSessions).toBe(true)) + + let loadMoreA!: Promise + act(() => { + loadMoreA = controller.result.current.loadMoreChatSessions() + }) + await waitFor(() => + expect(clerum.rpc.listSessions).toHaveBeenCalledWith('agent-a', undefined, { + agent: 'agent-a', + limit: 50, + cursor: 'agent-a-cursor', + }) + ) + + controller.rerender({ selectedAgent: 'agent-b', agentNames: ['agent-a', 'agent-b'] }) + await waitFor(() => + expect(controller.result.current.chatList.map(chat => chat.id)).toContain('agent-b-page-1') + ) + expect(controller.result.current.chatListHasMoreRemoteSessions).toBe(true) + + await act(async () => { + pendingA.reject( + await ipcHttpError('rpc:listSessions', 400, 'Bad Request', { + error: 'Invalid sessions cursor', + }) + ) + await loadMoreA + }) + + expect(controller.result.current.chatListHasMoreRemoteSessions).toBe(true) + // Liveness: the rejection was the terminal kind (no retryable toast). + expect(controller.spies.pushToast).not.toHaveBeenCalledWith( + expect.stringContaining("Couldn't load more chats"), + 'info' + ) + }) +}) + +describe('load-more terminal cursor while the same Host is held (NEW-fa-3)', () => { + it('leaves hasMore false and forgets the cursor when a cursor 400 arrives during an authority hold', async () => { + clerum.chat.getIndex.mockResolvedValue(localIndex([])) + clerum.rpc.listSessions.mockImplementation( + async (agentRef: string, _teamId: string | undefined, query: CatalogQuery) => { + if (query?.cursor) { + throw await ipcHttpError('rpc:listSessions', 400, 'Bad Request', { + error: 'Invalid sessions cursor', + }) + } + return serverSessions( + [{ agent: agentRef, chatId: `${agentRef}-page-1` }], + `${agentRef}-cursor` + ) + } + ) + const controller = renderController({ selectedAgent: 'agent-a', agentNames: ['agent-a'] }) + // Liveness: the first page left a valid cursor and a "Load more" button. + await waitFor(() => expect(controller.result.current.chatListHasMoreRemoteSessions).toBe(true)) + + // The Host is held before the load-more starts, so the request snapshots the + // held epoch and the rejection reaches the blocked-Host branch. + controller.hostAuthority.hold('agent-a', 'uncertain') + + await act(async () => { + await controller.result.current.loadMoreChatSessions() + }) + + // Liveness: the cursor request really went out and was rejected. + expect(clerum.rpc.listSessions).toHaveBeenCalledWith('agent-a', undefined, { + agent: 'agent-a', + limit: 50, + cursor: 'agent-a-cursor', + }) + expect(controller.result.current.chatListHasMoreRemoteSessions).toBe(false) + + // The cursor was forgotten: another load-more issues no further read. + const readsAfterFirst = clerum.rpc.listSessions.mock.calls.length + await act(async () => { + await controller.result.current.loadMoreChatSessions() + }) + expect(clerum.rpc.listSessions.mock.calls.length).toBe(readsAfterFirst) + // The terminal rejection is not a retryable failure: no toast. + expect(controller.spies.pushToast).not.toHaveBeenCalledWith( + expect.stringContaining("Couldn't load more chats"), + 'info' + ) + }) +}) + +describe('catalog offline toast (R2-L4, NEW-tq-5, NEW-dui-5/6)', () => { + it('does not report a waking Host as offline', async () => { + const waking = await ipcHttpError('rpc:listSessions', 503, 'Service Unavailable', { + code: 'host_waking', + hostRef: 'agent-a', + retryAfterMs: 2000, + message: 'Host is waking up', + }) + // Fixture sanity: the real producer's wrapped 503 carries the waking code. + expect(isHostAvailabilityError(waking)).toBe(true) + clerum.chat.getIndex.mockResolvedValue(localIndex([{ id: 'cached-a', title: 'Cached chat' }])) + clerum.rpc.listSessions.mockRejectedValue(waking) + const controller = renderController({ selectedAgent: 'agent-a', agentNames: ['agent-a'] }) + + await waitFor(() => expect(clerum.rpc.listSessions).toHaveBeenCalled()) + await waitFor(() => + expect(controller.result.current.chatList.map(chat => chat.id)).toEqual(['cached-a']) + ) + // Let the post-paint selected-agent read settle too. + await act(async () => { + await new Promise(resolve => setTimeout(resolve, 60)) + }) + await act(async () => { + await new Promise(resolve => setTimeout(resolve, 20)) + }) + expect(catalogToasts(controller.spies.pushToast)).toEqual([]) + expect(controller.hostAuthority.isBlocked('agent-a')).toBe(false) + }) + + it('shows one offline toast for several failing Hosts and none for a later failure', async () => { + const unavailable = await catalogUnavailable() + clerum.chat.getIndex.mockResolvedValue(localIndex([])) + clerum.rpc.listSessions.mockRejectedValue(unavailable) + const controller = renderController({ + selectedAgent: null, + agentNames: ['agent-a', 'agent-b'], + }) + + await waitFor(() => expect(catalogToasts(controller.spies.pushToast)).toHaveLength(1)) + expect(catalogToasts(controller.spies.pushToast)[0]).toEqual([ + 'Chat list for agent-a, agent-b is offline. Showing saved chats.', + 'info', + ]) + + // A later failure while the outage is already reported. + controller.rerender({ selectedAgent: null, agentNames: ['agent-a', 'agent-b', 'agent-c'] }) + await waitFor(() => + expect(clerum.rpc.listSessions).toHaveBeenCalledWith('agent-c', undefined, { + agent: 'agent-c', + limit: 50, + }) + ) + await act(async () => { + await new Promise(resolve => setTimeout(resolve, 20)) + }) + expect(catalogToasts(controller.spies.pushToast)).toHaveLength(1) + }) + + it('drops a queued offline toast when the controller unmounts', async () => { + const pending = deferred() + clerum.chat.getIndex.mockResolvedValue(localIndex([])) + clerum.rpc.listSessions.mockReturnValue(pending.promise) + const controller = renderController({ selectedAgent: null, agentNames: ['agent-a'] }) + await waitFor(() => expect(clerum.rpc.listSessions).toHaveBeenCalled()) + const unavailable = await catalogUnavailable() + + vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] }) + pending.reject(unavailable) + // Witness: the rejection reached the controller and queued the toast. + await flushMicrotasks(() => vi.getTimerCount() === 1) + + controller.unmount() + expect(vi.getTimerCount()).toBe(0) + vi.runOnlyPendingTimers() + expect(catalogToasts(controller.spies.pushToast)).toEqual([]) + }) + + it('drops the previous scope’s queued offline toast when the scope changes', async () => { + const pending = deferred() + const unavailable = await catalogUnavailable() + clerum.chat.getIndex.mockResolvedValue(localIndex([])) + clerum.rpc.listSessions.mockImplementationOnce(() => pending.promise) + clerum.rpc.listSessions.mockRejectedValue(unavailable) + const controller = renderController({ selectedAgent: null, agentNames: ['agent-a'] }) + await waitFor(() => expect(clerum.rpc.listSessions).toHaveBeenCalledTimes(1)) + + vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] }) + pending.reject(unavailable) + // Witness: the old scope's rejection queued its toast. + await flushMicrotasks(() => vi.getTimerCount() === 1) + + // The new scope is down as well and queues its own toast. + act(() => { + controller.rerender({ selectedAgent: null, agentNames: ['agent-a'], currentTeamId: 'team-2' }) + }) + await flushMicrotasks(() => clerum.rpc.listSessions.mock.calls.length === 2) + for (let turn = 0; turn < 200; turn += 1) await Promise.resolve() + act(() => { + vi.runOnlyPendingTimers() + }) + + // One toast for the one outage the user is looking at, never a second + // one flushed from the scope they left. + expect(catalogToasts(controller.spies.pushToast)).toEqual([ + ['Chat list for agent-a is offline. Showing saved chats.', 'info'], + ]) + }) + + it('ignores a success from an earlier scope when deduplicating the next outage', async () => { + const lateOldScopeSuccess = deferred() + const unavailable = await catalogUnavailable() + let teamOneReads = 0 + let scope: 'team-1' | 'team-2' = 'team-1' + clerum.chat.getIndex.mockResolvedValue(localIndex([])) + clerum.rpc.listSessions.mockImplementation(async () => { + // team-1: held until after the scope change, then succeeds late. + if (scope === 'team-1') { + teamOneReads += 1 + return lateOldScopeSuccess.promise + } + // team-2 is down. + throw unavailable + }) + // The selected-agent read reports catalog success before its own scope + // guard, so it is the path whose stale success must be ignored. + const controller = renderController({ selectedAgent: 'agent-a', agentNames: ['agent-a'] }) + await waitFor(() => expect(teamOneReads).toBeGreaterThanOrEqual(1)) + await waitFor(() => expect(clerum.chat.getIndex.mock.calls.length).toBeGreaterThanOrEqual(2)) + // Let the post-paint selected-agent read attach to the pending request. + await act(async () => { + await new Promise(resolve => setTimeout(resolve, 60)) + }) + + scope = 'team-2' + controller.rerender({ + selectedAgent: 'agent-a', + agentNames: ['agent-a'], + currentTeamId: 'team-2', + }) + await waitFor(() => expect(catalogToasts(controller.spies.pushToast)).toHaveLength(1)) + + await act(async () => { + lateOldScopeSuccess.resolve(serverSessions([{ agent: 'agent-a', chatId: 'old-scope' }])) + await lateOldScopeSuccess.promise + }) + + // Another failure in team-2, via a catalog change that re-reads every Host. + controller.rerender({ + selectedAgent: 'agent-a', + agentNames: ['agent-a', 'agent-b'], + currentTeamId: 'team-2', + }) + await waitFor(() => + expect(clerum.rpc.listSessions).toHaveBeenCalledWith('agent-b', undefined, { + agent: 'agent-b', + limit: 50, + }) + ) + await act(async () => { + await new Promise(resolve => setTimeout(resolve, 20)) + }) + expect(catalogToasts(controller.spies.pushToast)).toHaveLength(1) + expect(controller.result.current.chatList.map(chat => chat.id)).not.toContain('old-scope') + }) +}) + +describe('per-Host authority in the cross-agent list (R2-M5, R1-L11, R1-M10)', () => { + function pendingCatalogs() { + const pending = new Map>>>() + clerum.chat.getIndex.mockResolvedValue(localIndex([])) + clerum.rpc.listSessions.mockImplementation((agentRef: string) => { + const read = deferred() + pending.set(agentRef, [...(pending.get(agentRef) ?? []), read]) + return read.promise + }) + const resolveAll = (agentRef: string) => { + for (const read of pending.get(agentRef) ?? []) { + read.resolve(serverSessions([{ agent: agentRef, chatId: `${agentRef}-session` }])) + } + pending.delete(agentRef) + } + return { pending, resolveAll } + } + + it("keeps Host X's in-flight sessions when Host Y is held", async () => { + const { pending, resolveAll } = pendingCatalogs() + const controller = renderController({ + selectedAgent: null, + agentNames: ['host-x', 'host-y'], + }) + await waitFor(() => expect(pending.has('host-x') && pending.has('host-y')).toBe(true)) + const readsBeforeHold = clerum.rpc.listSessions.mock.calls.length + + controller.hostAuthority.hold('host-y', 'revoked') + await act(async () => { + resolveAll('host-x') + resolveAll('host-y') + }) + + await waitFor(() => + expect(controller.result.current.latestChatSessions.map(chat => chat.id)).toEqual([ + 'host-x-session', + ]) + ) + // A hold reloads nothing: the in-flight read is the one that published. + expect(clerum.rpc.listSessions.mock.calls.length).toBe(readsBeforeHold) + }) + + it('reloads a Host whose sessions were discarded while it was held', async () => { + const { pending, resolveAll } = pendingCatalogs() + const controller = renderController({ selectedAgent: null, agentNames: ['host-y'] }) + await waitFor(() => expect(pending.has('host-y')).toBe(true)) + + controller.hostAuthority.hold('host-y', 'uncertain') + await act(async () => { + resolveAll('host-y') + }) + await waitFor(() => expect(controller.result.current.latestChatSessionsLoading).toBe(false)) + expect(controller.result.current.latestChatSessions).toEqual([]) + + controller.hostAuthority.release('host-y') + // The reload may reuse the renderer's short-lived catalog request cache or + // issue a new read; resolve any new read so either path can publish. + await act(async () => { + await new Promise(resolve => setTimeout(resolve, 20)) + resolveAll('host-y') + }) + await waitFor(() => + expect(controller.result.current.latestChatSessions.map(chat => chat.id)).toEqual([ + 'host-y-session', + ]) + ) + }) + + it('returns the same latestChatSessions reference until its inputs change', async () => { + clerum.chat.getIndex.mockResolvedValue(localIndex([])) + clerum.rpc.listSessions.mockImplementation(async (agentRef: string) => + serverSessions([{ agent: agentRef, chatId: `${agentRef}-session` }]) + ) + const controller = renderController({ + selectedAgent: null, + agentNames: ['host-x', 'host-y'], + }) + await waitFor(() => expect(controller.result.current.latestChatSessions).toHaveLength(2)) + await waitFor(() => expect(controller.result.current.latestChatSessionsLoading).toBe(false)) + + const first = controller.result.current.latestChatSessions + controller.rerender({ selectedAgent: null, agentNames: ['host-x', 'host-y'] }) + expect(controller.result.current.latestChatSessions).toBe(first) + + controller.hostAuthority.hold('host-y', 'uncertain') + const afterHold = controller.result.current.latestChatSessions + expect(afterHold).not.toBe(first) + expect(afterHold.map(chat => chat.id)).toEqual(['host-x-session']) + }) +}) diff --git a/desktop-app/ui/src/hooks/domain/__tests__/chatListTitleMerge.test.tsx b/desktop-app/ui/src/hooks/domain/__tests__/chatListTitleMerge.test.tsx index 5e824d3de7..e3b079bbbb 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/chatListTitleMerge.test.tsx +++ b/desktop-app/ui/src/hooks/domain/__tests__/chatListTitleMerge.test.tsx @@ -10,10 +10,25 @@ * fixtures are derived from the producer instead of hand-built parsed shapes. */ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' -import { waitFor } from '@testing-library/react' -import { parseSessionsListResult } from '../../../../../src/rpcProxyClient' -import type { ChatIndex, SessionsListResult } from '../../../../../src/types' +import { makeTaskKey } from '@contexts/AgentTaskTrackerContext' +import { act, waitFor } from '@testing-library/react' +import type { SessionsListResult } from '../../../../../src/types' +import { + HOST_ACCESS_REVOKED_CODE, + HOST_ACCESS_REVOKED_MESSAGE, +} from '../../../../../src/upstreamErrors' +import { httpErrorStatus, isConfirmedHostAccessRevoked } from '../../../lib/format' +import { deferred, localIndex, serverSessions } from './__fixtures__/catalogFixtures' import { renderController } from './__fixtures__/controllerHarness' +import { + ipcGenericForbidden, + ipcHostAccessDenied, + ipcHostAccessRevoked, + ipcHostWaking, + ipcHttpError, + ipcServerErrorMentioning403, + wrapLikeElectronIpc, +} from './__fixtures__/ipcErrors' import { type MockClerum, installMockClerum, uninstallMockClerum } from './__fixtures__/mockClerum' ;(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true @@ -29,37 +44,96 @@ afterEach(() => { uninstallMockClerum() }) -const NOW = '2026-09-12T00:00:00.000Z' - -/** Parse a raw wire payload through the real producer parser (T1). */ -function serverSessions( - items: Array<{ agent: string; chatId: string; title?: string }> -): SessionsListResult { - return parseSessionsListResult({ - items: items.map(i => ({ - agent: i.agent, - chatId: i.chatId, - turnCount: 1, - lastActivityAt: NOW, - ...(i.title !== undefined ? { title: i.title } : {}), - })), - }) -} - -function localIndex(chats: Array<{ id: string; title: string }>): ChatIndex { - return { - version: 1, - lastActiveChatId: null, - onboardingDismissed: false, - chats: chats.map(c => ({ - id: c.id, - title: c.title, - createdAt: NOW, - updatedAt: NOW, - messageCount: 0, - })), - } -} +describe('HTTP status parsing through Electron IPC', () => { + it.each([ + [ + "Error invoking remote method 'rpc:listSessions': Error: 403 Forbidden: host_access_revoked", + 403, + ], + ["Error invoking remote method 'rpc:listSessions': Error: 401 Unauthorized", 401], + [ + "Error invoking remote method 'rpc:listSessions': Error: 503 Service Unavailable: upstream mentioned 403", + 503, + ], + ["Error invoking remote method 'rpc:invoke': Error: 403 Forbidden", 403], + ['request to support-401 failed', undefined], + ])('parses %s as %s', (message, expected) => { + expect(httpErrorStatus(message)).toBe(expected) + }) + + // Electron never emits the wrapper without the quoted channel, so a + // quote-less prefix is not stripped and no status is read through it. + it('does not strip a wrapper that lacks the quoted channel Electron always emits', () => { + expect( + httpErrorStatus("Error invoking remote method 'rpc:listSessions': Error: 403 Forbidden") + ).toBe(403) + expect(httpErrorStatus('Error invoking remote method: Error: 403 Forbidden')).toBeUndefined() + }) + + // R2-B1 / NEW-dui-3: every shape the real producers emit, wrapped by IPC. + it.each([ + ['listSessions revoked', () => ipcHostAccessRevoked('rpc:listSessions'), 403, true], + [ + 'loadSessionMessages revoked', + () => ipcHostAccessRevoked('rpc:loadSessionMessages'), + 403, + true, + ], + ['renameSession revoked', () => ipcHostAccessRevoked('rpc:renameSession'), 403, true], + ['invokeHostMessage revoked', () => ipcHostAccessRevoked('rpc:invokeHostMessage'), 403, true], + ['getTaskResult revoked', () => ipcHostAccessRevoked('rpc:getTaskResult'), 403, true], + ['listSessions host_access_denied', () => ipcHostAccessDenied('rpc:listSessions'), 403, false], + ['List sessions failed (403)', () => ipcGenericForbidden('rpc:listSessions'), 403, false], + ['Rename session failed (403)', () => ipcGenericForbidden('rpc:renameSession'), 403, false], + ['ApiError 403 Forbidden: …', () => ipcGenericForbidden('rpc:invokeHostMessage'), 403, false], + [ + 'ApiError 401 Unauthorized: …', + () => ipcHttpError('rpc:getTaskResult', 401, 'Unauthorized', { error: 'expired' }), + 401, + false, + ], + [ + 'List sessions 503 body mentions 403', + () => ipcServerErrorMentioning403('rpc:listSessions'), + 503, + false, + ], + [ + 'ApiError 503 body mentions 403', + () => ipcServerErrorMentioning403('rpc:invokeHostMessage'), + 503, + false, + ], + [ + 'host_waking for support-401', + async () => ipcHostWaking('rpc:invokeHostMessage', 'support-401'), + undefined, + false, + ], + ])('classifies the real %s', async (_label, build, status, revoked) => { + const error = await build() + expect(error.message).toMatch(/^Error invoking remote method 'rpc:[A-Za-z]+': Error: /) + expect(httpErrorStatus(error)).toBe(status) + expect(isConfirmedHostAccessRevoked(error)).toBe(revoked) + }) + + // R3-L16: only the upstream projection's exact text confirms a revocation. + // No producer emits a rename-prefixed revocation, so that spelling is not + // accepted either; every case still carries a parsed 403. + it.each([ + ['the IPC-wrapped Host-wide revocation', HOST_ACCESS_REVOKED_MESSAGE, true], + [ + 'a rename-prefixed revocation no producer emits', + `Rename session failed (403): ${HOST_ACCESS_REVOKED_CODE}`, + false, + ], + ['the real generic rename denial', 'Rename session failed (403)', false], + ])('confirms revocation only for %s', (_label, message, revoked) => { + const error = wrapLikeElectronIpc('rpc:renameSession', new Error(message)) + expect(httpErrorStatus(error)).toBe(403) + expect(isConfirmedHostAccessRevoked(error)).toBe(revoked) + }) +}) describe('chat list title merge (spec 15 §2.2, cases A–D)', () => { it('case A: server-only session with a title shows the server title (no placeholder)', async () => { @@ -166,3 +240,436 @@ describe('cross-agent "Latest sessions" title merge (spec 15 §2.2)', () => { ) }) }) + +describe('confirmed deletion catalog protection', () => { + it('keeps a deleted session out of selected-agent and cross-agent catalogs after reload', async () => { + clerum.chat.getIndex.mockImplementation(async (agentRef: string) => + agentRef === 'agent-y' + ? localIndex([], ['deleted-y']) + : localIndex([{ id: 'kept-x', title: 'Kept chat' }], ['deleted-x']) + ) + clerum.rpc.listSessions.mockImplementation(async (agentRef: string) => + agentRef === 'agent-y' + ? serverSessions([{ agent: 'agent-y', chatId: 'deleted-y' }]) + : serverSessions([ + { agent: 'agent-x', chatId: 'deleted-x' }, + { agent: 'agent-x', chatId: 'kept-x' }, + ]) + ) + + const { result } = renderController({ + selectedAgent: 'agent-x', + agentNames: ['agent-x', 'agent-y'], + }) + await waitFor(() => expect(result.current.chatList.map(chat => chat.id)).toEqual(['kept-x'])) + await waitFor(() => expect(result.current.latestChatSessionsLoading).toBe(false)) + expect(result.current.latestChatSessions.map(session => session.id)).not.toContain('deleted-x') + expect(result.current.latestChatSessions.map(session => session.id)).not.toContain('deleted-y') + }) +}) + +describe('revoked host catalog protection', () => { + it('filters held-host sessions from the sidebar while retaining them for recovery', async () => { + clerum.chat.getIndex.mockResolvedValue(localIndex([])) + clerum.rpc.listSessions.mockResolvedValue( + serverSessions([{ agent: 'agent-a', chatId: 'cached-session', title: 'Private title' }]) + ) + const controller = renderController({ + selectedAgent: null, + agentNames: ['agent-a'], + }) + + await waitFor(() => + expect(controller.result.current.latestChatSessions.map(session => session.id)).toEqual([ + 'cached-session', + ]) + ) + + controller.hostAuthority.hold('agent-a', 'uncertain') + expect(controller.hostAuthority.isBlocked('agent-a')).toBe(true) + expect(controller.result.current.latestChatSessions).toEqual([]) + + controller.hostAuthority.release('agent-a') + await waitFor(() => + expect(controller.result.current.latestChatSessions.map(session => session.id)).toEqual([ + 'cached-session', + ]) + ) + }) + + it('hides a nonselected host after 403 and blocks direct reselection', async () => { + const blocked = new Set() + clerum.chat.getIndex.mockResolvedValue(localIndex([])) + clerum.rpc.listSessions.mockImplementation(async (agentRef: string) => + agentRef === 'agent-y' + ? serverSessions([{ agent: 'agent-y', chatId: 'protected-y' }]) + : serverSessions([]) + ) + const controller = renderController({ + selectedAgent: 'agent-x', + agentNames: ['agent-x', 'agent-y'], + onHostAccessRevoked: agentRef => blocked.add(agentRef), + onHostAuthorityUncertain: agentRef => blocked.add(agentRef), + isHostAccessBlocked: agentRef => blocked.has(agentRef), + }) + await waitFor(() => + expect(controller.result.current.latestChatSessions.some(s => s.id === 'protected-y')).toBe( + true + ) + ) + + clerum.rpc.loadSessionMessages.mockRejectedValue( + await ipcHostAccessRevoked('rpc:loadSessionMessages') + ) + await expect( + controller.result.current.reconcileChat(makeTaskKey('agent-y', 'protected-y'), { + reason: 'background_refresh', + }) + ).resolves.toBe('revoked') + expect(blocked.has('agent-y')).toBe(true) + await waitFor(() => + expect(controller.result.current.latestChatSessions.some(s => s.id === 'protected-y')).toBe( + false + ) + ) + + await controller.result.current.switchToChat('agent-y', 'protected-y') + expect(controller.result.current.activeChatId).toBeNull() + expect(clerum.chat.loadMessages).not.toHaveBeenCalledWith( + 'agent-y', + 'protected-y', + expect.anything() + ) + }) + + it.each([ + [ + 'List sessions failed (401)', + () => ipcHttpError('rpc:listSessions', 401, 'Unauthorized', { error: 'expired' }), + ], + ['List sessions failed (403)', () => ipcGenericForbidden('rpc:listSessions')], + ['403 Forbidden: host_access_denied', () => ipcHostAccessDenied('rpc:listSessions')], + ])( + 'holds the Host as uncertain after a generic %s survives the forced retry', + async (_label, build) => { + const revoked = new Set() + const uncertain = new Set() + clerum.chat.getIndex.mockResolvedValue( + localIndex([{ id: 'protected-a', title: 'Cached protected chat' }]) + ) + clerum.rpc.listSessions.mockRejectedValue(await build()) + const { result } = renderController({ + selectedAgent: null, + agentNames: ['agent-a'], + onHostAccessRevoked: agentRef => revoked.add(agentRef), + onHostAuthorityUncertain: agentRef => uncertain.add(agentRef), + isHostAccessBlocked: agentRef => revoked.has(agentRef) || uncertain.has(agentRef), + }) + + await waitFor(() => expect(uncertain.has('agent-a')).toBe(true)) + expect(revoked.has('agent-a')).toBe(false) + expect(clerum.rpc.listSessions).toHaveBeenCalledTimes(2) + expect(result.current.latestChatSessions.some(chat => chat.id === 'protected-a')).toBe(false) + } + ) + + it('revokes the Host on the first confirmed catalog denial without a forced retry', async () => { + const revoked = new Set() + const uncertain = new Set() + clerum.chat.getIndex.mockResolvedValue( + localIndex([{ id: 'protected-a', title: 'Cached protected chat' }]) + ) + clerum.rpc.listSessions.mockRejectedValue(await ipcHostAccessRevoked('rpc:listSessions')) + const { result, spies } = renderController({ + selectedAgent: null, + agentNames: ['agent-a'], + onHostAccessRevoked: agentRef => revoked.add(agentRef), + onHostAuthorityUncertain: agentRef => uncertain.add(agentRef), + isHostAccessBlocked: agentRef => revoked.has(agentRef) || uncertain.has(agentRef), + }) + + await waitFor(() => expect(revoked.has('agent-a')).toBe(true)) + expect(uncertain.has('agent-a')).toBe(false) + expect(clerum.rpc.listSessions).toHaveBeenCalledTimes(1) + expect(result.current.latestChatSessions.some(chat => chat.id === 'protected-a')).toBe(false) + expect(spies.pushToast).not.toHaveBeenCalledWith(expect.stringContaining('Chat list'), 'info') + }) + + it('keeps the selected agent and cached chat list after a 503 catalog failure', async () => { + clerum.chat.getIndex.mockResolvedValue(localIndex([{ id: 'cached-a', title: 'Cached chat' }])) + clerum.rpc.listSessions.mockRejectedValue( + await ipcHttpError('rpc:listSessions', 503, 'Service Unavailable', { + error: 'upstream unavailable', + }) + ) + const controller = renderController({ + selectedAgent: 'agent-a', + agentNames: ['agent-a'], + // R2-L4: the toast names the Host the way the catalog displays it. + agentDisplayName: agentRef => (agentRef === 'agent-a' ? 'Research Agent' : agentRef), + }) + await waitFor(() => expect(clerum.rpc.listSessions).toHaveBeenCalled()) + await waitFor(() => + expect(controller.result.current.chatList.some(chat => chat.id === 'cached-a')).toBe(true) + ) + expect(controller.hostAuthority.isBlocked('agent-a')).toBe(false) + await waitFor(() => expect(controller.spies.pushToast).toHaveBeenCalledTimes(1)) + expect(controller.spies.pushToast).toHaveBeenCalledWith( + 'Chat list for Research Agent is offline. Showing saved chats.', + 'info' + ) + }) + + it('does not show the offline toast for a client-side catalog error', async () => { + clerum.chat.getIndex.mockResolvedValue(localIndex([{ id: 'cached-a', title: 'Cached chat' }])) + clerum.rpc.listSessions.mockRejectedValue( + await ipcHttpError('rpc:listSessions', 400, 'Bad Request', { error: 'invalid cursor' }) + ) + const controller = renderController({ + selectedAgent: 'agent-a', + agentNames: ['agent-a'], + }) + + await waitFor(() => expect(clerum.rpc.listSessions).toHaveBeenCalled()) + await act(async () => { + await Promise.resolve() + }) + expect(controller.spies.pushToast).not.toHaveBeenCalledWith( + expect.stringContaining('Chat list'), + 'info' + ) + }) + + // NEW-dui-2: every non-auth, non-429 client error ends the page chain. + it.each([ + [400, 'Bad Request', { error: 'Invalid sessions cursor' }], + [404, 'Not Found', { error: 'cursor session not found' }], + ])('clears a rejected load-more cursor after a terminal %s', async (status, statusText, body) => { + const rejection = await ipcHttpError('rpc:listSessions', status, statusText, body) + clerum.rpc.listSessions.mockImplementation( + async (_agentRef: string, _teamId: string | undefined, query?: { cursor?: string }) => { + if (query?.cursor) throw rejection + return serverSessions([{ agent: 'agent-x', chatId: 'remote-page-1' }], 'cursor-invalid') + } + ) + const controller = renderController() + + await waitFor(() => expect(controller.result.current.chatListHasMoreRemoteSessions).toBe(true)) + await act(async () => { + await controller.result.current.loadMoreChatSessions() + }) + expect(controller.result.current.chatListHasMoreRemoteSessions).toBe(false) + + // The cursor is gone: a second request never re-sends it. + await act(async () => { + await controller.result.current.loadMoreChatSessions() + }) + expect( + clerum.rpc.listSessions.mock.calls.filter( + call => (call[2] as { cursor?: string } | undefined)?.cursor === 'cursor-invalid' + ) + ).toHaveLength(1) + expect(controller.spies.pushToast).not.toHaveBeenCalledWith( + expect.stringContaining("Couldn't load more chats"), + 'info' + ) + }) + + // R3-L10: the main process validates the cursor before any request and throws + // without an HTTP status (`ipc.ts` sanitizeSessionsListQuery). Re-sending the + // same cursor repeats that refusal, so it ends the page chain like a 400. + it('clears a load-more cursor the main process refuses without an HTTP status', async () => { + const rejection = wrapLikeElectronIpc('rpc:listSessions', new Error('Invalid sessions cursor')) + expect(httpErrorStatus(rejection)).toBeUndefined() + clerum.rpc.listSessions.mockImplementation( + async (_agentRef: string, _teamId: string | undefined, query?: { cursor?: string }) => { + if (query?.cursor) throw rejection + return serverSessions([{ agent: 'agent-x', chatId: 'remote-page-1' }], 'cursor-refused') + } + ) + const controller = renderController() + + await waitFor(() => expect(controller.result.current.chatListHasMoreRemoteSessions).toBe(true)) + await act(async () => { + await controller.result.current.loadMoreChatSessions() + }) + await act(async () => { + await controller.result.current.loadMoreChatSessions() + }) + + // Witness: the refused cursor was sent exactly once. + expect( + clerum.rpc.listSessions.mock.calls.filter( + call => (call[2] as { cursor?: string } | undefined)?.cursor === 'cursor-refused' + ) + ).toHaveLength(1) + expect(controller.result.current.chatListHasMoreRemoteSessions).toBe(false) + expect(controller.spies.pushToast).not.toHaveBeenCalledWith( + expect.stringContaining("Couldn't load more chats"), + 'info' + ) + }) + + // NEW-dui-2: 408, 429 and 5xx keep the cursor, tell the user once per attempt + // and never retry on their own. A 408 is retryable (RFC 9110 §15.5.9). + it.each([ + [408, 'Request Timeout'], + [429, 'Too Many Requests'], + [503, 'Service Unavailable'], + ])('preserves the load-more cursor after a retryable %s', async (status, statusText) => { + const rejection = await ipcHttpError('rpc:listSessions', status, statusText, { + error: 'try later', + }) + clerum.rpc.listSessions.mockImplementation( + async (_agentRef: string, _teamId: string | undefined, query?: { cursor?: string }) => { + if (query?.cursor) throw rejection + return serverSessions([{ agent: 'agent-x', chatId: 'remote-page-1' }], 'cursor-retryable') + } + ) + const controller = renderController({ + agentDisplayName: agentRef => (agentRef === 'agent-x' ? 'Agent X' : agentRef), + }) + + await waitFor(() => expect(controller.result.current.chatListHasMoreRemoteSessions).toBe(true)) + await act(async () => { + await controller.result.current.loadMoreChatSessions() + }) + expect(controller.result.current.chatListHasMoreRemoteSessions).toBe(true) + expect(controller.spies.pushToast).toHaveBeenCalledTimes(1) + expect(controller.spies.pushToast).toHaveBeenCalledWith( + "Couldn't load more chats for Agent X. Try again shortly.", + 'info' + ) + + await act(async () => { + await controller.result.current.loadMoreChatSessions() + }) + await act(async () => { + await new Promise(resolve => setTimeout(resolve, 20)) + }) + + // Exactly the two user-initiated attempts: no automatic retry loop. + expect( + clerum.rpc.listSessions.mock.calls.filter( + call => (call[2] as { cursor?: string } | undefined)?.cursor === 'cursor-retryable' + ) + ).toHaveLength(2) + expect(controller.spies.pushToast).toHaveBeenCalledTimes(2) + }) +}) + +describe('host authority verification', () => { + // R3-M3: the epoch negative and its same-epoch control share one IPC-shaped + // revocation (never retried, so it reaches the classifier directly); the + // control proves the denial holds the Host when the epoch does not advance. + it.each([ + [ + 'discards a pre-verification authorization denial after the host authority epoch advances', + 1, + false, + ], + ['holds the Host on a catalog denial when the host authority epoch is unchanged', 0, true], + ] as const)('%s', async (_label, epochAdvance, expectedRevoked) => { + let hostAuthorityEpoch = 0 + const revoked = new Set() + const uncertain = new Set() + clerum.chat.getIndex.mockResolvedValue(localIndex([])) + let rejectFirst!: (reason: unknown) => void + let firstCatalogRead = true + clerum.rpc.listSessions.mockImplementation(() => { + if (!firstCatalogRead) return Promise.resolve(serverSessions([])) + firstCatalogRead = false + return new Promise((_resolve, reject) => { + rejectFirst = reject + }) + }) + + const controller = renderController({ + selectedAgent: 'agent-x', + agentNames: ['agent-x'], + onHostAccessRevoked: agentRef => revoked.add(agentRef), + onHostAuthorityUncertain: agentRef => uncertain.add(agentRef), + isHostAccessBlocked: agentRef => revoked.has(agentRef) || uncertain.has(agentRef), + getHostAuthorityEpoch: () => hostAuthorityEpoch, + }) + + const denial = await ipcHostAccessRevoked('rpc:listSessions') + await waitFor(() => expect(clerum.rpc.listSessions).toHaveBeenCalled()) + hostAuthorityEpoch += epochAdvance + rejectFirst(denial) + await waitFor(() => expect(controller.result.current.latestChatSessionsLoading).toBe(false)) + await act(async () => { + await new Promise(resolve => setTimeout(resolve, 20)) + }) + expect(revoked.has('agent-x')).toBe(expectedRevoked) + expect(uncertain.has('agent-x')).toBe(false) + }) +}) + +describe('late global catalog response', () => { + it.each(['revocation', 'confirmed deletion'])( + 'does not republish Host A after %s while Host B is pending', + async action => { + const blocked = new Set() + const deliveredA = deferred() + const pendingB = deferred() + clerum.chat.getIndex.mockResolvedValue(localIndex([])) + clerum.rpc.listSessions.mockImplementation(async (agentRef: string) => { + if (agentRef === 'agent-a') { + deliveredA.resolve() + return serverSessions([{ agent: 'agent-a', chatId: 'session-a' }]) + } + return pendingB.promise + }) + const controller = renderController({ + selectedAgent: null, + agentNames: ['agent-a', 'agent-b'], + onHostAccessRevoked: agentRef => blocked.add(agentRef), + onHostAuthorityUncertain: agentRef => blocked.add(agentRef), + isHostAccessBlocked: agentRef => blocked.has(agentRef), + }) + await deliveredA.promise + await waitFor(() => + expect(clerum.rpc.listSessions).toHaveBeenCalledWith('agent-b', undefined, { + agent: 'agent-b', + limit: 50, + }) + ) + + if (action === 'revocation') { + clerum.rpc.loadSessionMessages.mockRejectedValue( + await ipcHostAccessRevoked('rpc:loadSessionMessages') + ) + await act(async () => { + await controller.result.current.reconcileChat(makeTaskKey('agent-a', 'session-a'), { + reason: 'background_refresh', + }) + }) + expect(blocked.has('agent-a')).toBe(true) + } else { + await act(async () => { + const deletion = await controller.result.current.captureChatDeleteFence('agent-a') + await controller.result.current.handleDeleteChatForAgent('agent-a', 'session-a', deletion) + }) + expect(clerum.chat.delete).toHaveBeenCalledWith( + 'agent-a', + 'session-a', + expect.objectContaining({ version: 1, bindingGeneration: 1 }) + ) + } + + await act(async () => { + pendingB.resolve(serverSessions([{ agent: 'agent-b', chatId: 'session-b' }])) + await pendingB.promise + }) + await waitFor(() => + expect( + controller.result.current.latestChatSessions.some(chat => chat.id === 'session-b') + ).toBe(true) + ) + expect( + controller.result.current.latestChatSessions.some(chat => chat.id === 'session-a') + ).toBe(false) + } + ) +}) diff --git a/desktop-app/ui/src/hooks/domain/__tests__/imageInputJourney.test.tsx b/desktop-app/ui/src/hooks/domain/__tests__/imageInputJourney.test.tsx index f3f1d90278..1369a17ad3 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/imageInputJourney.test.tsx +++ b/desktop-app/ui/src/hooks/domain/__tests__/imageInputJourney.test.tsx @@ -73,6 +73,17 @@ async function mountedVisualController() { return rendered } +// The terminal handler releases the chat's task only after the reply is written +// to the store, and the real ChatStore writes it to disk. A second send issued +// before that write lands is refused while the first task still holds the chat. +async function waitForPersistedReply(chatId: string) { + await waitFor(async () => + expect( + (await bridge.persistedMessages('agent-x', chatId)).map(message => message.role) + ).toEqual(['user', 'assistant']) + ) +} + describe('#654 visual send and recovery', () => { it('blocks pending images after selecting a text-only model, before creating a chat', async () => { const { result } = await mountedVisualController() @@ -229,6 +240,7 @@ describe('#654 visual send and recovery', () => { }) }) await waitFor(() => expect(bridge.rpc.getTaskResult).toHaveBeenCalled()) + await waitForPersistedReply(chatId) // The chat now exists, so its own catalog read settles — carrying the server's // OWN revision 0, which is older than the ack and must not overwrite it. @@ -288,6 +300,7 @@ describe('#654 visual send and recovery', () => { }) }) await waitFor(() => expect(bridge.rpc.getTaskResult).toHaveBeenCalled()) + await waitForPersistedReply(chatId) await act(async () => { await loadHostModels(modelTransport, 'agent-x', chatId) @@ -331,6 +344,9 @@ describe('#654 visual send and recovery', () => { }) expect(result.current.failedAgentSend).toBeNull() expect(result.current.agentError).toBeNull() + // Witness: the outgoing turn was persisted once, before the POST. The real + // store keeps that write in `upsertMessages`; the failure path added none. + expect(bridge.chat.upsertMessages).toHaveBeenCalledTimes(1) expect(bridge.chat.appendMessages).not.toHaveBeenCalled() }) }) diff --git a/desktop-app/ui/src/hooks/domain/__tests__/mockClerum.test.ts b/desktop-app/ui/src/hooks/domain/__tests__/mockClerum.test.ts new file mode 100644 index 0000000000..dfe6b661e9 --- /dev/null +++ b/desktop-app/ui/src/hooks/domain/__tests__/mockClerum.test.ts @@ -0,0 +1,57 @@ +// @vitest-environment jsdom +/** + * R1-M11: the `window.clerum.chat` fixture must persist exactly like the real + * `ChatStore` behind the IPC bridge. A hand-rolled merge that drifts from it + * certifies suites against a store production never runs. + */ +import { afterEach, describe, expect, it } from 'vitest' +import type { ChatAuthorityScope } from '../../../../../src/types' +import { type MockClerum, installMockClerum, uninstallMockClerum } from './__fixtures__/mockClerum' + +const AGENT = 'agent-x' +const CHAT = 'chat-1' +const SCOPE: ChatAuthorityScope = { environmentKey: 'env-test', userId: 'user-1', teamId: null } + +async function persistedIds(clerum: MockClerum): Promise { + return (await clerum.persistedMessages(AGENT, CHAT)).map(message => message.id) +} + +describe('mockClerum chat persistence', () => { + afterEach(() => { + uninstallMockClerum() + }) + + it('reconciles a turnless local message with its server turn like ChatStore', async () => { + const clerum = installMockClerum() + await clerum.chat.create(AGENT, CHAT) + await clerum.chat.appendMessages(AGENT, CHAT, [ + { id: 'local-1', role: 'user', content: 'hello', timestamp: 1 }, + ]) + await clerum.chat.upsertMessages(AGENT, CHAT, [ + { id: 'srv-1', role: 'user', content: 'hello', timestamp: 2, serverTurnNumber: 1 }, + ]) + + expect(await persistedIds(clerum)).toEqual(['srv-1']) + // Witness: the bridge's own read returns the real store's result. + const loaded = (await clerum.chat.loadMessages(AGENT, CHAT)) as Array<{ id: string }> + expect(loaded.map(message => message.id)).toEqual(['srv-1']) + }) + + it('drops a late upsert into a chat deleted in the current scope like ChatStore', async () => { + const clerum = installMockClerum() + await clerum.chat.create(AGENT, CHAT) + await clerum.chat.appendMessages(AGENT, CHAT, [ + { id: 'before', role: 'user', content: 'kept until delete', timestamp: 1 }, + ]) + // Witness: the chat held a message before the delete. + expect(await persistedIds(clerum)).toEqual(['before']) + + const fence = await clerum.chat.captureDeleteFence(SCOPE) + await clerum.chat.delete(AGENT, CHAT, fence) + await clerum.chat.upsertMessages(AGENT, CHAT, [ + { id: 'late', role: 'assistant', content: 'late reply', timestamp: 3 }, + ]) + + expect(await persistedIds(clerum)).toEqual([]) + }) +}) diff --git a/desktop-app/ui/src/hooks/domain/__tests__/pendingChatSelection.characterization.test.tsx b/desktop-app/ui/src/hooks/domain/__tests__/pendingChatSelection.characterization.test.tsx index 2584ab8066..7e0a7078a4 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/pendingChatSelection.characterization.test.tsx +++ b/desktop-app/ui/src/hooks/domain/__tests__/pendingChatSelection.characterization.test.tsx @@ -127,12 +127,16 @@ describe('pendingChatSelection effect', () => { }) await waitFor(() => expect(result.current.activeChatId).toBe('c-latest')) - expect(clerum.rpc.loadSessionMessages).toHaveBeenCalledWith( - 'agent-x', - 'agent-x', - 'c-latest', - undefined, - { limit: 40 } + // The remote load follows the local store read, which crosses IPC (here, the + // real ChatStore's disk read), so it lands after the selection flips. + await waitFor(() => + expect(clerum.rpc.loadSessionMessages).toHaveBeenCalledWith( + 'agent-x', + 'agent-x', + 'c-latest', + undefined, + { limit: 40 } + ) ) }) @@ -161,12 +165,14 @@ describe('pendingChatSelection effect', () => { }) await waitFor(() => expect(result.current.activeChatId).toBe('c-newest')) - expect(clerum.rpc.loadSessionMessages).toHaveBeenCalledWith( - 'agent-x', - 'agent-x', - 'c-newest', - undefined, - { limit: 40 } + await waitFor(() => + expect(clerum.rpc.loadSessionMessages).toHaveBeenCalledWith( + 'agent-x', + 'agent-x', + 'c-newest', + undefined, + { limit: 40 } + ) ) }) diff --git a/desktop-app/ui/src/hooks/domain/__tests__/reconcileChat.test.ts b/desktop-app/ui/src/hooks/domain/__tests__/reconcileChat.test.ts index 29191d7bf9..f8d3da40cc 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/reconcileChat.test.ts +++ b/desktop-app/ui/src/hooks/domain/__tests__/reconcileChat.test.ts @@ -25,6 +25,8 @@ function liveResp(overrides: Partial = {}): SessionMessag class NetErr extends Error {} class NotFound extends Error {} +class Unavailable extends Error {} +class Unauthorized extends Error {} function buildDeps(overrides: Partial = {}): ReconcileChatDeps { const fsm = createSessionFsmStore() @@ -35,9 +37,13 @@ function buildDeps(overrides: Partial = {}): ReconcileChatDep // the view (so a rejoin can anchor to the rendered bubble, P1-A) before attach. attachLiveTask: vi.fn(async () => 'reconcile_rejoined' as const), settleIdle: vi.fn(async () => 'fell_through_to_resend' as const), - evictChat: vi.fn(async () => {}), + revokeAccess: vi.fn(), + holdAccess: vi.fn(), isNetworkError: (e): e is NetErr => e instanceof NetErr, isHttp404: (e): e is NotFound => e instanceof NotFound, + isAuthorizationError: (e): e is Unauthorized => e instanceof Unauthorized, + isConfirmedHostAccessRevoked: e => + e instanceof Unauthorized && e.message === 'verified-host-403', telemetry: vi.fn(), networkRetryBackoffMs: [0, 0, 0], ...overrides, @@ -78,22 +84,51 @@ describe('reconcileChat — precedence branches', () => { }) }) - it('404 evicts the chat, dispatches RESET, and does NOT re-finish', async () => { + it.each([ + ['transient 404', new NotFound('not found during wake')], + ['503', new Unavailable('service unavailable')], + ['timeout', new NetErr('timeout')], + ])('%s retains the selected conversation and draft state', async (_name, error) => { const deps = buildDeps({ loadSessionMessages: vi.fn(async () => { - throw new NotFound('gone') + throw error }), }) - // Seed an entry so we can observe RESET removing it. + // The FSM entry represents the selected conversation. A destructive reset + // would also cause the controller to clear its conversation and composer. deps.fsm.dispatch(chatKey, { type: 'SEND_STARTED', taskId: 't1' }) const reconcile = createReconcileChat(deps) await reconcile(chatKey, { reason: 'user_refresh' }) - expect(deps.evictChat).toHaveBeenCalledWith(chatKey) - expect(deps.fsm.getState(chatKey)).toBeUndefined() // RESET removed it, FINISHED skipped - expect(deps.telemetry).toHaveBeenCalledWith('stream_recovery', { - reason: 'user_refresh', - outcome: '404', + expect(deps.revokeAccess).not.toHaveBeenCalled() + expect(deps.fsm.getState(chatKey)).toBeDefined() + }) + + it.each([401, 403])('%i holds protected data pending renewed authority', async status => { + const deps = buildDeps({ + loadSessionMessages: vi.fn(async () => { + throw new Unauthorized(String(status)) + }), }) + deps.fsm.dispatch(chatKey, { type: 'SEND_STARTED', taskId: 't1' }) + const reconcile = createReconcileChat(deps) + await expect(reconcile(chatKey, { reason: 'user_refresh' })).resolves.toBe( + 'authority_unverified' + ) + expect(deps.holdAccess).toHaveBeenCalledWith(chatKey) + expect(deps.revokeAccess).not.toHaveBeenCalled() + expect(deps.fsm.getState(chatKey)).toBeUndefined() + }) + + it('revokes immediately for a verified Host denial', async () => { + const deps = buildDeps({ + loadSessionMessages: vi.fn(async () => { + throw new Unauthorized('verified-host-403') + }), + }) + const reconcile = createReconcileChat(deps) + await expect(reconcile(chatKey, { reason: 'user_refresh' })).resolves.toBe('revoked') + expect(deps.revokeAccess).toHaveBeenCalledWith(chatKey) + expect(deps.holdAccess).not.toHaveBeenCalled() }) it('exhausted network retries → WENT_OFFLINE + offline outcome', async () => { @@ -234,7 +269,7 @@ describe('reconcileChat — single-flight', () => { ) }) - it('drops a late 404 after reset instead of evicting the next session cache', async () => { + it('drops a late 404 after reset without changing the next session cache', async () => { let rejectLoad: (reason: unknown) => void = () => {} const load = vi.fn( (_agentRef: string, _chatId: string, _query: unknown, stillRelevant?: () => boolean) => { @@ -255,10 +290,53 @@ describe('reconcileChat — single-flight', () => { rejectLoad(new NotFound('late response')) await expect(pending).resolves.toBe('noop') - expect(deps.evictChat).not.toHaveBeenCalled() + expect(deps.revokeAccess).not.toHaveBeenCalled() expect(deps.fsm.getState(chatKey)).toBeUndefined() }) + it('holds a Host when an old chat read returns generic 403 after selection moves', async () => { + let rejectLoad: (reason: unknown) => void = () => {} + let firstChatSelected = true + const deps = buildDeps({ + loadSessionMessages: vi.fn( + () => + new Promise((_resolve, reject) => { + rejectLoad = reject + }) + ), + }) + const reconcile = createReconcileChat(deps) + const pending = reconcile(chatKey, { + reason: 'switch_to_chat', + isRelevant: () => firstChatSelected, + }) + firstChatSelected = false + rejectLoad(new Unauthorized('403 forbidden')) + + await expect(pending).resolves.toBe('authority_unverified') + expect(deps.holdAccess).toHaveBeenCalledWith(chatKey) + }) + + it('ignores a late 403 after reset tears down the principal scope', async () => { + let rejectLoad: (reason: unknown) => void = () => {} + const deps = buildDeps({ + loadSessionMessages: vi.fn( + () => + new Promise((_resolve, reject) => { + rejectLoad = reject + }) + ), + }) + const reconcile = createReconcileChat(deps) + const pending = reconcile(chatKey, { reason: 'old-principal' }) + reconcile.reset() + deps.fsm.reset() + rejectLoad(new Unauthorized('403 forbidden')) + + await expect(pending).resolves.toBe('stale_drop') + expect(deps.revokeAccess).not.toHaveBeenCalled() + }) + it('a coalesced caller donates its taskIdHint to a hint-less in-flight run (M6)', async () => { let resolveFirst: (v: SessionMessagesResult) => void = () => {} const load = vi diff --git a/desktop-app/ui/src/hooks/domain/__tests__/renameSessionQueue.test.tsx b/desktop-app/ui/src/hooks/domain/__tests__/renameSessionQueue.test.tsx index 075d63c2d0..3d2908e067 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/renameSessionQueue.test.tsx +++ b/desktop-app/ui/src/hooks/domain/__tests__/renameSessionQueue.test.tsx @@ -14,8 +14,9 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { act, cleanup, waitFor } from '@testing-library/react' import { parseSessionsListResult } from '../../../../../src/rpcProxyClient' import type { ChatIndex } from '../../../../../src/types' -import { MAX_RENAME_SYNC_ATTEMPTS } from '../useChatListController' +import { MAX_RENAME_SYNC_ATTEMPTS, deletedChatIdsForScope } from '../useChatListController' import { renderController } from './__fixtures__/controllerHarness' +import { ipcGenericForbidden, ipcHostAccessRevoked, ipcHttpError } from './__fixtures__/ipcErrors' import { type MockClerum, installMockClerum, uninstallMockClerum } from './__fixtures__/mockClerum' ;(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true @@ -53,6 +54,42 @@ function localIndex(chats: Array<{ id: string; title: string }>): ChatIndex { } } +describe('scoped deletion visibility', () => { + // Same identity as main's `sameChatDeletionIdentity`: environment + user. A + // deletion confirmed under one team keeps holding after a team switch. + it('filters tombstones by environment and user, across team switches', () => { + const index: ChatIndex = { + ...localIndex([]), + deletedChatTombstones: [ + { + chatId: 'team-a-delete', + authorityScope: { environmentKey: 'env-1', userId: 'user-1', teamId: 'team-a' }, + }, + { + chatId: 'other-environment-delete', + authorityScope: { environmentKey: 'env-2', userId: 'user-1', teamId: 'team-a' }, + }, + { + chatId: 'other-user-delete', + authorityScope: { environmentKey: 'env-1', userId: 'user-2', teamId: 'team-a' }, + }, + ], + } + + for (const teamId of ['team-a', 'team-b', null]) { + expect( + deletedChatIdsForScope(index, { environmentKey: 'env-1', userId: 'user-1', teamId }) + ).toEqual(['team-a-delete']) + } + expect( + deletedChatIdsForScope(index, { environmentKey: 'env-2', userId: 'user-1', teamId: null }) + ).toEqual(['other-environment-delete']) + expect( + deletedChatIdsForScope(index, { environmentKey: 'env-1', userId: 'user-2', teamId: 'x' }) + ).toEqual(['other-user-delete']) + }) +}) + function titleInList(current: { chatList: Array<{ id: string; title: string }> }, id: string) { return current.chatList.find(c => c.id === id)?.title } @@ -118,6 +155,87 @@ function reportedSessions(chats: Array<{ chatId: string; title?: string }>) { } describe('rename pending queue (spec 15 §2.5)', () => { + // R2-B1: the rename and read failures are the real RpcProxyClient rejections + // wrapped the way Electron IPC wraps them. + it.each([ + { + label: 'exact Host denial', + renameError: () => ipcHostAccessRevoked('rpc:renameSession'), + readError: null, + shouldRevoke: true, + shouldBeUncertain: false, + }, + { + label: 'operation-scope denial', + renameError: () => ipcGenericForbidden('rpc:renameSession'), + readError: null, + shouldRevoke: false, + shouldBeUncertain: false, + }, + { + label: 'read also denied', + renameError: () => ipcGenericForbidden('rpc:renameSession'), + readError: () => ipcGenericForbidden('rpc:listSessions'), + shouldRevoke: false, + shouldBeUncertain: true, + }, + { + label: 'read revoked', + renameError: () => ipcGenericForbidden('rpc:renameSession'), + readError: () => ipcHostAccessRevoked('rpc:listSessions'), + shouldRevoke: true, + shouldBeUncertain: false, + }, + { + label: 'read unavailable', + renameError: () => ipcGenericForbidden('rpc:renameSession'), + readError: () => + ipcHttpError('rpc:listSessions', 503, 'Service Unavailable', { error: 'unavailable' }), + shouldRevoke: false, + shouldBeUncertain: true, + }, + ])( + '$label applies the transcript authority decision', + async ({ renameError, readError, shouldRevoke, shouldBeUncertain }) => { + const revoked = new Set() + const uncertain = new Set() + const renameFailure = await renameError() + const readFailure = readError ? await readError() : null + clerum.chat.getIndex.mockResolvedValue(localIndex([{ id: 'c1', title: 'old' }])) + clerum.rpc.renameSession.mockRejectedValue(renameFailure) + clerum.rpc.listSessions.mockImplementation( + async (_hostRef: string, _teamId: string | undefined, query?: { limit?: number }) => { + if (query?.limit === 1 && readFailure !== null) throw readFailure + return reportedSessions([{ chatId: 'c1', title: 'old' }]) + } + ) + const { result, spies } = renderController({ + selectedAgent: 'agent-x', + agentNames: ['agent-x'], + onHostAccessRevoked: agentRef => revoked.add(agentRef), + onHostAuthorityUncertain: agentRef => uncertain.add(agentRef), + isHostAccessBlocked: agentRef => revoked.has(agentRef), + }) + await waitFor(() => expect(titleInList(result.current, 'c1')).toBe('old')) + await act(async () => { + await result.current.handleRenameChatForAgent('agent-x', 'c1', 'new') + }) + // Liveness witness: the rename reached the producer. + expect(clerum.rpc.renameSession).toHaveBeenCalledWith('agent-x', 'agent-x', 'c1', 'new') + expect(revoked.has('agent-x')).toBe(shouldRevoke) + expect(uncertain.has('agent-x')).toBe(shouldBeUncertain) + if (shouldRevoke) expect(result.current.chatList).toEqual([]) + else if (shouldBeUncertain) expect(titleInList(result.current, 'c1')).toBe('new') + else { + expect(titleInList(result.current, 'c1')).toBe('old') + expect(spies.pushToast).toHaveBeenCalledWith( + 'You do not have permission to rename this chat.', + 'error' + ) + } + } + ) + it('200: optimistic local title + RPC sync (hostRef===agent===agentRef), no toast', async () => { clerum.chat.getIndex.mockResolvedValue(localIndex([{ id: 'c1', title: 'old' }])) clerum.rpc.renameSession.mockResolvedValue({ title: 'renamed' }) @@ -222,7 +340,7 @@ describe('rename pending queue (spec 15 §2.5)', () => { // Next poll reports c1 → the queue retries the 404'd rename. await act(async () => { - rerender({ agentNames: ['agent-x'] }) + rerender({ agentNames: ['agent-x', 'agent-y'] }) }) await waitFor(() => expect(clerum.rpc.renameSession).toHaveBeenCalledTimes(2)) // Still shows the rename (never clobbered by the server title). @@ -290,11 +408,11 @@ describe('rename pending queue — concurrency (spec 15 §2.5)', () => { // trick drives a retry in the 404 case — see the T5 test — so it does invoke // flushPendingRenames; here the in-flight guard must make it a no-op.) await act(async () => { - rerender({ agentNames: ['agent-x'] }) + rerender({ agentNames: ['agent-x', 'agent-y'] }) await flushMicrotasks() }) await act(async () => { - rerender({ agentNames: ['agent-x'] }) + rerender({ agentNames: ['agent-x', 'agent-y'] }) await flushMicrotasks() }) @@ -651,7 +769,7 @@ describe('rename pending queue — bounded retry (spec 15 §2.5 / R1-M1)', () => reportedSessions([{ chatId: 'c1', title: 'server' }]) ) await act(async () => { - rerender({ agentNames: ['agent-x'] }) + rerender({ agentNames: ['agent-x', 'agent-y'] }) await flushMicrotasks() }) await waitFor(() => expect(clerum.rpc.renameSession).toHaveBeenCalledTimes(2)) diff --git a/desktop-app/ui/src/hooks/domain/__tests__/sendAgentMessage.characterization.test.tsx b/desktop-app/ui/src/hooks/domain/__tests__/sendAgentMessage.characterization.test.tsx index 3958fa963e..81a08120b3 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/sendAgentMessage.characterization.test.tsx +++ b/desktop-app/ui/src/hooks/domain/__tests__/sendAgentMessage.characterization.test.tsx @@ -49,8 +49,16 @@ describe('sendAgentMessage — synchronous (non-async) response', () => { // No async task → no progress subscription was ever opened. expect(clerum.rpc.subscribeTaskProgress).not.toHaveBeenCalled() - // user message + assistant reply persisted. - expect(clerum.chat.appendMessages).toHaveBeenCalledTimes(2) + // user message (upserted before the POST) + assistant reply (appended) are + // both in the store. + const chatId = result.current.activeChatId! + expect( + (await clerum.persistedMessages('agent-x', chatId)).map(m => [m.role, m.content]) + ).toEqual([ + ['user', 'quick question'], + ['assistant', 'direct answer'], + ]) + expect(clerum.chat.appendMessages).toHaveBeenCalledTimes(1) const assistant = clerum.chat.appendMessages.mock.calls.at(-1)?.[2] as | Array<{ role?: string; content?: string; isError?: boolean }> | undefined @@ -96,7 +104,7 @@ describe('sendAgentMessage — POST failure', () => { expect(result.current.failedAgentSend?.kind).toBe('network') // Late persist: the POST threw before the single post-taskId persist, so the // user's typed input is still written to the store (best-effort durability). - expect(clerum.chat.appendMessages).toHaveBeenCalledWith( + expect(clerum.chat.upsertMessages).toHaveBeenCalledWith( 'agent-x', expect.any(String), expect.arrayContaining([ @@ -218,8 +226,45 @@ describe('sendAgentMessage — auto-title on send (B10)', () => { expect(clerum.chat.rename).toHaveBeenCalledWith( 'agent-x', createdChatId, - 'summarize the quarterly report' + 'summarize the quarterly report', + 1 ) ) }) }) + +// R1-M11: the fake store must merge a second write of the same turn the way +// `ChatStore.upsertMessages` does. The send path persists the user turn before +// the POST and again with its task_id once the Host accepts it; a fake that +// drops the second write hides a lost task_id. +describe('sendAgentMessage — outgoing user turn persistence', () => { + it('keeps the task_id the acknowledgement adds to the persisted user turn', async () => { + clerum.rpc.invokeHostMessage.mockResolvedValue({ taskId: 'task-77' }) + const { result } = renderController() + await settleMount() + + await act(async () => { + await result.current.handleSendAgentMessage('hello') + }) + + // Witness: the same turn was written twice, the second time with its task. + await waitFor(() => expect(clerum.chat.upsertMessages).toHaveBeenCalledTimes(2)) + const [agentRef, chatId, firstWrite] = clerum.chat.upsertMessages.mock.calls[0] as [ + string, + string, + Array<{ id: string }>, + ] + const turnId = firstWrite[0]?.id + expect(clerum.chat.upsertMessages.mock.calls[1]?.[2]?.[0]).toMatchObject({ + id: turnId, + task_id: 'task-77', + }) + + const persistedUserTurns = (await clerum.persistedMessages(agentRef, chatId)).filter( + message => message.role === 'user' + ) + expect(persistedUserTurns).toEqual([ + expect.objectContaining({ id: turnId, content: 'hello', task_id: 'task-77' }), + ]) + }) +}) diff --git a/desktop-app/ui/src/hooks/domain/__tests__/sendAgentMessage.fileReferences.test.tsx b/desktop-app/ui/src/hooks/domain/__tests__/sendAgentMessage.fileReferences.test.tsx index 70f9bee063..8606e28b1e 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/sendAgentMessage.fileReferences.test.tsx +++ b/desktop-app/ui/src/hooks/domain/__tests__/sendAgentMessage.fileReferences.test.tsx @@ -170,7 +170,9 @@ function expectAcceptedAck( ]) return } - expect(clerum.chat.appendMessages).toHaveBeenCalledWith( + // The async ack persists the user turn with its task_id through the + // idempotent upsert (the turn was already written once before the send). + expect(clerum.chat.upsertMessages).toHaveBeenCalledWith( 'agent-x', expect.any(String), expect.arrayContaining([expect.objectContaining({ role: 'user', task_id: 'task-refs' })]) diff --git a/desktop-app/ui/src/hooks/domain/__tests__/sessionStateRecovery.characterization.test.tsx b/desktop-app/ui/src/hooks/domain/__tests__/sessionStateRecovery.characterization.test.tsx index 8065de5127..edff824e06 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/sessionStateRecovery.characterization.test.tsx +++ b/desktop-app/ui/src/hooks/domain/__tests__/sessionStateRecovery.characterization.test.tsx @@ -250,9 +250,12 @@ describe('onTrackerTerminal — recover via reconcile catch', () => { | undefined expect(appended?.[0]?.errorCode).toBe('BUDGET_EXCEEDED') expect(appended?.[0]?.content).toBe('Budget exceeded') - expect(spies.pushToast).toHaveBeenCalledWith( - 'Message to agent-x failed: Budget exceeded', - 'error' + // The failure toast follows the store write of the error reply. + await waitFor(() => + expect(spies.pushToast).toHaveBeenCalledWith( + 'Message to agent-x failed: Budget exceeded', + 'error' + ) ) // A durable failure still retains the input for explicit recovery/retry. expect(result.current.failedAgentSend).toMatchObject({ content: 'spend' }) diff --git a/desktop-app/ui/src/hooks/domain/__tests__/switchToChat.unified.test.tsx b/desktop-app/ui/src/hooks/domain/__tests__/switchToChat.unified.test.tsx index b1359eb4e6..3439f29033 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/switchToChat.unified.test.tsx +++ b/desktop-app/ui/src/hooks/domain/__tests__/switchToChat.unified.test.tsx @@ -10,7 +10,14 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { makeTaskKey } from '@contexts/AgentTaskTrackerContext/types' import { act, waitFor } from '@testing-library/react' +import { + getComposerDraft, + resetComposerDraftStore, + setComposerDraft, +} from '@lib/composerDraftStore' +import { localIndex, serverSessions } from './__fixtures__/catalogFixtures' import { renderController } from './__fixtures__/controllerHarness' +import { ipcGenericForbidden, ipcHostAccessRevoked } from './__fixtures__/ipcErrors' import { type MockClerum, installMockClerum, uninstallMockClerum } from './__fixtures__/mockClerum' let clerum: MockClerum @@ -29,6 +36,7 @@ beforeEach(() => { afterEach(() => { vi.restoreAllMocks() vi.useRealTimers() + resetComposerDraftStore() uninstallMockClerum() }) @@ -52,6 +60,8 @@ function deferred() { return { promise, resolve } } +type SessionResult = Awaited> + describe('switchToChat (unified, D.4)', () => { it('Phase 1 renders the cache, Phase 2 reconciles, and preserves the cache when there is no diff', async () => { const cached = [ @@ -78,7 +88,7 @@ describe('switchToChat (unified, D.4)', () => { 'agent-x', 'c1', undefined, - { limit: 40, afterTurn: 1 } + { limit: 40, afterTurn: 0 } ) expect(clerum.chat.replaceMessages).not.toHaveBeenCalled() expect(result.current.chatMessages).toHaveLength(2) @@ -374,7 +384,7 @@ describe('switchToChat (unified, D.4)', () => { 'agent-x', 'delta-pages', undefined, - { limit: 40, afterTurn: 1 } + { limit: 40, afterTurn: 0 } ) expect(clerum.rpc.loadSessionMessages).toHaveBeenNthCalledWith( 2, @@ -457,6 +467,46 @@ describe('switchToChat (unified, D.4)', () => { ]) }) + it('preserves turnless legacy history outside a partial authoritative window', async () => { + const legacyMessages = Array.from({ length: 100 }, (_, index) => ({ + id: `legacy-${index + 1}`, + role: index % 2 === 0 ? ('user' as const) : ('assistant' as const), + content: `legacy-${index + 1}`, + timestamp: index + 1, + })) + clerum.chat.loadMessages.mockResolvedValue(legacyMessages) + clerum.rpc.loadSessionMessages.mockResolvedValue({ + agent: 'agent-x', + chatId: 'legacy-partial-window', + state: 'idle', + totalTurns: 50, + oldestTurnNumber: 11, + latestTurnNumber: 50, + hasMoreBefore: true, + hasMoreAfter: false, + turns: Array.from({ length: 40 }, (_, index) => + turn(index + 11, `server-q${index + 11}`, `server-a${index + 11}`) + ), + }) + const { result } = renderController() + await settleMount() + + await act(async () => { + await result.current.switchToChat('agent-x', 'legacy-partial-window') + }) + + expect(result.current.chatMessages.map(message => message.id)).toContain('legacy-1') + expect(result.current.chatMessages.map(message => message.id)).toContain('turn-50-assistant') + expect(clerum.chat.replaceMessages).toHaveBeenCalledWith( + 'agent-x', + 'legacy-partial-window', + expect.arrayContaining([ + expect.objectContaining({ id: 'legacy-1' }), + expect.objectContaining({ id: 'turn-11-user' }), + ]) + ) + }) + it('caps server delta pagination so reconcile cannot walk an unbounded cursor chain', async () => { clerum.chat.loadMessages.mockResolvedValue([ { id: 'turn-1-user', role: 'user' as const, content: 'q1', timestamp: 1 }, @@ -512,8 +562,6 @@ describe('switchToChat (unified, D.4)', () => { 'turn-4-assistant', 'turn-5-user', 'turn-5-assistant', - 'turn-6-user', - 'turn-6-assistant', 'turn-999-user', 'turn-999-assistant', ]) @@ -883,7 +931,7 @@ describe('switchToChat (unified, D.4)', () => { 'agent-x', 'synced-chat', undefined, - { limit: 40, afterTurn: 1 } + { limit: 40, afterTurn: 0 } ) expect(result.current.chatMessages.map(message => message.id)).toEqual([ 'turn-1-user', @@ -989,6 +1037,439 @@ describe('switchToChat (unified, D.4)', () => { expect(result.current.olderMessagesLoading).toBe(false) }) + it.each([401, 403, 404])( + 'older-message HTTP %i applies the correct Host authority decision', + async status => { + const revoked = new Set() + const uncertain = new Set() + clerum.chat.loadMessages.mockImplementation(async (_agentRef, _chatId, _limit, offset) => + offset === undefined + ? [ + { + id: 'turn-10-user', + role: 'user', + content: 'cached', + timestamp: 10, + serverTurnNumber: 10, + }, + ] + : [] + ) + clerum.rpc.loadSessionMessages.mockImplementation( + async (_hostRef, _agent, chatId, _teamId, query?: { beforeTurn?: number }) => { + if (query?.beforeTurn !== undefined) throw new Error(`${status} response`) + return { + agent: 'agent-x', + chatId, + state: 'idle', + turns: [], + hasMoreBefore: true, + hasMoreAfter: false, + } + } + ) + const { result } = renderController({ + onHostAccessRevoked: agentRef => revoked.add(agentRef), + onHostAuthorityUncertain: agentRef => uncertain.add(agentRef), + isHostAccessBlocked: agentRef => revoked.has(agentRef) || uncertain.has(agentRef), + }) + await settleMount() + await act(async () => { + await result.current.switchToChat('agent-x', 'older-authority') + }) + expect(result.current.hasOlderMessages).toBe(true) + + await act(async () => { + await result.current.handleLoadOlderMessages() + }) + if (status === 404) { + expect(revoked.has('agent-x')).toBe(false) + expect(uncertain.has('agent-x')).toBe(false) + expect(result.current.activeChatId).toBe('older-authority') + expect(result.current.chatMessages.map(message => message.content)).toEqual(['cached']) + } else { + expect(revoked.has('agent-x')).toBe(false) + expect(uncertain.has('agent-x')).toBe(true) + expect(result.current.activeChatId).toBe('older-authority') + expect(result.current.chatMessages.map(message => message.content)).toEqual(['cached']) + } + } + ) + + // R1-M8: the two hold kinds share one tear-down of the Host's live work and + // differ only in what they hide. An uncertain hold (401 after refresh, or a 403 + // without a revocation code) must not destroy state the user may still own + // once the hold is released: the chat and its list stay. The deselection + // itself is the `onHostAuthorityUncertain` callback (`useAppController`). + it('uncertain host-access hold releases the tracker, resets the chat state and deselects the agent, but does not hide the agent or clear the list; a confirmed revocation does', async () => { + const chatKey = makeTaskKey('agent-x', 'older-hold') + const runScenario = async (readError: () => Promise) => { + const revoked = new Set() + const uncertain = new Set() + clerum.rpc.loadSessionMessages.mockReset() + clerum.chat.getIndex.mockResolvedValue(localIndex([{ id: 'older-hold', title: 'Held chat' }])) + clerum.chat.loadMessages.mockResolvedValue([]) + clerum.rpc.listSessions.mockResolvedValue( + serverSessions([{ agent: 'agent-x', chatId: 'older-hold' }]) + ) + clerum.rpc.loadSessionMessages.mockImplementation( + async (_hostRef, _agent, chatId, _teamId, query?: { beforeTurn?: number }) => { + if (query?.beforeTurn !== undefined) throw await readError() + return { + agent: 'agent-x', + chatId, + state: 'processing', + activeTaskId: 'task-hold', + turns: [turn(5, 'question')], + hasMoreBefore: true, + hasMoreAfter: false, + } + } + ) + // Stable callbacks, like the production `useCallback`s: a new + // `isHostAccessBlocked` identity per render would re-run the + // agent-selection effect, which clears the list of a blocked Host by itself. + const isHostAccessBlocked = (agentRef: string) => + revoked.has(agentRef) || uncertain.has(agentRef) + const controller = renderController({ + onHostAccessRevoked: agentRef => revoked.add(agentRef), + onHostAuthorityUncertain: agentRef => uncertain.add(agentRef), + isHostAccessBlocked, + }) + await settleMount() + await waitFor(() => + expect(controller.result.current.chatList.map(chat => chat.id)).toContain('older-hold') + ) + await waitFor(() => + expect(controller.result.current.latestChatSessions.map(chat => chat.id)).toContain( + 'older-hold' + ) + ) + await act(async () => { + await controller.result.current.switchToChat('agent-x', 'older-hold') + }) + // Liveness: the tracker is following the task, the FSM tracks the chat and + // an older page exists, so the tear-down below has something to undo. + await waitFor(() => expect(clerum.hasProgressHandler('task-hold')).toBe(true)) + const phaseBefore = controller.result.current.sessionFsmStore.getState(chatKey)?.phase + expect(phaseBefore).toBeDefined() + expect(phaseBefore).not.toBe('idle') + expect(controller.result.current.hasOlderMessages).toBe(true) + + await act(async () => { + await controller.result.current.handleLoadOlderMessages() + }) + const observed = { + revoked: revoked.has('agent-x'), + uncertain: uncertain.has('agent-x'), + trackerFollowsTask: clerum.hasProgressHandler('task-hold'), + phaseAfter: controller.result.current.sessionFsmStore.getState(chatKey)?.phase, + activeChatId: controller.result.current.activeChatId, + } + // The published lists mask a held Host on their own. Lift the hold to read + // the state underneath: what the hold tore down is what is gone. + revoked.clear() + uncertain.clear() + act(() => controller.rerender()) + const unmasked = { + chatListIds: controller.result.current.chatList.map(chat => chat.id), + latestIds: controller.result.current.latestChatSessions.map(chat => chat.id), + } + controller.unmount() + return { phaseBefore, ...observed, ...unmasked } + } + + const held = await runScenario(() => ipcGenericForbidden('rpc:loadSessionMessages')) + expect(held.uncertain).toBe(true) + expect(held.revoked).toBe(false) + expect(held.trackerFollowsTask).toBe(false) + expect(held.phaseAfter).not.toBe(held.phaseBefore) + expect(held.chatListIds).toEqual(['older-hold']) + expect(held.latestIds).toEqual(['older-hold']) + expect(held.activeChatId).toBe('older-hold') + + const revokedRun = await runScenario(() => ipcHostAccessRevoked('rpc:loadSessionMessages')) + expect(revokedRun.revoked).toBe(true) + expect(revokedRun.uncertain).toBe(false) + // Same tear-down of the Host's live work... + expect(revokedRun.trackerFollowsTask).toBe(false) + expect(revokedRun.phaseAfter).not.toBe(revokedRun.phaseBefore) + // ...plus the hide: the list is cleared and the chat is closed. + expect(revokedRun.chatListIds).toEqual([]) + expect(revokedRun.latestIds).toEqual([]) + expect(revokedRun.activeChatId).toBeNull() + }) + + it('re-fetches the newest cached turn so a response completed after caching is recovered', async () => { + clerum.chat.loadMessages.mockResolvedValue([ + { id: 'turn-1-user', role: 'user' as const, content: 'q', timestamp: 1, serverTurnNumber: 1 }, + ]) + clerum.rpc.loadSessionMessages.mockResolvedValue({ + agent: 'agent-x', + chatId: 'partial-turn', + state: 'idle', + totalTurns: 1, + turns: [turn(1, 'q', 'completed answer')], + }) + const { result } = renderController() + await settleMount() + + await act(async () => { + await result.current.switchToChat('agent-x', 'partial-turn') + }) + + expect(clerum.rpc.loadSessionMessages).toHaveBeenCalledWith( + 'agent-x', + 'agent-x', + 'partial-turn', + undefined, + { limit: 40, afterTurn: 0 } + ) + expect(result.current.chatMessages.map(message => message.content)).toEqual([ + 'q', + 'completed answer', + ]) + }) + + it('reuses the in-flight selection when the same chat is reopened', async () => { + const response = deferred() + clerum.chat.loadMessages.mockResolvedValue([]) + clerum.rpc.loadSessionMessages.mockImplementationOnce(() => response.promise) + const { result } = renderController() + await settleMount() + + let firstSwitch!: Promise + await act(async () => { + firstSwitch = result.current.switchToChat('agent-x', 'same-chat') + await waitFor(() => expect(clerum.rpc.loadSessionMessages).toHaveBeenCalledTimes(1)) + }) + let secondSwitch!: Promise + await act(async () => { + secondSwitch = result.current.switchToChat('agent-x', 'same-chat') + await Promise.resolve() + }) + + response.resolve({ + agent: 'agent-x', + chatId: 'same-chat', + state: 'idle', + totalTurns: 1, + turns: [turn(1, 'question', 'fresh answer')], + }) + await act(async () => { + await Promise.all([firstSwitch, secondSwitch]) + }) + + expect(clerum.rpc.loadSessionMessages).toHaveBeenCalledTimes(1) + expect(result.current.chatMessages.map(message => message.content)).toEqual([ + 'question', + 'fresh answer', + ]) + }) + + // R3-M3: the scope-change negative and its same-scope control share one + // IPC-shaped revocation, so the control proves the classifier fires when the + // team does not change and the negative cannot pass by never classifying. + it.each([ + ['ignores a send 403 from the prior team scope', 'team-after', false], + ['holds the Host on a send 403 within the same team scope', 'team-before', true], + ] as const)('%s', async (_label, teamAtRejection, expectedRevoked) => { + const revoked = new Set() + let rejectSend!: (reason: unknown) => void + clerum.rpc.invokeHostMessage.mockImplementation( + () => + new Promise((_resolve, reject) => { + rejectSend = reject + }) + ) + const controller = renderController({ + currentTeamId: 'team-before', + onHostAccessRevoked: agentRef => revoked.add(agentRef), + onHostAuthorityUncertain: agentRef => revoked.add(agentRef), + isHostAccessBlocked: agentRef => revoked.has(agentRef), + }) + await settleMount() + await act(async () => { + await controller.result.current.switchToChat('agent-x', 'old-team-chat') + }) + + const denial = await ipcHostAccessRevoked('rpc:invokeHostMessage') + const pendingSend = controller.result.current.handleSendAgentMessage('hello') + await waitFor(() => expect(clerum.rpc.invokeHostMessage).toHaveBeenCalledTimes(1)) + controller.rerender({ currentTeamId: teamAtRejection }) + await act(async () => { + rejectSend(denial) + await pendingSend + }) + expect(revoked.has('agent-x')).toBe(expectedRevoked) + }) + + it('ignores an older-message 403 from the prior team scope', async () => { + const revoked = new Set() + let rejectOlder!: (reason: unknown) => void + clerum.chat.loadMessages.mockImplementation(async (_agentRef, _chatId, _limit, offset) => + offset === undefined + ? [ + { + id: 'turn-10-user', + role: 'user', + content: 'cached', + timestamp: 10, + serverTurnNumber: 10, + }, + ] + : [] + ) + clerum.rpc.loadSessionMessages.mockImplementation( + async (_hostRef, _agent, chatId, _teamId, query?: { beforeTurn?: number }) => { + if (query?.beforeTurn !== undefined) { + return new Promise((_resolve, reject) => { + rejectOlder = reject + }) + } + return { agent: 'agent-x', chatId, state: 'idle', turns: [], hasMoreBefore: true } + } + ) + const controller = renderController({ + currentTeamId: 'team-before', + onHostAccessRevoked: agentRef => revoked.add(agentRef), + onHostAuthorityUncertain: agentRef => revoked.add(agentRef), + isHostAccessBlocked: agentRef => revoked.has(agentRef), + }) + await settleMount() + await act(async () => { + await controller.result.current.switchToChat('agent-x', 'old-team-chat') + }) + const denial = await ipcGenericForbidden('rpc:loadSessionMessages') + const pendingOlder = controller.result.current.handleLoadOlderMessages() + await waitFor(() => expect(rejectOlder).toEqual(expect.any(Function))) + controller.rerender({ currentTeamId: 'team-after' }) + await act(async () => { + rejectOlder(denial) + await pendingOlder + }) + expect(revoked.has('agent-x')).toBe(false) + }) + + it.each([401, 403, 404])( + 'durable task-result HTTP %i applies the correct Host authority decision', + async status => { + const revoked = new Set() + const uncertain = new Set() + clerum.chat.loadMessages.mockResolvedValue([]) + clerum.rpc.loadSessionMessages.mockResolvedValue({ + agent: 'agent-x', + chatId: 'result-authority', + state: 'idle', + turns: [], + }) + clerum.rpc.getTaskResult.mockRejectedValue(new Error(`${status} response`)) + const { result } = renderController({ + onHostAccessRevoked: agentRef => revoked.add(agentRef), + onHostAuthorityUncertain: agentRef => uncertain.add(agentRef), + isHostAccessBlocked: agentRef => revoked.has(agentRef) || uncertain.has(agentRef), + }) + await settleMount() + await act(async () => { + await result.current.switchToChat('agent-x', 'result-authority') + }) + + await act(async () => { + await result.current.reconcileChat(makeTaskKey('agent-x', 'result-authority'), { + reason: 'task_result_recovery', + taskIdHint: 'task-missing-turn', + }) + }) + expect(clerum.rpc.getTaskResult).toHaveBeenCalled() + if (status === 404) { + expect(revoked.has('agent-x')).toBe(false) + expect(uncertain.has('agent-x')).toBe(false) + expect(result.current.activeChatId).toBe('result-authority') + } else { + expect(revoked.has('agent-x')).toBe(false) + expect(uncertain.has('agent-x')).toBe(true) + expect(result.current.activeChatId).toBe('result-authority') + } + } + ) + + it.each(['revocation', 'confirmed deletion', 'team switch'])( + 'does not publish a late durable task reply after %s', + async action => { + const revoked = new Set() + const lateResult = deferred<{ response: string }>() + clerum.chat.loadMessages.mockResolvedValue([]) + clerum.rpc.loadSessionMessages.mockResolvedValueOnce({ + agent: 'agent-x', + chatId: 'late-result', + state: 'processing', + activeTaskId: 'task-late-result', + turns: [turn(1, 'question')], + }) + const controller = renderController({ + currentTeamId: 'team-before', + onHostAccessRevoked: agentRef => revoked.add(agentRef), + onHostAuthorityUncertain: agentRef => revoked.add(agentRef), + isHostAccessBlocked: agentRef => revoked.has(agentRef), + }) + await settleMount() + await act(async () => { + await controller.result.current.switchToChat('agent-x', 'late-result') + }) + await waitFor(() => expect(clerum.hasProgressHandler('task-late-result')).toBe(true)) + + // The initial processing snapshot may fetch attachments. Hold only the + // terminal fallback read whose late completion is under test. + clerum.rpc.getTaskResult.mockClear() + clerum.rpc.getTaskResult.mockImplementation(() => lateResult.promise) + clerum.rpc.loadSessionMessages.mockRejectedValue(new Error('unexpected transcript failure')) + clerum.emitTaskProgress('task-late-result', { type: 'error', message: 'stream dropped' }) + await waitFor(() => expect(clerum.rpc.getTaskResult).toHaveBeenCalledTimes(1)) + + if (action === 'revocation') { + clerum.rpc.loadSessionMessages.mockRejectedValue( + await ipcHostAccessRevoked('rpc:loadSessionMessages') + ) + await act(async () => { + await controller.result.current.reconcileChat(makeTaskKey('agent-x', 'late-result'), { + reason: 'authority_refresh', + }) + }) + expect(revoked.has('agent-x')).toBe(true) + } else if (action === 'confirmed deletion') { + await act(async () => { + const deletion = await controller.result.current.captureChatDeleteFence('agent-x') + await controller.result.current.handleDeleteChatForAgent( + 'agent-x', + 'late-result', + deletion + ) + }) + expect(clerum.chat.delete).toHaveBeenCalledWith( + 'agent-x', + 'late-result', + expect.objectContaining({ version: 1, bindingGeneration: 1 }) + ) + } else { + controller.rerender({ currentTeamId: 'team-after' }) + } + + const writesBefore = clerum.chat.appendMessages.mock.calls.length + const notificationsBefore = controller.spies.pushNotification.mock.calls.length + const toastsBefore = controller.spies.pushToast.mock.calls.length + await act(async () => { + lateResult.resolve({ response: 'late protected reply' }) + await lateResult.promise + }) + expect(clerum.chat.appendMessages.mock.calls.length).toBe(writesBefore) + expect(controller.spies.pushNotification.mock.calls.length).toBe(notificationsBefore) + expect(controller.spies.pushToast.mock.calls.length).toBe(toastsBefore) + expect(controller.result.current.chatMessages.map(message => message.content)).not.toContain( + 'late protected reply' + ) + } + ) + it('ignores an older-page response when switching away and back before it resolves', async () => { let resolveOlder!: (messages: unknown[]) => void const olderPage = new Promise(resolve => { @@ -1665,10 +2146,12 @@ describe('switchToChat (unified, D.4)', () => { expect(clerum.hasProgressHandler('task-NEW')).toBe(true) }) - it('404 evicts the stale local chat and resets the active selection', async () => { - const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) - clerum.chat.loadMessages.mockResolvedValue([]) + it('ambiguous 404 preserves the selected chat, cached messages and unsent draft', async () => { + clerum.chat.loadMessages.mockResolvedValue([ + { id: 'cached-turn', role: 'user' as const, content: 'known conversation', timestamp: 1 }, + ]) clerum.rpc.loadSessionMessages.mockRejectedValue(new Error('404 Not Found')) + setComposerDraft('gone', 'unsent message') const { result } = renderController() await settleMount() @@ -1676,10 +2159,16 @@ describe('switchToChat (unified, D.4)', () => { await result.current.switchToChat('agent-x', 'gone') }) - expect(clerum.chat.delete).toHaveBeenCalledWith('agent-x', 'gone') - expect(result.current.activeChatId).toBeNull() - expect(result.current.sessionStateByChatKey['agent-x::gone']).toBeUndefined() - expect(warn).toHaveBeenCalled() + expect(clerum.chat.delete).not.toHaveBeenCalled() + expect(result.current.activeChatId).toBe('gone') + expect(result.current.chatMessages.map(message => message.content)).toEqual([ + 'known conversation', + ]) + expect(getComposerDraft('gone')).toBe('unsent message') + expect(result.current.sessionStateByChatKey['agent-x::gone']).toMatchObject({ + syncing: false, + offlineMode: true, + }) }) it('network error keeps the Phase-1 cache and flags offline mode', async () => { diff --git a/desktop-app/ui/src/hooks/domain/__tests__/useAgentChatController.crossChat.test.tsx b/desktop-app/ui/src/hooks/domain/__tests__/useAgentChatController.crossChat.test.tsx index 563745c9a1..36aad7b892 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/useAgentChatController.crossChat.test.tsx +++ b/desktop-app/ui/src/hooks/domain/__tests__/useAgentChatController.crossChat.test.tsx @@ -12,10 +12,26 @@ import { AgentTaskTrackerProvider } from '@contexts/AgentTaskTrackerContext' import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react' import { useAgentChatController } from '@hooks/domain/useAgentChatController' import { useComposerDraft } from '@hooks/useComposerDraft' -import { resetComposerDraftStore } from '@lib/composerDraftStore' +import { getComposerDraft, resetComposerDraftStore } from '@lib/composerDraftStore' import type { TaskProgressStreamEvent } from '../../../../../src/types' +import { useHarnessHostAuthority } from './__fixtures__/hostAuthorityHarness' +import { + ipcGenericForbidden, + ipcHostAccessDenied, + ipcHostAccessRevoked, + ipcHostWaking, + ipcHttpError, + ipcServerErrorMentioning403, +} from './__fixtures__/ipcErrors' type ProgressHandler = (event: TaskProgressStreamEvent) => void | Promise +// Record every hold the controller requests; the hold itself goes through the +// production store (useHarnessHostAuthority), so epochs are real. +const revokedAgents = new Set() +const uncertainAgents = new Set() +const recordHold = (agentRef: string, kind: 'revoked' | 'uncertain') => { + ;(kind === 'revoked' ? revokedAgents : uncertainAgents).add(agentRef) +} function getDraftInputValue(): string { return (screen.getByTestId('draft-input') as HTMLInputElement).value @@ -34,9 +50,27 @@ function createChatMeta(chatId: string) { function installClerumHarness() { let taskIndex = 0 - const chats: Array> = [] - const messagesByChat = new Map() + const chatsByAgent = new Map>>() + const deletedChatIds = new Set() + const messagesByAgentChat = new Map() + const messageKey = (agentRef: string, chatId: string) => `${agentRef}\u0000${chatId}` const progressHandlers = new Map() + const loadSessionMessages = vi.fn(async () => ({ agent: 'trader', chatId: '', turns: [] })) + const listSessions = vi.fn(async () => ({ items: [] })) + const appendMessages = vi.fn(async (agentRef: string, chatId: string, messages: unknown[]) => { + const key = messageKey(agentRef, chatId) + messagesByAgentChat.set(key, [...(messagesByAgentChat.get(key) || []), ...messages]) + }) + const upsertMessagesIntoMap = (agentRef: string, chatId: string, messages: unknown[]) => { + const key = messageKey(agentRef, chatId) + const existing = messagesByAgentChat.get(key) || [] + const byId = new Map(existing.map(message => [(message as { id: string }).id, message])) + for (const message of messages) byId.set((message as { id: string }).id, message) + messagesByAgentChat.set(key, [...byId.values()]) + } + const upsertMessages = vi.fn(async (agentRef: string, chatId: string, messages: unknown[]) => { + upsertMessagesIntoMap(agentRef, chatId, messages) + }) const invokeHostMessage = vi.fn(async (_agentRef: string, _payload: { threadId?: string }) => { taskIndex += 1 @@ -48,34 +82,53 @@ function installClerumHarness() { writable: true, value: { chat: { - list: vi.fn(async () => chats), + list: vi.fn(async (agentRef: string) => chatsByAgent.get(agentRef) || []), create: vi.fn(async (_agentRef: string, chatId: string) => { const meta = createChatMeta(chatId) - chats.push(meta) - messagesByChat.set(chatId, []) + const agentChats = chatsByAgent.get(_agentRef) || [] + agentChats.push(meta) + chatsByAgent.set(_agentRef, agentChats) + messagesByAgentChat.set(messageKey(_agentRef, chatId), []) return meta }), rename: vi.fn(async () => undefined), - delete: vi.fn(async () => undefined), + getBindingGeneration: vi.fn(async () => 1), + captureDeleteFence: vi.fn(async (authorityScope: unknown) => ({ + version: 1, + authorityScope, + bindingGeneration: 1, + sessionGeneration: 1, + })), + delete: vi.fn(async (_agentRef: string, chatId: string) => { + deletedChatIds.add(chatId) + const chats = chatsByAgent.get(_agentRef) || [] + const index = chats.findIndex(chat => chat.id === chatId) + if (index >= 0) chats.splice(index, 1) + messagesByAgentChat.delete(messageKey(_agentRef, chatId)) + return { cleanupPending: false } + }), loadMessages: vi.fn( - async (_agentRef: string, chatId: string) => messagesByChat.get(chatId) || [] + async (agentRef: string, chatId: string) => + messagesByAgentChat.get(messageKey(agentRef, chatId)) || [] ), - appendMessages: vi.fn(async (_agentRef: string, chatId: string, messages: unknown[]) => { - messagesByChat.set(chatId, [...(messagesByChat.get(chatId) || []), ...messages]) - }), - replaceMessages: vi.fn(async (_agentRef: string, chatId: string, messages: unknown[]) => { - messagesByChat.set(chatId, [...messages]) + appendMessages, + upsertMessages, + replaceMessages: vi.fn(async (agentRef: string, chatId: string, messages: unknown[]) => { + messagesByAgentChat.set(messageKey(agentRef, chatId), [...messages]) }), markUnreadTerminal: vi.fn(async () => undefined), clearUnreadTerminal: vi.fn(async () => undefined), getLastActive: vi.fn(async () => null), setLastActive: vi.fn(async () => undefined), - getIndex: vi.fn(async () => ({ chats })), + getIndex: vi.fn(async (agentRef: string) => ({ + chats: chatsByAgent.get(agentRef) || [], + deletedChatIds: [...deletedChatIds], + })), dismissOnboarding: vi.fn(async () => undefined), }, rpc: { - listSessions: vi.fn(async () => ({ items: [] })), - loadSessionMessages: vi.fn(async () => ({ agent: 'trader', chatId: '', turns: [] })), + listSessions, + loadSessionMessages, subscribeHostActivity: vi.fn(async () => async () => undefined), invokeHostMessage, subscribeTaskProgress: vi.fn( @@ -94,18 +147,40 @@ function installClerumHarness() { }, }) - return { invokeHostMessage, progressHandlers } + return { + invokeHostMessage, + appendMessages, + upsertMessages, + progressHandlers, + loadSessionMessages, + listSessions, + seedMessage: (chatId: string, content: string) => + messagesByAgentChat.set(messageKey('trader', chatId), [ + { id: `message-${chatId}`, role: 'user', content, createdAt: new Date().toISOString() }, + ]), + } } function AgentChatHarness() { + const [selectionState, setSelectionState] = React.useState('idle') + const [sendState, setSendState] = React.useState('idle') + const [selectedAgent, setSelectedAgent] = React.useState('trader') + const hostAuthority = useHarnessHostAuthority(recordHold) const vm = useAgentChatController({ - selectedAgent: 'trader', - agentNames: ['trader'], + selectedAgent, + agentNames: ['trader', 'chatllm-stateless'], currentTeamId: 'team-1', + chatAuthorityTeamId: 'team-1', + currentEnvironmentKey: 'env-test', currentTeamName: 'Team One', isAuthenticated: true, loadMenuData: true, navItem: 'chat', + onHostAccessRevoked: hostAuthority.onHostAccessRevoked, + onHostAuthorityUncertain: hostAuthority.onHostAuthorityUncertain, + isHostAccessBlocked: hostAuthority.isHostAccessBlocked, + getHostAuthorityEpoch: hostAuthority.getHostAuthorityEpoch, + hostAuthorityRevision: hostAuthority.revision, pushToast: vi.fn(), pushNotification: vi.fn(), agentDisplayName: (agentName: string) => agentName, @@ -123,6 +198,18 @@ function AgentChatHarness() {
{vm.activeChatId || ''}
{String(vm.agentSending)}
+
{vm.chatMessages.length}
+ {vm.chatMessages.map(message => ( +
+ {message.content} +
+ ))} +
{vm.chatList.map(chat => chat.id).join(',')}
+
+ {vm.latestChatSessions.map(chat => chat.id).join(',')} +
+
{selectionState}
+
{sendState}
void vm.handleCreateChat()}> Create chat - + ))} + {vm.chatList.map(chat => ( - + + + + ))}
) @@ -147,7 +269,9 @@ describe('useAgentChatController (cross-chat, migrated)', () => { afterEach(() => { cleanup() vi.restoreAllMocks() + uncertainAgents.clear() resetComposerDraftStore() + revokedAgents.clear() delete (window as { clerum?: unknown }).clerum }) @@ -204,6 +328,108 @@ describe('useAgentChatController (cross-chat, migrated)', () => { await waitFor(() => expect(progressHandlers.size).toBe(0)) }) + it('persists the outgoing row before invoking the host and leaves it visible while the acknowledgement is pending', async () => { + const { invokeHostMessage, upsertMessages, progressHandlers } = installClerumHarness() + let acknowledge!: (result: { taskId: string; status: string }) => void + invokeHostMessage.mockImplementationOnce( + () => + new Promise(resolve => { + acknowledge = resolve + }) + ) + + render( + + + + ) + fireEvent.click(screen.getByRole('button', { name: 'Create chat' })) + await waitFor(() => expect(screen.getByTestId('active-chat-id').textContent).not.toBe('')) + + fireEvent.click(screen.getByRole('button', { name: 'Send message' })) + await waitFor(() => expect(invokeHostMessage).toHaveBeenCalledTimes(1)) + + expect(upsertMessages.mock.invocationCallOrder[0]).toBeLessThan( + invokeHostMessage.mock.invocationCallOrder[0]! + ) + expect(upsertMessages.mock.calls[0]?.[2]?.[0]).toMatchObject({ + role: 'user', + content: 'hello', + }) + expect(upsertMessages.mock.calls[0]?.[2]?.[0]).not.toHaveProperty('task_id') + expect(screen.getByTestId('chat-message-row').textContent).toBe('hello') + expect(screen.getByTestId('send-state').textContent).toBe('pending') + expect(progressHandlers.size).toBe(0) + + acknowledge({ taskId: 'acknowledged-task', status: 'pending' }) + await waitFor(() => expect(screen.getByTestId('send-state').textContent).toBe('settled')) + await waitFor(() => expect(progressHandlers.has('acknowledged-task')).toBe(true)) + expect(upsertMessages.mock.calls[1]?.[2]?.[0]).toMatchObject({ + id: (upsertMessages.mock.calls[0]?.[2]?.[0] as { id: string }).id, + task_id: 'acknowledged-task', + }) + expect(screen.getByTestId('chat-message-row').textContent).toBe('hello') + + await progressHandlers.get('acknowledged-task')?.({ + type: 'terminal', + data: { status: 'completed' }, + } as TaskProgressStreamEvent) + await waitFor(() => expect(progressHandlers.size).toBe(0)) + }) + + it('keeps an outgoing message visible across a Host switch while its POST is pending', async () => { + const { invokeHostMessage, upsertMessages } = installClerumHarness() + const persistOutgoingToMap = upsertMessages.getMockImplementation() + let persistOutgoing!: () => void + upsertMessages.mockImplementationOnce( + (agentRef: string, chatId: string, messages: unknown[]) => + new Promise(resolve => { + persistOutgoing = () => { + persistOutgoingToMap?.(agentRef, chatId, messages) + resolve(undefined) + } + }) + ) + let finishInvoke!: (result: { taskId: string; status: string }) => void + invokeHostMessage.mockImplementationOnce( + () => + new Promise(resolve => { + finishInvoke = resolve + }) + ) + render( + + + + ) + + fireEvent.click(screen.getByRole('button', { name: 'Create chat' })) + await waitFor(() => expect(screen.getByTestId('active-chat-id').textContent).not.toBe('')) + const chatId = screen.getByTestId('active-chat-id').textContent || '' + fireEvent.click(screen.getByRole('button', { name: 'Send message' })) + await waitFor(() => expect(upsertMessages).toHaveBeenCalledTimes(1)) + + fireEvent.click(screen.getByRole('button', { name: 'Select agent chatllm-stateless' })) + await waitFor(() => expect(screen.getByTestId('active-chat-id').textContent).not.toBe(chatId)) + persistOutgoing() + await waitFor(() => expect(invokeHostMessage).toHaveBeenCalledTimes(1)) + expect(screen.getByTestId('chat-message-count').textContent).toBe('0') + fireEvent.click(screen.getByRole('button', { name: 'Select agent trader' })) + await waitFor(() => expect(screen.getByTestId('active-chat-id').textContent).toBe(chatId)) + await waitFor(() => expect(screen.getByTestId('chat-message-count').textContent).toBe('1')) + + finishInvoke({ taskId: 'late-task', status: 'pending' }) + await waitFor(() => expect(screen.getByTestId('send-state').textContent).toBe('settled')) + expect(screen.getByTestId('chat-message-count').textContent).toBe('1') + expect(upsertMessages).toHaveBeenCalledTimes(2) + expect(upsertMessages.mock.calls[0]?.[2]?.[0]).toMatchObject({ role: 'user' }) + expect(upsertMessages.mock.calls[0]?.[2]?.[0]).not.toHaveProperty('task_id') + expect(upsertMessages.mock.calls[1]?.[2]?.[0]).toMatchObject({ + role: 'user', + task_id: 'late-task', + }) + }) + it('keeps composer drafts scoped to their chat session', async () => { installClerumHarness() @@ -243,4 +469,227 @@ describe('useAgentChatController (cross-chat, migrated)', () => { await waitFor(() => expect(screen.getByTestId('active-chat-id').textContent).toBe(secondChatId)) expect(getDraftInputValue()).toBe('draft for session B') }) + + it.each(['404', '503', 'timeout'])( + 'keeps chat, selection and draft after a transient transcript %s', + async failure => { + const { loadSessionMessages, seedMessage } = installClerumHarness() + render( + + + + ) + + fireEvent.click(screen.getByRole('button', { name: 'Create chat' })) + await waitFor(() => expect(screen.getByTestId('active-chat-id').textContent).not.toBe('')) + const firstChatId = screen.getByTestId('active-chat-id').textContent || '' + seedMessage(firstChatId, 'known conversation turn') + fireEvent.click(screen.getByRole('button', { name: 'Create chat' })) + await waitFor(() => + expect(screen.getByTestId('active-chat-id').textContent).not.toBe(firstChatId) + ) + fireEvent.click(screen.getByRole('button', { name: `Select chat ${firstChatId}` })) + await waitFor(() => expect(screen.getByTestId('chat-message-count').textContent).toBe('1')) + fireEvent.change(screen.getByTestId('draft-input'), { + target: { value: 'unsent draft' }, + }) + + const readsBeforeFault = loadSessionMessages.mock.calls.length + loadSessionMessages.mockRejectedValue(new Error(failure)) + fireEvent.click(screen.getByRole('button', { name: `Select chat ${firstChatId}` })) + await waitFor(() => + expect(loadSessionMessages.mock.calls.length).toBeGreaterThan(readsBeforeFault) + ) + await waitFor(() => expect(screen.getByTestId('selection-state').textContent).toBe('settled')) + await waitFor(() => expect(screen.getByTestId('chat-message-count').textContent).toBe('1')) + expect(screen.getByTestId('active-chat-id').textContent).toBe(firstChatId) + expect(screen.getByTestId('chat-message-count').textContent).toBe('1') + expect(getDraftInputValue()).toBe('unsent draft') + expect(screen.getByRole('button', { name: `Select chat ${firstChatId}` })).toBeTruthy() + } + ) + + it('does not restore protected chat or messages when a pending send resolves after revocation', async () => { + const { invokeHostMessage, loadSessionMessages, progressHandlers } = installClerumHarness() + let finishInvoke!: (result: { taskId: string; status: string }) => void + invokeHostMessage.mockImplementationOnce( + () => + new Promise(resolve => { + finishInvoke = resolve + }) + ) + render( + + + + ) + + fireEvent.click(screen.getByRole('button', { name: 'Create chat' })) + await waitFor(() => expect(screen.getByTestId('active-chat-id').textContent).not.toBe('')) + const chatId = screen.getByTestId('active-chat-id').textContent || '' + fireEvent.click(screen.getByRole('button', { name: 'Send message' })) + await waitFor(() => expect(invokeHostMessage).toHaveBeenCalledTimes(1)) + + loadSessionMessages.mockRejectedValue(await ipcHostAccessRevoked('rpc:loadSessionMessages')) + fireEvent.click(screen.getByRole('button', { name: `Select chat ${chatId}` })) + await waitFor(() => expect(revokedAgents.has('trader')).toBe(true)) + await waitFor(() => expect(screen.getByTestId('active-chat-id').textContent).toBe('')) + expect(screen.getByTestId('chat-message-count').textContent).toBe('0') + + finishInvoke({ taskId: 'late-task', status: 'pending' }) + await waitFor(() => expect(screen.getByTestId('send-state').textContent).toBe('settled')) + await waitFor(() => expect(screen.getByTestId('agent-sending').textContent).toBe('false')) + expect(screen.getByTestId('active-chat-id').textContent).toBe('') + expect(screen.getByTestId('chat-message-count').textContent).toBe('0') + expect(screen.queryByRole('button', { name: `Select chat ${chatId}` })).toBeNull() + expect(progressHandlers.has('late-task')).toBe(false) + }) + + // R2-B1: every failure is the real RpcProxyClient rejection wrapped the way + // Electron IPC wraps it, never a hand-written bare message. + it.each([ + { + label: '401', + sendError: () => + ipcHttpError('rpc:invokeHostMessage', 401, 'Unauthorized', { error: 'expired' }), + readError: null, + revoked: false, + uncertain: true, + }, + { + label: 'exact Host-wide 403', + sendError: () => ipcHostAccessRevoked('rpc:invokeHostMessage'), + readError: () => + ipcHttpError('rpc:listSessions', 503, 'Service Unavailable', { error: 'unavailable' }), + revoked: true, + uncertain: false, + }, + { + label: 'host_access_denied 403 with readable catalog', + sendError: () => ipcHostAccessDenied('rpc:invokeHostMessage'), + readError: null, + revoked: false, + uncertain: false, + }, + { + label: 'generic 403 with readable catalog', + sendError: () => ipcGenericForbidden('rpc:invokeHostMessage'), + readError: null, + revoked: false, + uncertain: false, + }, + { + label: 'generic 403 with denied catalog', + sendError: () => ipcGenericForbidden('rpc:invokeHostMessage'), + readError: () => ipcGenericForbidden('rpc:listSessions'), + revoked: false, + uncertain: true, + }, + { + label: 'generic 403 with revoked catalog', + sendError: () => ipcGenericForbidden('rpc:invokeHostMessage'), + readError: () => ipcHostAccessRevoked('rpc:listSessions'), + revoked: true, + uncertain: false, + }, + { + label: '404', + sendError: () => ipcHttpError('rpc:invokeHostMessage', 404, 'Not Found', { error: 'gone' }), + readError: null, + revoked: false, + uncertain: false, + }, + { + label: '503 whose body mentions 403', + sendError: () => ipcServerErrorMentioning403('rpc:invokeHostMessage'), + readError: null, + revoked: false, + uncertain: false, + }, + { + label: 'waking Host named support-401', + sendError: async () => ipcHostWaking('rpc:invokeHostMessage', 'support-401'), + readError: null, + revoked: false, + uncertain: false, + }, + ])( + 'send $label applies the transcript authority decision', + async ({ sendError, readError, revoked, uncertain }) => { + const sendFailure = await sendError() + const readFailure = readError ? await readError() : null + const { invokeHostMessage, listSessions } = installClerumHarness() + invokeHostMessage.mockRejectedValueOnce(sendFailure) + render( + + + + ) + fireEvent.click(screen.getByRole('button', { name: 'Create chat' })) + await waitFor(() => expect(screen.getByTestId('active-chat-id').textContent).not.toBe('')) + const chatId = screen.getByTestId('active-chat-id').textContent || '' + if (readFailure) listSessions.mockRejectedValue(readFailure) + fireEvent.click(screen.getByRole('button', { name: 'Send message' })) + await waitFor(() => expect(screen.getByTestId('send-state').textContent).toBe('settled')) + // Liveness witness: the send reached the producer and failed with it. + expect(invokeHostMessage).toHaveBeenCalledTimes(1) + expect(revokedAgents.has('trader')).toBe(revoked) + expect(uncertainAgents.has('trader')).toBe(uncertain) + if (revoked) { + expect(screen.getByTestId('active-chat-id').textContent).toBe('') + expect(screen.getByTestId('chat-message-count').textContent).toBe('0') + } else { + expect(screen.getByTestId('active-chat-id').textContent).toBe(chatId) + } + } + ) + + it('does not republish a confirmed deleted chat after its pending send resolves', async () => { + const { invokeHostMessage, appendMessages } = installClerumHarness() + let finishInvoke!: (result: { taskId: string; status: string }) => void + invokeHostMessage.mockImplementationOnce( + () => + new Promise(resolve => { + finishInvoke = resolve + }) + ) + render( + + + + ) + + fireEvent.click(screen.getByRole('button', { name: 'Create chat' })) + await waitFor(() => expect(screen.getByTestId('active-chat-id').textContent).not.toBe('')) + const deletedChatId = screen.getByTestId('active-chat-id').textContent || '' + fireEvent.click(screen.getByRole('button', { name: 'Send message' })) + await waitFor(() => expect(invokeHostMessage).toHaveBeenCalledTimes(1)) + fireEvent.change(screen.getByTestId('draft-input'), { + target: { value: 'draft to clear on delete' }, + }) + expect(getComposerDraft(deletedChatId)).toBe('draft to clear on delete') + + fireEvent.click(screen.getByRole('button', { name: `Delete chat ${deletedChatId}` })) + await waitFor(() => + expect(screen.queryByRole('button', { name: `Select chat ${deletedChatId}` })).toBeNull() + ) + expect(getComposerDraft(deletedChatId)).toBe('') + const writesBeforeLateAck = appendMessages.mock.calls.length + + finishInvoke({ taskId: 'late-deleted-task', status: 'pending' }) + await waitFor(() => expect(screen.getByTestId('send-state').textContent).toBe('settled')) + expect(screen.getByTestId('chat-list-ids').textContent).not.toContain(deletedChatId) + await waitFor(() => + expect(screen.getByTestId('latest-list-ids').textContent).not.toContain(deletedChatId) + ) + expect(getComposerDraft(deletedChatId)).toBe('') + expect(appendMessages.mock.calls.length).toBe(writesBeforeLateAck) + + const survivingChatId = screen.getByTestId('active-chat-id').textContent || '' + expect(survivingChatId).not.toBe('') + expect(survivingChatId).not.toBe(deletedChatId) + fireEvent.click(screen.getByRole('button', { name: 'Send message' })) + await waitFor(() => expect(invokeHostMessage).toHaveBeenCalledTimes(2)) + expect(invokeHostMessage.mock.calls[1]?.[1]).toMatchObject({ threadId: survivingChatId }) + }) }) diff --git a/desktop-app/ui/src/hooks/domain/__tests__/useAgentChatController.pendingModel.test.tsx b/desktop-app/ui/src/hooks/domain/__tests__/useAgentChatController.pendingModel.test.tsx index 041c5b4fb9..0e93b4f9de 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/useAgentChatController.pendingModel.test.tsx +++ b/desktop-app/ui/src/hooks/domain/__tests__/useAgentChatController.pendingModel.test.tsx @@ -17,6 +17,7 @@ import { useComposerDraft } from '@hooks/useComposerDraft' import { resetComposerDraftStore } from '@lib/composerDraftStore' import { resetHostModelSelectionStore } from '@lib/hostModelSelectionStore' import type { TaskProgressStreamEvent } from '../../../../../src/types' +import { useHarnessHostAuthority } from './__fixtures__/hostAuthorityHarness' type ProgressHandler = (event: TaskProgressStreamEvent) => void | Promise @@ -63,13 +64,26 @@ function installClerumHarness() { return meta }), rename: vi.fn(async () => undefined), - delete: vi.fn(async () => undefined), + getBindingGeneration: vi.fn(async () => 1), + captureDeleteFence: vi.fn(async (authorityScope: unknown) => ({ + version: 1, + authorityScope, + bindingGeneration: 1, + sessionGeneration: 1, + })), + delete: vi.fn(async () => ({ cleanupPending: false })), loadMessages: vi.fn( async (_agentRef: string, chatId: string) => messagesByChat.get(chatId) || [] ), appendMessages: vi.fn(async (_agentRef: string, chatId: string, messages: unknown[]) => { messagesByChat.set(chatId, [...(messagesByChat.get(chatId) || []), ...messages]) }), + upsertMessages: vi.fn(async (_agentRef: string, chatId: string, messages: unknown[]) => { + const existing = messagesByChat.get(chatId) || [] + const byId = new Map(existing.map(message => [(message as { id: string }).id, message])) + for (const message of messages) byId.set((message as { id: string }).id, message) + messagesByChat.set(chatId, [...byId.values()]) + }), replaceMessages: vi.fn(async (_agentRef: string, chatId: string, messages: unknown[]) => { messagesByChat.set(chatId, [...messages]) }), @@ -113,14 +127,22 @@ function installClerumHarness() { } function AgentChatHarness() { + const hostAuthority = useHarnessHostAuthority() const vm = useAgentChatController({ selectedAgent: 'trader', agentNames: ['trader'], currentTeamId: 'team-1', + chatAuthorityTeamId: 'team-1', + currentEnvironmentKey: 'env-test', currentTeamName: 'Team One', isAuthenticated: true, loadMenuData: true, navItem: 'chat', + onHostAccessRevoked: hostAuthority.onHostAccessRevoked, + onHostAuthorityUncertain: hostAuthority.onHostAuthorityUncertain, + isHostAccessBlocked: hostAuthority.isHostAccessBlocked, + getHostAuthorityEpoch: hostAuthority.getHostAuthorityEpoch, + hostAuthorityRevision: hostAuthority.revision, pushToast: vi.fn(), pushNotification: vi.fn(), agentDisplayName: (agentName: string) => agentName, diff --git a/desktop-app/ui/src/hooks/domain/__tests__/useAgentChatController.test.tsx b/desktop-app/ui/src/hooks/domain/__tests__/useAgentChatController.test.tsx index 9e82e9687c..98fd39c541 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/useAgentChatController.test.tsx +++ b/desktop-app/ui/src/hooks/domain/__tests__/useAgentChatController.test.tsx @@ -124,9 +124,11 @@ describe('useAgentChatController — characterization (D.0)', () => { { limit: 40 } ) expect(clerum.chat.create).toHaveBeenCalledWith('agent-x', 'chat-2') - // Server is source of truth → replace, not append. + // Server is source of truth → replace, not append (nor upsert, which the + // real store keeps apart from append). expect(clerum.chat.replaceMessages).toHaveBeenCalled() expect(clerum.chat.appendMessages).not.toHaveBeenCalled() + expect(clerum.chat.upsertMessages).not.toHaveBeenCalled() // spec 15 §2.2/A19: the first-turn auto-title is shown EPHEMERALLY in the // sidebar (observable output, T4) but is NOT persisted via rename — the // server title is authoritative and a persisted client-derived title would @@ -325,8 +327,18 @@ describe('useAgentChatController — characterization (D.0)', () => { await sendPromise expect(sendResolved).toBe(true) - // user message + assistant reply - expect(clerum.chat.appendMessages).toHaveBeenCalledTimes(2) + // The user message is upserted before the POST and the assistant reply is + // appended after the terminal; the store holds both, in order. + const chatId = result.current.activeChatId! + await waitFor(async () => + expect( + (await clerum.persistedMessages('agent-x', chatId)).map(m => [m.role, m.content]) + ).toEqual([ + ['user', 'hola'], + ['assistant', 'done!'], + ]) + ) + expect(clerum.chat.appendMessages).toHaveBeenCalledTimes(1) expect(clerum.rpc.getTaskResult).toHaveBeenCalledWith('agent-x', 'task-abc', ['agent-x']) expect(spies.pushToast).toHaveBeenCalledWith('Message sent to agent-x.', 'success') }) @@ -423,7 +435,10 @@ describe('useAgentChatController — characterization (D.0)', () => { }) await sendPromise - expect(spies.pushToast).toHaveBeenCalledWith('Message to agent-x failed: LLM down', 'error') + // The failure toast follows the store write of the error reply. + await waitFor(() => + expect(spies.pushToast).toHaveBeenCalledWith('Message to agent-x failed: LLM down', 'error') + ) const appended = clerum.chat.appendMessages.mock.calls.at(-1)?.[2] as | Array<{ isError?: boolean; errorCode?: string }> | undefined @@ -431,6 +446,50 @@ describe('useAgentChatController — characterization (D.0)', () => { expect(appended?.[0]?.errorCode).toBe('provider_error') }) + it('does not toast or repaint a terminal error after its append crosses a team switch', async () => { + const pendingAppend = deferred() + clerum.rpc.invokeHostMessage.mockResolvedValue({ taskId: 'task-old-team' }) + clerum.chat.appendMessages.mockImplementation( + async (_agentRef: string, _chatId: string, messages: Array<{ role?: string }>) => { + if (messages[0]?.role === 'assistant') await pendingAppend.promise + } + ) + const controller = renderController({ currentTeamId: 'team-before' }) + await settleMount() + await act(async () => { + await controller.result.current.handleSendAgentMessage('question') + }) + await waitFor(() => expect(clerum.hasProgressHandler('task-old-team')).toBe(true)) + + clerum.emitTaskProgress('task-old-team', { + type: 'terminal', + data: { + taskId: 'task-old-team', + status: 'failed', + error: { message: 'protected old-team failure', code: 'provider_error' }, + }, + }) + await waitFor(() => + expect( + clerum.chat.appendMessages.mock.calls.some( + call => (call[2] as Array<{ role?: string }>)[0]?.role === 'assistant' + ) + ).toBe(true) + ) + controller.rerender({ currentTeamId: 'team-after' }) + const toastCount = controller.spies.pushToast.mock.calls.length + const notificationCount = controller.spies.pushNotification.mock.calls.length + const progressAfterSwitch = controller.result.current.progressByMessageId + + await act(async () => { + pendingAppend.resolve() + await pendingAppend.promise + }) + expect(controller.spies.pushToast.mock.calls.length).toBe(toastCount) + expect(controller.spies.pushNotification.mock.calls.length).toBe(notificationCount) + expect(controller.result.current.progressByMessageId).toEqual(progressAfterSwitch) + }) + it('4.6 marks the task cancelled without fetching a result', async () => { clerum.rpc.invokeHostMessage.mockResolvedValue({ taskId: 'task-abc' }) const { result } = renderController() @@ -518,11 +577,12 @@ describe('useAgentChatController — characterization (D.0)', () => { sendError = e }) - // Flush the async send setup until the progress subscription is wired. + // Flush the async send setup until the progress subscription is wired. The + // send persists the user turn to the real ChatStore first, which is disk + // I/O that fake timers do not advance: poll for the subscription instead. await act(async () => { - await vi.advanceTimersByTimeAsync(0) + await vi.waitFor(() => expect(clerum.hasProgressHandler('task-abc')).toBe(true)) }) - expect(clerum.hasProgressHandler('task-abc')).toBe(true) await act(async () => { // Open the stream, then go silent and advance past the 30s watchdog. @@ -537,7 +597,12 @@ describe('useAgentChatController — characterization (D.0)', () => { }) await sendPromise - expect(spies.pushToast).toHaveBeenCalledWith(expect.stringContaining('failed'), 'error') + // The failure toast follows the store write of the error reply. + await act(async () => { + await vi.waitFor(() => + expect(spies.pushToast).toHaveBeenCalledWith(expect.stringContaining('failed'), 'error') + ) + }) expect(result.current.agentSending).toBe(false) expect(sendError).toBeNull() // controller catches internally, does not reject }) @@ -568,9 +633,12 @@ describe('useAgentChatController — characterization (D.0)', () => { | undefined expect(appended?.[0]?.isError).toBe(true) expect(appended?.[0]?.content).toContain('Failed to retrieve task result') - expect(spies.pushToast).toHaveBeenCalledWith( - expect.stringContaining('Failed to retrieve result'), - 'error' + // The failure toast follows the store write of the error reply. + await waitFor(() => + expect(spies.pushToast).toHaveBeenCalledWith( + expect.stringContaining('Failed to retrieve result'), + 'error' + ) ) }) @@ -586,10 +654,11 @@ describe('useAgentChatController — characterization (D.0)', () => { }) const sendPromise = result.current.handleSendAgentMessage('hola').catch(() => undefined) + // The user turn is persisted to the real ChatStore (disk I/O fake timers do + // not advance) before the subscription is wired: poll for it. await act(async () => { - await vi.advanceTimersByTimeAsync(0) + await vi.waitFor(() => expect(clerum.hasProgressHandler('task-abc')).toBe(true)) }) - expect(clerum.hasProgressHandler('task-abc')).toBe(true) await act(async () => { // Never emit 'open'. Advance past the 5s connection timeout. @@ -597,9 +666,15 @@ describe('useAgentChatController — characterization (D.0)', () => { }) await sendPromise - const progress = result.current.progressByAgentMessage['agent-x'] - const entry = progress && Object.values(progress)[0] - expect(entry?.status).toBe('error') + // The timeout's recovery reads the store (disk I/O) before it settles the + // entry as an error: poll for the settled state. + await act(async () => { + await vi.waitFor(() => { + const progress = result.current.progressByAgentMessage['agent-x'] + const entry = progress && Object.values(progress)[0] + expect(entry?.status).toBe('error') + }) + }) }) // ── stream-recovery spec: a lost stream reconciles instead of failing ── @@ -796,6 +871,9 @@ describe('useAgentChatController — characterization (D.0)', () => { }) }) + // The reconcile reads the local store (IPC; here the real ChatStore's disk + // read) before it fetches the server turns, so wait for that fetch. + await waitFor(() => expect(clerum.rpc.loadSessionMessages).toHaveBeenCalled()) // The live subscription has fired for the `failed` state, but the reconcile // is still pending → no message may be painted as an error (no flash). const midProgress = result.current.progressByAgentMessage['agent-x'] ?? {} @@ -889,8 +967,22 @@ describe('useAgentChatController — characterization (D.0)', () => { }) }) - // The durable reply lands as a delta; no error/resend and no zombie block. - await waitFor(() => expect(clerum.chat.appendMessages).toHaveBeenCalled()) + // The durable reply lands in the reconciled transcript the controller writes + // back with `replaceMessages`; no error/resend and no zombie block. (The old + // wait on `appendMessages` was met by the user turn the fake store forwarded + // there, never by the reply.) + await waitFor(() => + expect( + clerum.chat.replaceMessages.mock.calls.some(call => + (call[2] as Array<{ role?: string; content?: string }>).some( + message => message.role === 'assistant' && message.content === 'here is your report' + ) + ) + ).toBe(true) + ) + expect(result.current.chatMessages).toContainEqual( + expect.objectContaining({ role: 'assistant', content: 'here is your report' }) + ) expect(spies.pushToast).not.toHaveBeenCalledWith(expect.stringContaining('failed'), 'error') expect(result.current.failedAgentSend).toBeNull() await sendPromise @@ -937,10 +1029,9 @@ describe('useAgentChatController — characterization (D.0)', () => { }) // The durable error is rendered as a failed assistant message. Wait for the - // ASSISTANT append specifically: `appendMessages` fires first for the user - // message, so a bare "was called" wait would race the recovery's append - // (which lands after the `getTaskResult` await) — read the last call only - // once its message is the error bubble. + // ASSISTANT append specifically: the recovery's append lands after the + // `getTaskResult` await — read the last call only once its message is the + // error bubble. await waitFor(() => { const last = clerum.chat.appendMessages.mock.calls.at(-1)?.[2]?.[0] as | { isError?: boolean } @@ -953,9 +1044,12 @@ describe('useAgentChatController — characterization (D.0)', () => { expect(appended?.[0]?.isError).toBe(true) expect(appended?.[0]?.errorCode).toBe('BUDGET_EXCEEDED') expect(appended?.[0]?.content).toContain('Token budget exceeded') - expect(spies.pushToast).toHaveBeenCalledWith( - 'Message to agent-x failed: Token budget exceeded for this workspace.', - 'error' + // The failure toast follows the store write of the error reply. + await waitFor(() => + expect(spies.pushToast).toHaveBeenCalledWith( + 'Message to agent-x failed: Token budget exceeded for this workspace.', + 'error' + ) ) // #654 deliberately retains failed input, while preserving the real // budget error instead of presenting a misleading lost-connection error. @@ -1002,8 +1096,8 @@ describe('useAgentChatController — characterization (D.0)', () => { }) }) - // Wait for the ASSISTANT append specifically (the user-message append fires - // first; the recovery's reply lands after the `getTaskResult` await). + // Wait for the ASSISTANT append specifically (the recovery's reply lands + // after the `getTaskResult` await). await waitFor(() => { const last = clerum.chat.appendMessages.mock.calls.at(-1)?.[2]?.[0] as | { role?: string } @@ -1708,11 +1802,15 @@ describe('useAgentChatController — characterization (D.0)', () => { // The user message was persisted (a chat was auto-created and the typed // input written to it) — the input is durable across reload. expect(clerum.chat.create).toHaveBeenCalled() - expect(clerum.chat.appendMessages).toHaveBeenCalledWith( + expect(clerum.chat.upsertMessages).toHaveBeenCalledWith( 'agent-x', expect.any(String), expect.arrayContaining([expect.objectContaining({ role: 'user', content: 'persist me' })]) ) + const persisted = await clerum.persistedMessages('agent-x', result.current.activeChatId!) + expect(persisted).toContainEqual( + expect.objectContaining({ role: 'user', content: 'persist me', task_id: 'task-ga' }) + ) }) }) }) diff --git a/desktop-app/ui/src/hooks/domain/__tests__/useHostPrewarmController.chatIntegration.test.tsx b/desktop-app/ui/src/hooks/domain/__tests__/useHostPrewarmController.chatIntegration.test.tsx index 9c323fbd27..589b92d67e 100644 --- a/desktop-app/ui/src/hooks/domain/__tests__/useHostPrewarmController.chatIntegration.test.tsx +++ b/desktop-app/ui/src/hooks/domain/__tests__/useHostPrewarmController.chatIntegration.test.tsx @@ -27,6 +27,7 @@ import type { NavItem } from '../../../uiTypes' import { useAgentChatController } from '../useAgentChatController' // React 18/19 needs this flag for act() to flush effects in the test env. import { useHostPrewarmController } from '../useHostPrewarmController' +import { useHarnessHostAuthority } from './__fixtures__/hostAuthorityHarness' import { installMockClerum, uninstallMockClerum } from './__fixtures__/mockClerum' ;(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true @@ -81,18 +82,25 @@ interface HarnessProps { isAuthenticated: boolean navItem: NavItem } - /** Both real hooks, mounted together under the task-tracker the chat controller needs. */ function useBothControllers({ selectedAgent, agentNames, isAuthenticated, navItem }: HarnessProps) { useHostPrewarmController({ agentNames, isAuthenticated }) + const hostAuthority = useHarnessHostAuthority() return useAgentChatController({ selectedAgent, agentNames, currentTeamId: 'team-1', + chatAuthorityTeamId: 'team-1', + currentEnvironmentKey: 'env-test', currentTeamName: 'Team 1', isAuthenticated, loadMenuData: true, navItem, + onHostAccessRevoked: hostAuthority.onHostAccessRevoked, + onHostAuthorityUncertain: hostAuthority.onHostAuthorityUncertain, + isHostAccessBlocked: hostAuthority.isHostAccessBlocked, + getHostAuthorityEpoch: hostAuthority.getHostAuthorityEpoch, + hostAuthorityRevision: hostAuthority.revision, ...stableCallbacks, }) } diff --git a/desktop-app/ui/src/hooks/domain/reconcileChat.ts b/desktop-app/ui/src/hooks/domain/reconcileChat.ts index a186fe027a..cbd395fe2d 100644 --- a/desktop-app/ui/src/hooks/domain/reconcileChat.ts +++ b/desktop-app/ui/src/hooks/domain/reconcileChat.ts @@ -8,14 +8,14 @@ import type { SessionFsmStore } from './sessionFsm' * second invocation while one runs coalesces onto the same promise) and routes * the server truth through the four precedence branches, dispatching * `SERVER_SNAPSHOT`/`RECONCILE_*` to the FSM. The side-effectful branch bodies - * (attach a live stream, materialize/persist idle turns + durable result, evict a - * 404 chat) are injected so the module stays pure and exhaustively unit-testable. + * (attach a live stream, materialize/persist idle turns + durable result, hide + * protected data after an authorization rejection) are injected. * * Telemetry (§4.8): one `stream_recovery` emission per `RECONCILE_FINISHED`, * carrying `{reason, outcome}` and preserving the existing outcome names * (`reconcile_rejoined`, `rejoin_capped_offline`, `reconcile_replaced`, - * `recovered_from_task_result`, `fell_through_to_resend`, `404`) plus new ones - * (`stale_drop`, `offline`, `error`). + * `recovered_from_task_result`, `fell_through_to_resend`) plus `revoked`, + * `stale_drop`, `offline`, and `error`. */ /** Precedence-branch outcomes (stable telemetry names + Fase-3 additions). */ @@ -30,7 +30,8 @@ export type ReconcileOutcome = | 'recovered_error' | 'fell_through_to_resend' | 'stale_drop' - | '404' + | 'revoked' + | 'authority_unverified' | 'offline' | 'error' | 'noop' @@ -77,10 +78,13 @@ export interface ReconcileChatDeps { /** Same post-await teardown guard as `attachLiveTask` (see there). */ stillRelevant: () => boolean ) => Promise - /** Local eviction for a chat the server 404s (cache + sidebar + deselect). */ - evictChat: (chatKey: string) => Promise + /** Hide protected data after an explicit authorization rejection. */ + revokeAccess: (chatKey: string) => void + holdAccess: (chatKey: string) => void isNetworkError: (err: unknown) => boolean isHttp404: (err: unknown) => boolean + isAuthorizationError: (err: unknown) => boolean + isConfirmedHostAccessRevoked: (err: unknown) => boolean telemetry: (event: string, data: Record) => void /** Guard: `false` aborts the reconcile mid-flight (chat switched away). */ isStillRelevant?: (chatKey: string) => boolean @@ -165,9 +169,13 @@ export function createReconcileChat(deps: ReconcileChatDeps): ReconcileChat { try { return await deps.loadSessionMessages(agentRef, chatId, query, stillRelevant) } catch (err) { + // Chat selection may change while this request is pending, but an + // explicit Host authorization denial still applies to the active scope. + // The outer catch checks the principal/team generation before revoking. + if (deps.isAuthorizationError(err)) throw err if (!stillRelevant()) return undefined - // Non-network (404, etc.) rethrows to the branch handler; a network blip - // is retried with backoff up to the cap (mirrors switchToChat P2-A). + // Transport failures retry with backoff. An ambiguous 404 and an + // authorization rejection reach the classification branch immediately. if (!deps.isNetworkError(err) || attempt === attempts - 1) throw err } if (!stillRelevant()) return undefined @@ -216,15 +224,22 @@ export function createReconcileChat(deps: ReconcileChatDeps): ReconcileChat { ) return outcome } catch (err) { - if (!stillRelevant()) { + // An authority rejection applies to the Host, even if the user switched + // from chat A to B while A's request was pending. The reset generation is + // the principal/team boundary; an old scope cannot revoke a new one. + if (deps.isAuthorizationError(err) && generation === startGeneration) { + const confirmed = deps.isConfirmedHostAccessRevoked(err) + if (confirmed) deps.revokeAccess(chatKey) + else deps.holdAccess(chatKey) + deps.fsm.dispatch(chatKey, { type: 'RESET' }) + outcome = confirmed ? 'revoked' : 'authority_unverified' + } else if (!stillRelevant()) { outcome = 'stale_drop' return outcome - } - if (deps.isHttp404(err)) { - await deps.evictChat(chatKey) - deps.fsm.dispatch(chatKey, { type: 'RESET' }) - outcome = '404' - } else if (deps.isNetworkError(err)) { + } else if (deps.isHttp404(err) || deps.isNetworkError(err)) { + // A transcript 404 is anti-enumeration and can also occur while a Host + // wakes. It does not prove the session was deleted. Keep the local chat, + // selection and draft for a later successful reconcile. deps.fsm.dispatch(chatKey, { type: 'WENT_OFFLINE' }) outcome = 'offline' } else { @@ -239,10 +254,11 @@ export function createReconcileChat(deps: ReconcileChatDeps): ReconcileChat { } return outcome } finally { - // RESET already tore the entry down on 404; re-dispatching FINISHED would + // RESET already tore the entry down on revocation; re-dispatching FINISHED would // resurrect an empty entry, so skip the finalizer in that case. if ( - outcome !== '404' && + outcome !== 'revoked' && + outcome !== 'authority_unverified' && generation === startGeneration && (chatGenerations.get(chatKey) ?? 0) === startChatGeneration ) { diff --git a/desktop-app/ui/src/hooks/domain/useAgentChatController.ts b/desktop-app/ui/src/hooks/domain/useAgentChatController.ts index 2f97a56eb9..dca0f70343 100644 --- a/desktop-app/ui/src/hooks/domain/useAgentChatController.ts +++ b/desktop-app/ui/src/hooks/domain/useAgentChatController.ts @@ -12,6 +12,8 @@ import { } from '@lib/chatMessageAttachments' import { truncateTitle } from '@lib/chatTitle' import { + clearAllComposerDrafts, + clearComposerDraft, getComposerDraft, getComposerDraftRevision, setComposerDraft, @@ -44,6 +46,9 @@ import { classifyErrorKind, errorRecoveryHint, extractAssistantReply, + isAuthorizationError, + isConfirmedHostAccessRevoked, + isHttp403, isHttp404, isNetworkError, } from '../../lib/format' @@ -317,7 +322,8 @@ async function loadTaskResultResponseFileAttachments( try { const result = await window.clerum.rpc.getTaskResult(agentRef, taskId, [agentRef]) return buildResponseFileAttachments(result) - } catch { + } catch (error) { + if (isAuthorizationError(error)) throw error return [] } } @@ -333,6 +339,7 @@ interface UseAgentChatControllerParams { agentNames: string[] currentUserId?: string currentTeamId: string + currentEnvironmentKey: string currentTeamName: string isAuthenticated: boolean loadMenuData: boolean @@ -357,6 +364,18 @@ interface UseAgentChatControllerParams { target: AgentConversationNotificationTarget ) => Promise decideApprovalFromNotification: (target: AgentApprovalNotificationTarget) => Promise + onHostAccessRevoked: (agentRef: string) => void + onHostAuthorityUncertain: (agentRef: string) => void + isHostAccessBlocked: (agentRef: string) => boolean + getHostAuthorityEpoch: (agentRef: string) => number + /** Changes whenever a Host is held, released, or the authority scope resets. */ + hostAuthorityRevision: number + /** + * The authenticated principal's own team (`me.teamId`), with no directory + * fallback. The chat delete fence and its tombstones are scoped to it, the + * same value AppService's `getChatDeletionFenceAuthority` uses. + */ + chatAuthorityTeamId: string } export function useAgentChatController({ @@ -364,6 +383,7 @@ export function useAgentChatController({ agentNames, currentUserId, currentTeamId, + currentEnvironmentKey, currentTeamName, isAuthenticated, loadMenuData, @@ -375,9 +395,26 @@ export function useAgentChatController({ showDesktopNotification, openAgentConversationFromNotification, decideApprovalFromNotification, + onHostAccessRevoked, + onHostAuthorityUncertain, + isHostAccessBlocked, + getHostAuthorityEpoch, + hostAuthorityRevision, + chatAuthorityTeamId, }: UseAgentChatControllerParams) { const chatStore = useChatStore() - const authenticatedScope = `${currentUserId ?? 'unknown-user'}:${currentTeamId}` + const userScopeKey = currentUserId ?? 'unknown-user' + const normalizedTeamId = String(currentTeamId || '').trim() || null + const fenceTeamId = chatAuthorityTeamId.trim() || null + const authorityScope = useMemo( + () => ({ + environmentKey: currentEnvironmentKey, + userId: userScopeKey, + teamId: fenceTeamId, + }), + [currentEnvironmentKey, fenceTeamId, userScopeKey] + ) + const authenticatedScope = `${currentEnvironmentKey}:${userScopeKey}:${normalizedTeamId ?? ''}` useEffect(() => { chatStore.setRemoteCacheScope( @@ -404,6 +441,17 @@ export function useAgentChatController({ // activeChatId) are injected through `chatListHostRef`, filled each render once // those parent callbacks are defined (created here so the hook can receive it). const chatListHostRef = useRef(null) + const hideHostAccessRef = useRef< + ((agentRef: string, kind: 'revoked' | 'uncertain') => void) | null + >(null) + const onCatalogHostAccessRevoked = useCallback( + (agentRef: string) => hideHostAccessRef.current!(agentRef, 'revoked'), + [] + ) + const onCatalogHostAuthorityUncertain = useCallback( + (agentRef: string) => hideHostAccessRef.current!(agentRef, 'uncertain'), + [] + ) const autoSelectedChatIdRef = useRef(null) const chatListCtl = useChatListController({ selectedAgent, @@ -413,10 +461,16 @@ export function useAgentChatController({ // User portion of the scope only (NOT the combined user:team scopeKey): the // pending-rename queue is per-user and must survive a team-switch (R1-H1). authUserKey: currentUserId ?? 'unknown-user', + authorityScope, loadMenuData, chatStore, fsm, host: chatListHostRef, + isHostAccessBlocked, + getHostAuthorityEpoch, + hostAuthorityRevision, + onHostAccessRevoked: onCatalogHostAccessRevoked, + onHostAuthorityUncertain: onCatalogHostAuthorityUncertain, }) const { chatList, @@ -425,6 +479,7 @@ export function useAgentChatController({ chatListHasMoreRemoteSessions, latestChatSessions, latestChatSessionsLoading, + hideAgent, loadChatList, loadMoreChatSessions, setChatListLoading, @@ -441,6 +496,7 @@ export function useAgentChatController({ readPendingSelection, writePendingSelection, clearPendingSelection, + isChatDeleted, } = chatListCtl const [chatMessages, setChatMessages] = useState([]) const [chatMessagesLoading, setChatMessagesLoading] = useState(false) @@ -500,6 +556,8 @@ export function useAgentChatController({ failRetainedSend, } = retainedSends const sendScopeGeneration = useRef(0) + const currentAuthScopeRef = useRef(`${isAuthenticated}:${authenticatedScope}`) + currentAuthScopeRef.current = `${isAuthenticated}:${authenticatedScope}` // #654 M5 — the userMessageId of the most recent send that retained its // payload. A retry compares it before and after to learn whether it retained // a snapshot of its own; `sendAgentMessage` has several early returns (image @@ -510,7 +568,9 @@ export function useAgentChatController({ const scope = `${isAuthenticated}:${authenticatedScope}` if (lastRetentionScope.current === scope) return lastRetentionScope.current = scope + clearAllComposerDrafts() sendScopeGeneration.current += 1 + agentSendSetupOwnerRef.current = null agentSendInFlightRef.current = false setAgentSending(false) retainedSends.resetRetainedSendStore() @@ -522,6 +582,7 @@ export function useAgentChatController({ const activityInFlightByAgentRef = useRef>({}) const activityTaskToMessageByAgentRef = useRef>>({}) const agentSendInFlightRef = useRef(false) + const agentSendSetupOwnerRef = useRef<{ id: symbol; agentRef: string } | null>(null) // (Phase-2 AbortController + `recoveringTaskIdByKeyRef` removed in Fase 5b: the // reconcile gate's per-call `isRelevant` covers the A→B→A abort, and the // epoch-anchored zombie-ack in `settleIdle` (R2) covers the residual-recreated- @@ -537,6 +598,7 @@ export function useAgentChatController({ selectedAgent, }) const activeChatSwitchRequestRef = useRef(null) + const activeChatSwitchKeyRef = useRef(null) // Latest `pushToast` for the hoisted tracker callbacks (onTrackerTerminal reads // it) so that `tracker.setCallbacks` can run once (cross-ref D.3 M1) without // closing over a stale toast fn. Assigned in an effect, not the render body @@ -556,7 +618,8 @@ export function useAgentChatController({ loadSessionMessages: ReconcileChatDeps['loadSessionMessages'] attachLiveTask: ReconcileChatDeps['attachLiveTask'] settleIdle: ReconcileChatDeps['settleIdle'] - evictChat: ReconcileChatDeps['evictChat'] + revokeAccess: ReconcileChatDeps['revokeAccess'] + holdAccess: ReconcileChatDeps['holdAccess'] } | null>(null) const reconcileChatRef = useRef(null) if (!reconcileChatRef.current) { @@ -568,9 +631,12 @@ export function useAgentChatController({ reconcileBranchesRef.current!.attachLiveTask(chatKey, resp, epoch, stillRelevant), settleIdle: (chatKey, resp, epoch, hint, stillRelevant) => reconcileBranchesRef.current!.settleIdle(chatKey, resp, epoch, hint, stillRelevant), - evictChat: chatKey => reconcileBranchesRef.current!.evictChat(chatKey), + revokeAccess: chatKey => reconcileBranchesRef.current!.revokeAccess(chatKey), + holdAccess: chatKey => reconcileBranchesRef.current!.holdAccess(chatKey), isNetworkError, isHttp404, + isAuthorizationError, + isConfirmedHostAccessRevoked, telemetry: (event, data) => console.log(`[telemetry] ${event}`, data), }) } @@ -811,6 +877,7 @@ export function useAgentChatController({ }, []) const resetChat = useCallback(() => { + clearAllComposerDrafts() setActivityByAgentMessage({}) setProgressByAgentMessage({}) resetComposerAttachments() @@ -828,6 +895,7 @@ export function useAgentChatController({ // main-process half (`stopAllStreams` on logout) is Fase 4. tracker.releaseAll() sendScopeGeneration.current += 1 + agentSendSetupOwnerRef.current = null agentSendInFlightRef.current = false retainedSends.resetRetainedSendStore() // Abort any in-flight reconcile so a run mid-backoff can't resurrect a @@ -864,13 +932,22 @@ export function useAgentChatController({ // chat, and a task in flight is rejoined via the tracker D.3 already mounts. const switchToChat = useCallback( async (agentRef: string, chatId: string) => { - const switchRequest = Symbol(`${agentRef}:${chatId}`) - activeChatSwitchRequestRef.current = switchRequest + if (isHostAccessBlocked(agentRef) || isChatDeleted(agentRef, chatId)) return const key = makeTaskKey(agentRef, chatId) const visibleBeforeSwitch = activeChatVisibilityRef.current const reopeningActiveChat = visibleBeforeSwitch.selectedAgent === agentRef && visibleBeforeSwitch.activeChatId === chatId + const switchKey = `${currentTeamId}\u0000${key}` + const canReuseActiveSwitch = + reopeningActiveChat && activeChatSwitchKeyRef.current === switchKey + const switchRequest = canReuseActiveSwitch + ? activeChatSwitchRequestRef.current! + : Symbol(`${agentRef}:${chatId}`) + if (!canReuseActiveSwitch) { + activeChatSwitchRequestRef.current = switchRequest + activeChatSwitchKeyRef.current = switchKey + } // A same-chat click must coalesce onto an already-running loud recovery. // Superseding it would discard the stream-loss caller that owns the Resend // or durable-result UX. Genuine chat changes still cancel stale work. @@ -882,11 +959,13 @@ export function useAgentChatController({ selectedAgent: agentRef, } setActiveChatId(chatId) - setChatMessages([]) - setChatMessagesLoading(true) - setHasOlderMessages(false) - cancelOlderMessagesLoad() - loadedLocalMessageCountRef.current = 0 + if (!canReuseActiveSwitch) { + setChatMessages([]) + setChatMessagesLoading(true) + setHasOlderMessages(false) + cancelOlderMessagesLoad() + loadedLocalMessageCountRef.current = 0 + } // The error/resend banner is per-chat view state, not global: clear it on // every chat switch (same-agent switches don't trip the selectedAgent // cleanup effect, so without this a stream error from chat A would bleed @@ -894,6 +973,7 @@ export function useAgentChatController({ setAgentError(null) setFailedAgentSend(null) await chatStore.setLastActive(agentRef, chatId) + if (isHostAccessBlocked(agentRef) || isChatDeleted(agentRef, chatId)) return unfillableServerGapUpperBoundsRef.current.delete(key) @@ -915,7 +995,13 @@ export function useAgentChatController({ // below requests a delta after the newest cached server turn. let cached: Awaited> = [] try { - cached = await chatStore.loadMessages(agentRef, chatId, LOCAL_MESSAGE_PAGE_SIZE) + cached = canReuseActiveSwitch + ? ([...chatMessagesRef.current] as AgentChatMessage[]) + : ((await chatStore.loadMessages( + agentRef, + chatId, + LOCAL_MESSAGE_PAGE_SIZE + )) as AgentChatMessage[]) } catch (error) { console.warn('[chat-history] failed to load local messages', { agentRef, chatId, error }) } @@ -928,7 +1014,7 @@ export function useAgentChatController({ if (!isStillActive()) return loadedLocalMessageCountRef.current = cached.length setHasOlderMessages(hasOlderLocalMessages || hasOlderServerMessages) - setChatMessages(cached as AgentChatMessage[]) + if (!canReuseActiveSwitch) setChatMessages(cached as AgentChatMessage[]) // Phase 1 rendered the cache → clear the blocking spinner. The reconcile // runs under the `syncing` indicator (RECONCILE_STARTED, dispatched by the // reconcile gate) instead. @@ -951,7 +1037,7 @@ export function useAgentChatController({ // clean. tracker.resetRejoinAttempts(key) - // PHASE 2/3 — the single reconcile gate (§4.3) owns fetch+retry, 404 evict, + // PHASE 2/3 — the single reconcile gate (§4.3) owns fetch+retry, ambiguous 404, // offline, the SERVER_SNAPSHOT, the idle replace (auto-title / S4 upsert / // epoch-anchored zombie-ack) and the live rejoin (approval re-seed). // `taskIdHint` = the pre-await zombie's task, which drives the durable @@ -968,18 +1054,22 @@ export function useAgentChatController({ // own `isActive()` guard still prevents rendering into the wrong chat. const zombieBefore = tracker.get(key) const latestCachedTurn = latestServerTurnNumber(cached as AgentChatMessage[]) + const reconcileFromTurn = + latestCachedTurn === undefined ? undefined : Math.max(0, latestCachedTurn - 1) await reconcileChat(key, { reason: 'switch_to_chat', taskIdHint: zombieBefore?.taskId, isRelevant: isStillActive, messagesQuery: { limit: SERVER_TURN_PAGE_SIZE, - ...(latestCachedTurn !== undefined ? { afterTurn: latestCachedTurn } : {}), + ...(reconcileFromTurn !== undefined ? { afterTurn: reconcileFromTurn } : {}), }, }) }, [ currentTeamId, + isHostAccessBlocked, + isChatDeleted, dispatchSession, tracker, reconcileChat, @@ -1002,6 +1092,7 @@ export function useAgentChatController({ handleRenameChat, handleRenameChatForAgent, applyLocalTitleOnly, + captureChatDeleteFence, handleDeleteChat, handleDeleteChatForAgent, } = chatListCtl @@ -1018,22 +1109,30 @@ export function useAgentChatController({ }, shouldAutoSelectLatest: () => navItem === DESKTOP_ROUTES.chat, pushToast, + agentDisplayName, } }) const handleSelectChat = useCallback( async (chatId: string) => { - if (!selectedAgent) return + if ( + !selectedAgent || + isHostAccessBlocked(selectedAgent) || + isChatDeleted(selectedAgent, chatId) + ) + return autoSelectedChatIdRef.current = null await switchToChat(selectedAgent, chatId) }, - [selectedAgent, switchToChat] + [selectedAgent, isHostAccessBlocked, isChatDeleted, switchToChat] ) const handleLoadOlderMessages = useCallback(async () => { if (!selectedAgent || !activeChatId || olderMessagesRequestRef.current) return const requestAgent = selectedAgent const requestChatId = activeChatId + const requestScope = sendScopeGeneration.current + const requestScopeIdentity = currentAuthScopeRef.current const requestToken = Symbol(`${requestAgent}:${requestChatId}`) olderMessagesRequestRef.current = requestToken const isStillRelevant = () => { @@ -1132,6 +1231,18 @@ export function useAgentChatController({ oldestMergedTurn > 1) ) } catch (error) { + if ( + isAuthorizationError(error) && + requestScope === sendScopeGeneration.current && + requestScopeIdentity === currentAuthScopeRef.current && + !isHostAccessBlocked(requestAgent) + ) { + hideHostAccessRef.current!( + requestAgent, + isConfirmedHostAccessRevoked(error) ? 'revoked' : 'uncertain' + ) + return + } console.warn('[chat-history] failed to load older messages', { agentRef: requestAgent, chatId: requestChatId, @@ -1152,12 +1263,13 @@ export function useAgentChatController({ ignoredServerGapUpperBounds, markServerGapUnfillable, selectedAgent, + isHostAccessBlocked, tracker, ]) // Agent selection → load chats useEffect(() => { - if (!selectedAgent) { + if (!selectedAgent || isHostAccessBlocked(selectedAgent)) { autoSelectedChatIdRef.current = null activeChatVisibilityRef.current = { ...activeChatVisibilityRef.current, @@ -1215,7 +1327,7 @@ export function useAgentChatController({ setChatListLoading(true) ;(async () => { const result = await loadChatList(selectedAgent) - if (cancelled) return + if (cancelled || isHostAccessBlocked(selectedAgent)) return setChatListLoading(false) if (!result) { setChatMessagesLoading(false) @@ -1279,7 +1391,7 @@ export function useAgentChatController({ return () => { cancelled = true } - }, [cancelOlderMessagesLoad, currentTeamId, navItem, selectedAgent]) + }, [cancelOlderMessagesLoad, currentTeamId, navItem, selectedAgent, isHostAccessBlocked]) const mapComposerAttachmentsToHostRequest = ( attachments: ComposerImageAttachment[] @@ -1449,6 +1561,14 @@ export function useAgentChatController({ const appendAssistantMessage = useCallback( async (agentName: string, chatId: string | null, message: AgentChatMessage) => { + const messageScope = sendScopeGeneration.current + const messageScopeIdentity = currentAuthScopeRef.current + const mayAppend = () => + messageScope === sendScopeGeneration.current && + messageScopeIdentity === currentAuthScopeRef.current && + !isHostAccessBlocked(agentName) && + (!chatId || !isChatDeleted(agentName, chatId)) + if (!mayAppend()) return // Update the in-memory view FIRST (synchronously), then persist. For a // rejoined task the in-flight placeholder hides the moment the tracker // emits 'completed'; landing the reply before the await closes the visual @@ -1460,6 +1580,7 @@ export function useAgentChatController({ if (chatId) { try { await chatStore.appendMessages(agentName, chatId, [message]) + if (!mayAppend()) return bumpActivity(agentName, chatId, new Date(message.timestamp).toISOString()) } catch { // persistence is best-effort; the in-memory view already updated @@ -1469,12 +1590,13 @@ export function useAgentChatController({ // onTrackerTerminal can't escape before its final tracker.ack(key) — a // throw here would otherwise leak the tracked (agent, chat) entry. try { + if (!mayAppend()) return pushAssistantReplyNotification(agentName, message, chatId) } catch { // notification delivery is best-effort } }, - [bumpActivity, chatStore, pushAssistantReplyNotification] + [bumpActivity, chatStore, pushAssistantReplyNotification, isHostAccessBlocked, isChatDeleted] ) // ─── reconcileChat branch callbacks (§4.3) ─── @@ -1511,6 +1633,8 @@ export function useAgentChatController({ const v = activeChatVisibilityRef.current return v.selectedAgent === agentRef && v.activeChatId === chatId } + const mayHydrate = () => + stillRelevant() && !isHostAccessBlocked(agentRef) && !isChatDeleted(agentRef, chatId) const cached = (await chatStore .loadMessages(agentRef, chatId, LOCAL_MESSAGE_PAGE_SIZE) .catch(error => { @@ -1534,7 +1658,7 @@ export function useAgentChatController({ cached: localMessages, stale: true, }) - if (!stillRelevant()) return staleResult() + if (!mayHydrate()) return staleResult() if (!isActive() || tracker.get(chatKey)) { return { rendered: localMessages, replaced: false, cached: localMessages, stale: false } } @@ -1545,9 +1669,9 @@ export function useAgentChatController({ const localHasServerTurns = localMessages.some( message => serverTurnNumber(message) !== undefined ) + const responseContainsAllTurns = + resp.totalTurns !== undefined ? resp.turns.length >= resp.totalTurns : !resp.hasMoreBefore if (!localHasServerTurns && localMessages.length > 0) { - const responseContainsAllTurns = - resp.totalTurns === undefined || resp.totalTurns === resp.turns.length if (responseContainsAllTurns && hydrated.length <= localMessages.length) { return { rendered: localMessages, replaced: false, cached: localMessages, stale: false } } @@ -1556,7 +1680,7 @@ export function useAgentChatController({ hydrated, taskIdHint ? await loadTaskResultResponseFileAttachments(agentRef, taskIdHint) : [] ) - if (!stillRelevant()) return staleResult() + if (!mayHydrate()) return staleResult() if (!isActive() || tracker.get(chatKey)) { return { rendered: localMessages, replaced: false, cached: localMessages, stale: false } } @@ -1566,7 +1690,10 @@ export function useAgentChatController({ // of persisting both copies. A reported live task keeps the merge path so // its optimistic bubbles remain visible until task-scoped reconciliation. const replaceSettledLegacyCache = - localMessages.length > 0 && !localHasServerTurns && !resp.activeTaskId + localMessages.length > 0 && + !localHasServerTurns && + !resp.activeTaskId && + responseContainsAllTurns const rendered = localMessages.length > 0 ? mergeServerMessages( @@ -1578,7 +1705,7 @@ export function useAgentChatController({ : hydratedWithAttachments const hasOlderAfterMerge = hasOlderFromServer || nextServerBackfillBeforeTurn(rendered) !== undefined - if (!stillRelevant()) return staleResult(rendered) + if (!mayHydrate()) return staleResult(rendered) if (latestPageFallback) { setHasOlderMessages(hasOlderAfterMerge) } else { @@ -1587,11 +1714,11 @@ export function useAgentChatController({ if (sameMessageSequence(rendered, localMessages)) { return { rendered: localMessages, replaced: false, cached: localMessages, stale: false } } - if (!stillRelevant()) return staleResult(rendered) + if (!mayHydrate()) return staleResult(rendered) chatMessagesRef.current = rendered setChatMessages(rendered) const meta = await chatStore.createChat(agentRef, chatId) - if (!stillRelevant()) return staleResult(rendered) + if (!mayHydrate()) return staleResult(rendered) if (resp.activeTaskId) { await chatStore.replaceMessages(agentRef, chatId, rendered, { activeTaskIds: [resp.activeTaskId], @@ -1599,7 +1726,7 @@ export function useAgentChatController({ } else { await chatStore.replaceMessages(agentRef, chatId, rendered) } - if (!stillRelevant()) return staleResult(rendered) + if (!mayHydrate()) return staleResult(rendered) if (!isActive()) { return { rendered, replaced: true, cached: localMessages, stale: false } } @@ -1627,6 +1754,8 @@ export function useAgentChatController({ }, [ tracker, + isHostAccessBlocked, + isChatDeleted, chatStore.loadMessages, chatStore.createChat, chatStore.replaceMessages, @@ -1738,7 +1867,10 @@ export function useAgentChatController({ if (cached.some(m => m.role === 'assistant' && m.task_id === taskIdHint)) return 'noop' const taskResult = await window.clerum.rpc .getTaskResult(agentRef, taskIdHint, [agentRef]) - .catch(() => null) + .catch(error => { + if (isAuthorizationError(error)) throw error + return null + }) if (!taskResult) return 'fell_through_to_resend' // Post-await teardown guard (parity with attachLiveTask): a `reset()` during // the getTaskResult await must not let the durable materialization persist a @@ -1789,29 +1921,89 @@ export function useAgentChatController({ [fsm, tracker, hydrateActiveChatFromServer, appendAssistantMessage] ) - const reconcileEvictChat = useCallback( - async chatKey => { - const { agentRef, chatId } = parseTaskKey(chatKey) - // The local cache referenced a chat the server 404s. Post-spec this - // shouldn't happen (every chat originates from a POST the server records) → - // treat as a stale-cache bug: log + evict locally (parity with the old - // switchToChat 404 branch, so its characterization pin keeps warning). - console.warn('[reconcileChat] chat unknown to server, evicting local cache', { - agentRef, - chatId, + const hideHostAccess = useCallback( + (agentRef: string, kind: 'revoked' | 'uncertain') => { + for (const chatKey of Object.keys(fsm.getSnapshot())) { + if (parseTaskKey(chatKey).agentRef !== agentRef) continue + tracker.release(chatKey as TaskKey) + reconcileChatRef.current?.supersede(chatKey) + fsm.dispatch(chatKey, { type: 'RESET' }) + } + const stopActivity = activityUnsubByAgentRef.current[agentRef] + if (stopActivity) { + delete activityUnsubByAgentRef.current[agentRef] + void stopActivity().catch(() => undefined) + } + delete activityInFlightByAgentRef.current[agentRef] + delete activityTaskToMessageByAgentRef.current[agentRef] + setActivityByAgentMessage(previous => { + const { [agentRef]: _removed, ...remaining } = previous + return remaining }) - tracker.release(chatKey as TaskKey) - await chatStore.deleteChat(agentRef, chatId).catch(() => undefined) - chatStore.clearCachedRemoteData() - removeFromList(chatId) - const view = activeChatVisibilityRef.current - if (view.selectedAgent === agentRef && view.activeChatId === chatId) { - activeChatVisibilityRef.current = { ...view, activeChatId: null } - setActiveChatId(null) - setChatMessages([]) + setProgressByAgentMessage(previous => { + const { [agentRef]: _removed, ...remaining } = previous + return remaining + }) + if (kind === 'revoked') { + hideAgent(agentRef) + const view = activeChatVisibilityRef.current + if (agentSendSetupOwnerRef.current?.agentRef === agentRef) { + agentSendSetupOwnerRef.current = null + agentSendInFlightRef.current = false + setAgentSending(false) + } + if (view.selectedAgent === agentRef) { + chatStore.clearCachedRemoteData() + clearList() + resetComposerAttachments() + activeChatVisibilityRef.current = { ...view, selectedAgent: null, activeChatId: null } + setActiveChatId(null) + setChatMessages([]) + setChatMessagesLoading(false) + setHasOlderMessages(false) + cancelOlderMessagesLoad() + } } + if (kind === 'revoked') onHostAccessRevoked(agentRef) + else onHostAuthorityUncertain(agentRef) + }, + [ + tracker, + fsm, + hideAgent, + chatStore.clearCachedRemoteData, + clearList, + resetComposerAttachments, + cancelOlderMessagesLoad, + onHostAccessRevoked, + onHostAuthorityUncertain, + ] + ) + const revokeHostAccess = useCallback( + (agentRef: string) => hideHostAccess(agentRef, 'revoked'), + [hideHostAccess] + ) + const holdHostAccess = useCallback( + (agentRef: string) => hideHostAccess(agentRef, 'uncertain'), + [hideHostAccess] + ) + useEffect(() => { + hideHostAccessRef.current = hideHostAccess + }, [hideHostAccess]) + + const reconcileRevokeAccess = useCallback( + chatKey => { + const { agentRef } = parseTaskKey(chatKey) + revokeHostAccess(agentRef) }, - [tracker, chatStore.deleteChat, chatStore.clearCachedRemoteData, removeFromList] + [revokeHostAccess] + ) + const reconcileHoldAccess = useCallback( + chatKey => { + const { agentRef } = parseTaskKey(chatKey) + holdHostAccess(agentRef) + }, + [holdHostAccess] ) useEffect(() => { @@ -1830,9 +2022,11 @@ export function useAgentChatController({ return [] })) as AgentChatMessage[] const latestCachedTurn = latestServerTurnNumber(cached) + const reconcileFromTurn = + latestCachedTurn === undefined ? undefined : Math.max(0, latestCachedTurn - 1) requestedQuery = { ...requestedQuery, - ...(latestCachedTurn !== undefined ? { afterTurn: latestCachedTurn } : {}), + ...(reconcileFromTurn !== undefined ? { afterTurn: reconcileFromTurn } : {}), } } let response = await chatStore.loadSessionMessages( @@ -1921,14 +2115,16 @@ export function useAgentChatController({ }, attachLiveTask: reconcileAttachLiveTask, settleIdle: reconcileSettleIdle, - evictChat: reconcileEvictChat, + revokeAccess: reconcileRevokeAccess, + holdAccess: reconcileHoldAccess, } }, [ chatStore.loadSessionMessages, chatStore.loadMessages, reconcileAttachLiveTask, reconcileSettleIdle, - reconcileEvictChat, + reconcileRevokeAccess, + reconcileHoldAccess, ]) // ─── Tracker callbacks + subscription (the post-D.3 fire & forget glue) ─── @@ -1936,9 +2132,27 @@ export function useAgentChatController({ const onTrackerTerminal = useCallback( async (key: TaskKey, state: TaskState) => { const { agentRef, chatId } = parseTaskKey(key) + const terminalScope = sendScopeGeneration.current + const terminalScopeIdentity = currentAuthScopeRef.current + const terminalStillAuthorized = () => + terminalScope === sendScopeGeneration.current && + terminalScopeIdentity === currentAuthScopeRef.current && + !isHostAccessBlocked(agentRef) && + !isChatDeleted(agentRef, chatId) // `task_duration_seconds` telemetry is now emitted by the coordinator // (`fireTerminal`, §4.8) — the lifecycle owner — so it is not duplicated here. const result = state.terminalResult + if (result?.kind === 'error' && result.source === 'authority') { + if ( + terminalScope === sendScopeGeneration.current && + terminalScopeIdentity === currentAuthScopeRef.current && + !isHostAccessBlocked(agentRef) + ) { + if (result.authority === 'revoked') revokeHostAccess(agentRef) + else holdHostAccess(agentRef) + } + return + } // ── stream-recovery: a lost progress stream is NOT a task failure ── // The task is durable server-side (D.1/T2.1), so before surfacing the scary @@ -1980,6 +2194,7 @@ export function useAgentChatController({ // Telemetry for this recovery is emitted once by the reconcile gate (§4.8, // `deps.telemetry`) with the real outcome, so no manual log here. const applyFallback = () => { + if (!terminalStillAuthorized()) return dropActivity() liveDepsRef.current.pushToast(`Message to ${agentRef} failed.`, 'error') updateMessageProgress(agentRef, state.userMessageId, () => ({ @@ -2023,9 +2238,31 @@ export function useAgentChatController({ // Mirrors taskTracker's interpretation of `getTaskResult` (source:'failed' // vs a durable reply) so the rendered bubble matches the direct path. const recoverFromDurableTaskResult = async (): Promise => { - const taskResult = await window.clerum.rpc - .getTaskResult(agentRef, state.taskId, [agentRef]) - .catch(() => null) + const mayRecover = () => + terminalScope === sendScopeGeneration.current && + terminalScopeIdentity === currentAuthScopeRef.current && + !isHostAccessBlocked(agentRef) && + !isChatDeleted(agentRef, chatId) + let taskResult: Awaited> + try { + taskResult = await window.clerum.rpc.getTaskResult(agentRef, state.taskId, [agentRef]) + } catch (error) { + if (isAuthorizationError(error)) { + if ( + terminalScope === sendScopeGeneration.current && + terminalScopeIdentity === currentAuthScopeRef.current && + !isHostAccessBlocked(agentRef) + ) { + hideHostAccessRef.current!( + agentRef, + isConfirmedHostAccessRevoked(error) ? 'revoked' : 'uncertain' + ) + } + return true + } + return false + } + if (!mayRecover()) return true if (!taskResult) return false // The awaits in this branch (incl. `getTaskResult` above) open a window @@ -2061,6 +2298,7 @@ export function useAgentChatController({ ...(errorCode ? { errorCode } : {}), ...(errorProvider ? { errorProvider } : {}), }) + if (!mayRecover()) return true updateMessageProgress(agentRef, state.userMessageId, () => ({ taskId: state.taskId, status: 'error', @@ -2099,6 +2337,7 @@ export function useAgentChatController({ task_id: state.taskId, ...(attachments.length ? { attachments } : {}), }) + if (!mayRecover()) return true updateMessageProgress(agentRef, state.userMessageId, () => ({ taskId: state.taskId, status: 'completed', @@ -2157,6 +2396,7 @@ export function useAgentChatController({ taskIdHint: state.taskId, isRelevant: isActive, }) + if (!terminalStillAuthorized()) return switch (outcome) { case 'reconcile_rejoined': case 'rejoin_capped_offline': @@ -2210,8 +2450,9 @@ export function useAgentChatController({ dropActivity() setIdle() break - case '404': - // The reconcile evicted the chat + reset the FSM already. + case 'revoked': + case 'authority_unverified': + // The authorization rejection hid the protected chat + reset the FSM. dropActivity() break case 'offline': @@ -2257,6 +2498,7 @@ export function useAgentChatController({ ...(toolSteps ? { toolSteps } : {}), }) } + if (!terminalStillAuthorized()) return liveDepsRef.current.pushToast(`Message sent to ${agentRef}.`, 'success') } else if (result?.kind === 'error') { // Terminal failure: the payload stays retained for explicit recovery. @@ -2278,6 +2520,7 @@ export function useAgentChatController({ errorCode: result.code, errorProvider: result.provider, }) + if (!terminalStillAuthorized()) return liveDepsRef.current.pushToast(`Message to ${agentRef} failed: ${result.message}`, 'error') } else if (result.source === 'result_fetch') { await appendAssistantMessage(agentRef, chatId, { @@ -2288,10 +2531,12 @@ export function useAgentChatController({ task_id: state.taskId, isError: true, }) + if (!terminalStillAuthorized()) return liveDepsRef.current.pushToast(`Failed to retrieve result for ${agentRef}.`, 'error') } // `source: 'stream'` is intercepted + reconciled at the top of this handler. } + if (!terminalStillAuthorized()) return // The Resend payload is dropped by the coordinator on `release` below (B15). // Reflect the terminal state into progress/activity directly. The live // subscription also does this, but a task can finish before its chat's @@ -2357,6 +2602,10 @@ export function useAgentChatController({ updateMessageProgress, updateMessageActivity, reconcileChat, + isHostAccessBlocked, + isChatDeleted, + revokeHostAccess, + holdHostAccess, ] ) @@ -2498,14 +2747,20 @@ export function useAgentChatController({ preserveComposer = false ) => { const sendScope = sendScopeGeneration.current + const sendScopeIdentity = currentAuthScopeRef.current const originalDraftChat = activeChatVisibilityRef.current.activeChatId - const originalDraftRevision = getComposerDraftRevision(originalDraftChat) + const originalDraftAgent = selectedAgent + const originalDraftRevision = getComposerDraftRevision( + originalDraftChat, + originalDraftAgent ?? undefined + ) const originalAttachmentRevision = composerAttachmentRevisionRef.current const trimmedContent = content.trim() const effectiveAttachments = [...attachments] const effectiveReferences = [...references] if ( !selectedAgent || + isHostAccessBlocked(selectedAgent) || (!trimmedContent && !effectiveAttachments.length && !effectiveReferences.length) ) return @@ -2514,10 +2769,19 @@ export function useAgentChatController({ // Read the chat id from the ref (updated synchronously by auto-create / // switchToChat) rather than state, which may not have committed yet. const currentChatId = activeChatVisibilityRef.current.activeChatId + if (currentChatId && isChatDeleted(selectedAgent, currentChatId)) return if (currentChatId && tracker.get(makeTaskKey(selectedAgent, currentChatId))) return if (agentSendInFlightRef.current) return - agentSendInFlightRef.current = true const sendAgent = selectedAgent + const sendSetupOwner = { id: Symbol('agent-send-setup'), agentRef: sendAgent } + agentSendSetupOwnerRef.current = sendSetupOwner + agentSendInFlightRef.current = true + const releaseSendSetup = () => { + if (agentSendSetupOwnerRef.current !== sendSetupOwner) return + agentSendSetupOwnerRef.current = null + agentSendInFlightRef.current = false + setAgentSending(false) + } // Issue #654 — send-time image guard. Runs BEFORE the chat is created (a // blocked visual send must not leave a stray empty chat) and long before // the optimistic message/attachment clear. `readHostModelSelection` is the @@ -2537,7 +2801,7 @@ export function useAgentChatController({ 'Image attachments are not available for the selected model yet.' setAgentError(blocker) pushToast(blocker, 'error') - agentSendInFlightRef.current = false + releaseSendSetup() return } // Capture the model the guard just validated; the request below must @@ -2556,10 +2820,18 @@ export function useAgentChatController({ const blocker = error instanceof Error ? error.message : String(error) setAgentError(blocker) pushToast(blocker, 'error') - agentSendInFlightRef.current = false + releaseSendSetup() return } - let sendChatId = activeChatId + let sendChatId = currentChatId ?? activeChatId + const sendStillAuthorized = () => + sendScope === sendScopeGeneration.current && + sendScopeIdentity === currentAuthScopeRef.current && + !isHostAccessBlocked(sendAgent) && + (!sendChatId || !isChatDeleted(sendAgent, sendChatId)) + const sendChatIsVisible = () => + activeChatVisibilityRef.current.selectedAgent === sendAgent && + activeChatVisibilityRef.current.activeChatId === sendChatId // Captured for the retention snapshot: the model this attempt actually // asked for (issue #654), readable from the catch below. let requestModelForRetention: string | undefined @@ -2569,14 +2841,18 @@ export function useAgentChatController({ // the freshly-created meta so the lookup can fall back to it. let autoCreatedMeta: Awaited> | undefined if (!sendChatId) { + const wasSendChatVisible = sendChatIsVisible() try { const chatId = crypto.randomUUID() const meta = await chatStore.createChat(sendAgent, chatId) - if (sendScope !== sendScopeGeneration.current) return + sendChatId = chatId + if (!sendStillAuthorized()) { + releaseSendSetup() + return + } chatStore.clearCachedRemoteData() autoCreatedMeta = meta appendNewEntry(sendAgent, meta) - sendChatId = chatId // R2 new-chat composer: a per-session model picked BEFORE this chat // existed was held in the agent-keyed pre-chat slot (no chatId to key a // pending entry, no session to POST to). Now that the chatId exists, @@ -2588,24 +2864,35 @@ export function useAgentChatController({ chatStore.setPendingModel(sendAgent, chatId, preChatModel) chatStore.clearPreChatModel(sendAgent) } - activeChatVisibilityRef.current = { - ...activeChatVisibilityRef.current, - activeChatId: chatId, - selectedAgent: sendAgent, + if (wasSendChatVisible) { + activeChatVisibilityRef.current = { + ...activeChatVisibilityRef.current, + activeChatId: chatId, + selectedAgent: sendAgent, + } + setActiveChatId(chatId) + setChatMessages([]) } - setActiveChatId(chatId) - setChatMessages([]) await chatStore.setLastActive(sendAgent, chatId) - if (sendScope !== sendScopeGeneration.current) return + if (!sendStillAuthorized()) { + releaseSendSetup() + return + } } catch (error) { - if (sendScope !== sendScopeGeneration.current) return + if (!sendStillAuthorized()) { + releaseSendSetup() + return + } const message = error instanceof Error ? error.message : String(error) pushToast(`Could not create a chat session: ${message}`, 'error') - agentSendInFlightRef.current = false - setAgentSending(false) + releaseSendSetup() return } } + if (!sendStillAuthorized()) { + releaseSendSetup() + return + } const userMessageId = crypto.randomUUID() const baseContentForRequest = trimmedContent || @@ -2627,15 +2914,34 @@ export function useAgentChatController({ timestamp: Date.now(), ...(displayAttachments.length ? { attachments: displayAttachments } : {}), } + // Persist before the async POST so a Host wake, approval, tab switch, or + // Desktop restart cannot erase the only copy of the user's outgoing turn. + // The acknowledgement below updates this same ID with its task id. + if (sendChatId) { + try { + await chatStore.upsertMessages(sendAgent, sendChatId, [userMessage]) + } catch (error) { + console.warn('[chat-history] failed to persist outgoing message before send', { + agentRef: sendAgent, + chatId: sendChatId, + error, + }) + pushToast('Could not save this message locally. Please retry.', 'error') + releaseSendSetup() + return + } + } // Sidebar freshness (dev): bump this chat's updatedAt/messageCount so the // cross-agent list and the chat list re-sort to the top on send. The user - // message itself is persisted once, later, carrying its task_id (post-D.3), - // so we do NOT eager-persist here (that would double-write without task_id). + // message is already durably staged by the upsert above and is updated in + // place when the send acknowledgement carries its task_id. if (selectedAgent && sendChatId) { const updatedAt = new Date(userMessage.timestamp).toISOString() bumpActivity(selectedAgent, sendChatId, updatedAt) } - setChatMessages(prev => [...prev, userMessage]) + if (sendChatIsVisible()) { + setChatMessages(prev => [...prev, userMessage]) + } // Auto-title on first message if (selectedAgent && sendChatId) { // B10: fall back to the meta just returned by createChat — the closure's @@ -2704,9 +3010,11 @@ export function useAgentChatController({ activeChatVisibilityRef.current.selectedAgent === sendAgent && activeChatVisibilityRef.current.activeChatId === sendChatId && composerAttachmentRevisionRef.current === originalAttachmentRevision && - getComposerDraftRevision(originalDraftChat) === originalDraftRevision + getComposerDraftRevision(originalDraftChat, originalDraftAgent ?? undefined) === + originalDraftRevision ) { clearComposerAfterSend(sendChatId) + clearComposerDraft(null, sendAgent) } setAgentError(null) setFailedAgentSend(null) @@ -2717,7 +3025,7 @@ export function useAgentChatController({ // R2 "Option A": a per-session model chosen while the host was suspended // couldn't be persisted server-side, so it was held as pending. Piggyback // it here — this send wakes the host and applies the model to this task. - if (sendScope !== sendScopeGeneration.current) return + if (!sendStillAuthorized()) return const pendingModel = pendingModelForSend const requestModel = requestModelForRetention const fileReferences = fileReferencesForSend @@ -2748,7 +3056,7 @@ export function useAgentChatController({ // A successful send creates or updates the durable server session. Do not // let the short-lived sidebar cache hide that new catalog state. chatStore.clearCachedRemoteData() - if (sendScope !== sendScopeGeneration.current) return + if (!sendStillAuthorized()) return const responseRecord = response as Record const ackOk = !responseRecord.error && responseRecord.success !== false // #654 H1 — the send IS the write. Its ack carries the revision that @@ -2846,7 +3154,7 @@ export function useAgentChatController({ }), } if (sendChatId) { - await chatStore.appendMessages(sendAgent, sendChatId, [userMessage]) + await chatStore.upsertMessages(sendAgent, sendChatId, [userMessage]) } await appendAssistantMessage(sendAgent, sendChatId, assistantMessage) if (isErrorResponse) { @@ -2908,9 +3216,13 @@ export function useAgentChatController({ // Async task accepted. Persist the user message once, now carrying its // task_id (schema v2), then hand the SSE lifecycle to the tracker. const persistedUserMessage: AgentChatMessage = { ...userMessage, task_id: taskId } - setChatMessages(prev => prev.map(m => (m.id === userMessageId ? persistedUserMessage : m))) + if (sendChatIsVisible()) { + setChatMessages(prev => + prev.map(m => (m.id === userMessageId ? persistedUserMessage : m)) + ) + } if (sendChatId) { - await chatStore.appendMessages(sendAgent, sendChatId, [persistedUserMessage]) + await chatStore.upsertMessages(sendAgent, sendChatId, [persistedUserMessage]) } attachTaskIdToRetainedSend(sendAgent, sendChatId ?? null, userMessageId, taskId) activityTaskToMessageByAgentRef.current[sendAgent] = { @@ -2948,7 +3260,36 @@ export function useAgentChatController({ fsm.dispatch(taskKey, { type: 'TASK_CREATED', taskId }) } } catch (error) { - if (sendScope !== sendScopeGeneration.current) return + if ( + isAuthorizationError(error) && + sendScope === sendScopeGeneration.current && + sendScopeIdentity === currentAuthScopeRef.current && + !isHostAccessBlocked(sendAgent) + ) { + if (isConfirmedHostAccessRevoked(error)) { + revokeHostAccess(sendAgent) + return + } + if (!isHttp403(error)) { + holdHostAccess(sendAgent) + return + } + try { + await chatStore.listSessions(sendAgent, { agent: sendAgent, limit: 1 }, { force: true }) + } catch (readError) { + if ( + isAuthorizationError(readError) && + sendScope === sendScopeGeneration.current && + sendScopeIdentity === currentAuthScopeRef.current && + !isHostAccessBlocked(sendAgent) + ) { + if (isConfirmedHostAccessRevoked(readError)) revokeHostAccess(sendAgent) + else holdHostAccess(sendAgent) + return + } + } + } + if (!sendStillAuthorized()) return const message = error instanceof Error ? error.message : String(error) const normalized = message.toLowerCase() const isRequestEntityTooLarge = @@ -3002,10 +3343,12 @@ export function useAgentChatController({ if (sendChatId) { try { const failedUserMessage: AgentChatMessage = { ...userMessage, preserveLocal: true } - setChatMessages(prev => - prev.map(message => (message.id === userMessageId ? failedUserMessage : message)) - ) - await chatStore.appendMessages(sendAgent, sendChatId, [failedUserMessage]) + if (sendChatIsVisible()) { + setChatMessages(prev => + prev.map(message => (message.id === userMessageId ? failedUserMessage : message)) + ) + } + await chatStore.upsertMessages(sendAgent, sendChatId, [failedUserMessage]) } catch { // persistence is best-effort; the in-memory view still shows it } @@ -3019,14 +3362,15 @@ export function useAgentChatController({ } finally { // Fire & forget: release the synchronous setup guard immediately. The // task (if any) keeps running in the tracker. - if (sendScope === sendScopeGeneration.current) { - agentSendInFlightRef.current = false - setAgentSending(false) - } + releaseSendSetup() } }, [ selectedAgent, + isHostAccessBlocked, + isChatDeleted, + revokeHostAccess, + holdHostAccess, activeChatId, composerImageAttachments, composerReferenceAttachments, @@ -3123,14 +3467,14 @@ export function useAgentChatController({ ) return if ( - getComposerDraft(activeChatId) || + getComposerDraft(activeChatId, selectedAgent ?? undefined) || composerImageAttachments.length || composerReferenceAttachments.length ) { pushToast('Keep or clear the current draft before recovering the earlier input.', 'error') return } - setComposerDraft(activeChatId, visibleFailure.content) + setComposerDraft(activeChatId, visibleFailure.content, selectedAgent ?? undefined) handleAddComposerImageAttachments( visibleFailure.attachments.map(attachment => ({ ...attachment, @@ -3228,6 +3572,7 @@ export function useAgentChatController({ isRemote?: boolean } = {} ) => { + if (isHostAccessBlocked(agentName) || (chatId && isChatDeleted(agentName, chatId))) return if (options.selectLatest) { writePendingSelection(agentName, { mode: 'latest', chatId: null }) return @@ -3259,7 +3604,14 @@ export function useAgentChatController({ upsertProvisionalEntry(chatId, options.title, options.isRemote === true) } }, - [cancelOlderMessagesLoad, currentTeamId, writePendingSelection, upsertProvisionalEntry] + [ + cancelOlderMessagesLoad, + currentTeamId, + writePendingSelection, + upsertProvisionalEntry, + isHostAccessBlocked, + isChatDeleted, + ] ) const clearActiveChat = useCallback(() => { @@ -3344,6 +3696,7 @@ export function useAgentChatController({ handleRenameChat, handleRenameChatForAgent, applyLocalTitleOnly, + captureChatDeleteFence, handleDeleteChat, handleDeleteChatForAgent, handleSelectChat, diff --git a/desktop-app/ui/src/hooks/domain/useChatListController.ts b/desktop-app/ui/src/hooks/domain/useChatListController.ts index 2620728fec..154f4f3f97 100644 --- a/desktop-app/ui/src/hooks/domain/useChatListController.ts +++ b/desktop-app/ui/src/hooks/domain/useChatListController.ts @@ -1,14 +1,47 @@ -import { type MutableRefObject, useCallback, useEffect, useRef, useState } from 'react' +import { type MutableRefObject, useCallback, useEffect, useMemo, useRef, useState } from 'react' import { makeTaskKey } from '@contexts/AgentTaskTrackerContext' +import { byUpdatedDesc, mergeCatalogPage } from '@lib/catalogMerge' import { agentChatPlaceholder, remotePlaceholder } from '@lib/chatTitle' -import { type PendingRename, resolveSessionTitle } from '@lib/resolveSessionTitle' -import type { ChatIndex, ChatMetadata, SessionsListResult } from '../../../../src/types' +import type { PendingRename } from '@lib/resolveSessionTitle' +import type { + ChatAuthorityScope, + ChatDeleteFence, + ChatIndex, + ChatMetadata, + SessionsListResult, +} from '../../../../src/types' +import { + httpErrorStatus, + isAuthorizationError, + isConfirmedHostAccessRevoked, + isHostAvailabilityError, + isHttpServerError, + isInvalidSessionsCursorError, + isNetworkError, +} from '../../lib/format' import { scheduleAfterFirstPaint } from '../scheduleAfterFirstPaint' import type { useChatStore } from '../useChatStore' import { type SessionFsmEvent, type SessionFsmStore, seedSessionSnapshots } from './sessionFsm' const SESSION_CATALOG_PAGE_LIMIT = 50 +async function readSessionCatalog( + chatStore: ReturnType, + agentRef: string, + query: Parameters['listSessions']>[1], + options: { force?: boolean } = {} +): Promise { + try { + return await chatStore.listSessions(agentRef, query, options) + } catch (error) { + // listSessions in the main process already refreshes an expired RPC token + // and retries once on 401. A second uncached read also distinguishes a + // catalog-specific denial from loss of Host authority. + if (!isAuthorizationError(error) || isConfirmedHostAccessRevoked(error)) throw error + return chatStore.listSessions(agentRef, query, { force: true }) + } +} + // R1-M1: an 'offline' pending rename (a genuine network / 5xx failure) retries on // every listSessions poll and every `window 'online'` event. A deterministic 5xx // / 501 would otherwise re-issue the PATCH forever, so cap the auto-retries per @@ -81,6 +114,8 @@ export interface ChatListControllerHost { * rather than taking `pushToast` as a param (its callers don't have it). */ pushToast: (message: string, tone: 'success' | 'error' | 'info') => void + /** The catalog display name for an agent reference, used in user-facing copy. */ + agentDisplayName: (agentRef: string) => string } interface UseChatListControllerParams { @@ -96,10 +131,17 @@ interface UseChatListControllerParams { * team-switch of the same user — see the teardown effect below. */ authUserKey: string + authorityScope: ChatAuthorityScope loadMenuData: boolean chatStore: ReturnType fsm: SessionFsmStore host: MutableRefObject + isHostAccessBlocked: (agentRef: string) => boolean + getHostAuthorityEpoch: (agentRef: string) => number + /** Changes whenever `isHostAccessBlocked` may answer differently. */ + hostAuthorityRevision: number + onHostAccessRevoked: (agentRef: string) => void + onHostAuthorityUncertain: (agentRef: string) => void } function sortableTimestamp(value: string): number { @@ -107,15 +149,34 @@ function sortableTimestamp(value: string): number { return Number.isFinite(timestamp) ? timestamp : 0 } -const byUpdatedDesc = (a: { updatedAt: string }, b: { updatedAt: string }) => - sortableTimestamp(b.updatedAt) - sortableTimestamp(a.updatedAt) - const byLastActivityDesc = (a: { lastActivityAt: string }, b: { lastActivityAt: string }) => sortableTimestamp(b.lastActivityAt) - sortableTimestamp(a.lastActivityAt) -function isRecoverableCatalogCursorError(error: unknown): boolean { - const message = error instanceof Error ? error.message : String(error) - return /\b4\d\d\b/.test(message) || message.toLowerCase().includes('invalid') +/** + * A load-more failure the same cursor can never recover from: any client error + * other than an authorization status (handled as authority), a request timeout + * or a rate limit (both retryable, RFC 9110), plus the main process's own + * status-less cursor refusal. Re-sending the cursor after a 400/404/409/410 + * only repeats the rejection, so the page chain ends here. + */ +function isTerminalCatalogCursorError(error: unknown): boolean { + const status = httpErrorStatus(error) + if (status === undefined) return isInvalidSessionsCursorError(error) + return ( + status >= 400 && + status < 500 && + status !== 401 && + status !== 403 && + status !== 408 && + status !== 429 + ) +} + +function isCatalogOutageError(error: unknown): boolean { + if (isHostAvailabilityError(error)) return false + const status = httpErrorStatus(error) + if (status !== undefined) return isHttpServerError(error) + return isNetworkError(error) } /** @@ -125,8 +186,9 @@ function isRecoverableCatalogCursorError(error: unknown): boolean { * error) is treated as a network failure. * - 'not-found' (404): session not materialized server-side yet → keep pending, * retry once it appears in listSessions. NOT an error; no rollback. - * - 'client-error' (other 4xx: 400 invalid title, 401/403 access): a genuine - * rejection → roll the optimistic title back and toast. + * - 'client-error' (other 4xx, such as 400 invalid title): a genuine rejection + * → roll the optimistic title back and toast. Authorization failures are + * handled separately because a generic 401/403 does not prove revocation. * - 'network' (5xx / no status): transient → queue offline, retry on reconnect. */ function classifyRenameError(error: unknown): 'not-found' | 'client-error' | 'network' { @@ -134,6 +196,7 @@ function classifyRenameError(error: unknown): 'not-found' | 'client-error' | 'ne const match = message.match(/\((\d{3})\)/) const status = match ? Number(match[1]) : null if (status === 404) return 'not-found' + if (status === 401 || status === 403) return 'network' if (status !== null && status >= 400 && status < 500) return 'client-error' return 'network' } @@ -152,16 +215,81 @@ function knownServerMessageCount(session: SessionsListResult['items'][number]): return typeof count === 'number' && Number.isFinite(count) ? Math.max(0, Math.floor(count)) : 0 } +/** A cross-agent catalog session tagged with the agent whose page reported it. */ +type LatestCatalogSession = SessionsListResult['items'][number] & { agentRef: string } + +/** The selected agent's sidebar: entries keyed by chat id, "Chat " placeholder. */ +function mergeSelectedAgentCatalogPage( + cached: SidebarChatEntry[], + sessions: SessionsListResult['items'], + agentRef: string, + pendingRenameStateFor: (agentRef: string, chatId: string) => PendingRename +): SidebarChatEntry[] { + return mergeCatalogPage({ + cached, + sessions, + entryKey: entry => entry.id, + sessionKey: session => session.chatId, + entryPendingRename: entry => pendingRenameStateFor(agentRef, entry.id), + sessionPendingRename: session => pendingRenameStateFor(agentRef, session.chatId), + placeholderFor: agentChatPlaceholder, + serverOnlyEntry: (session, title) => ({ + id: session.chatId, + title, + createdAt: session.lastActivityAt, + updatedAt: session.lastActivityAt, + // Older hosts omit messageCount. Keep that unknown value at zero instead + // of fabricating two messages per turn and overstating Activity totals + // when tool/system messages vary by session. + messageCount: knownServerMessageCount(session), + remote: true, + }), + }) +} + +function sameAuthorityScope(left: ChatAuthorityScope, right: ChatAuthorityScope): boolean { + return ( + left.environmentKey === right.environmentKey && + left.userId === right.userId && + left.teamId === right.teamId + ) +} + +/** + * The identity a local deletion is bound to, identical to main's + * `sameChatDeletionIdentity`: environment + user. The team is not part of it, + * so a confirmed deletion keeps holding after a team switch. + */ +function sameChatDeletionIdentity(left: ChatAuthorityScope, right: ChatAuthorityScope): boolean { + return left.environmentKey === right.environmentKey && left.userId === right.userId +} + +export function deletedChatIdsForScope(index: ChatIndex, scope: ChatAuthorityScope): string[] { + return [ + ...new Set( + (index.deletedChatTombstones ?? []) + .filter(tombstone => sameChatDeletionIdentity(tombstone.authorityScope, scope)) + .map(tombstone => tombstone.chatId) + ), + ] +} + export function useChatListController({ selectedAgent, agentNames, isAuthenticated, scopeKey, authUserKey, + authorityScope, loadMenuData, chatStore, fsm, host, + isHostAccessBlocked, + getHostAuthorityEpoch, + hostAuthorityRevision, + onHostAccessRevoked, + onHostAuthorityUncertain, }: UseChatListControllerParams) { // Per-agent list for the SELECTED agent (the sidebar's chat list). const [chatList, setChatList] = useState([]) @@ -171,6 +299,11 @@ export function useChatListController({ // Cross-agent "Latest sessions" list (badges live in the FSM, seeded below). const [latestChatSessions, setLatestChatSessions] = useState([]) const [latestChatSessionsLoading, setLatestChatSessionsLoading] = useState(false) + const catalogOfflineByAgentRef = useRef(new Set()) + const catalogOfflineToastShownRef = useRef(false) + const catalogOfflineToastQueuedRef = useRef(false) + const catalogOfflineToastTimerRef = useRef | null>(null) + const [latestCatalogMutationRevision, setLatestCatalogMutationRevision] = useState(0) // Selection requested for an agent before its chats have loaded, consumed by // the parent's agent-selection effect. Ref (not state): imperative, per-agent. const pendingChatSelectionByAgentRef = useRef>({}) @@ -178,7 +311,14 @@ export function useChatListController({ const suppressAutoSelectionSequenceRef = useRef(0) const chatListNextCursorByAgentRef = useRef>({}) const chatListLoadingMoreByAgentRef = useRef>(new Set()) + const deletedChatIdsByAgentRef = useRef(new Map>()) const requestGenerationRef = useRef(0) + const authorityScopeGenerationRef = useRef(0) + const agentNamesRef = useRef(agentNames) + agentNamesRef.current = agentNames + const agentNamesKey = agentNames.join('\n') + const currentAuthorityScopeRef = useRef(`${isAuthenticated}:${scopeKey}`) + currentAuthorityScopeRef.current = `${isAuthenticated}:${scopeKey}` // Spec 15 §2.5 (B21) — pending-rename queue. A user rename is optimistic-local // first, then synced by RPC; while unconfirmed the local title wins over the @@ -217,6 +357,11 @@ export function useChatListController({ >(new Map()) const pendingRenameKey = (agentRef: string, chatId: string): string => `${agentRef}:${chatId}` + const authUserKeyRef = useRef(authUserKey) + + useEffect(() => { + authUserKeyRef.current = authUserKey + }, [authUserKey]) const pendingRenameStateFor = useCallback((agentRef: string, chatId: string): PendingRename => { return pendingRenamesRef.current.get(pendingRenameKey(agentRef, chatId))?.state ?? 'none' @@ -229,6 +374,19 @@ export function useChatListController({ useEffect(() => { requestGenerationRef.current += 1 + authorityScopeGenerationRef.current += 1 + deletedChatIdsByAgentRef.current.clear() + catalogOfflineByAgentRef.current.clear() + catalogOfflineToastShownRef.current = false + // A toast queued for this scope must never flush into the next scope or + // after unmount: the cleanup runs on both. + return () => { + if (catalogOfflineToastTimerRef.current !== null) { + clearTimeout(catalogOfflineToastTimerRef.current) + catalogOfflineToastTimerRef.current = null + } + catalogOfflineToastQueuedRef.current = false + } }, [isAuthenticated, scopeKey]) // R1-H1 — the pending-rename queue is per-USER identity. mcp-host keys a chat @@ -248,6 +406,35 @@ export function useChatListController({ pendingRenamesRef.current.clear() }, [authUserKey]) + const reportCatalogOffline = (agentRef: string) => { + if (!catalogOfflineByAgentRef.current.has(agentRef)) { + catalogOfflineByAgentRef.current.add(agentRef) + } + if (catalogOfflineToastShownRef.current || catalogOfflineToastQueuedRef.current) return + catalogOfflineToastQueuedRef.current = true + const authorityScopeGeneration = authorityScopeGenerationRef.current + catalogOfflineToastTimerRef.current = setTimeout(() => { + catalogOfflineToastTimerRef.current = null + catalogOfflineToastQueuedRef.current = false + if (authorityScopeGenerationRef.current !== authorityScopeGeneration) return + const affectedAgents = [...catalogOfflineByAgentRef.current] + const currentHost = host.current + if (!currentHost || !affectedAgents.length) return + catalogOfflineToastShownRef.current = true + const names = affectedAgents.map(ref => currentHost.agentDisplayName(ref)).join(', ') + currentHost.pushToast(`Chat list for ${names} is offline. Showing saved chats.`, 'info') + }, 0) + } + + /** A success from an earlier authority scope says nothing about this one. */ + const reportCatalogOnline = (agentRef: string, authorityScopeGeneration: number) => { + if (authorityScopeGenerationRef.current !== authorityScopeGeneration) return + catalogOfflineByAgentRef.current.delete(agentRef) + if (catalogOfflineByAgentRef.current.size === 0) { + catalogOfflineToastShownRef.current = false + } + } + // Live `selectedAgent` for the stable callbacks below (they gate a chatList // write on "is this the selected agent"). A ref keeps the callbacks stable // while always reading the committed value at call time. @@ -269,14 +456,27 @@ export function useChatListController({ // ─── Cross-agent latest-sessions mutators ─── - const upsertLatestChatSession = useCallback((agentRef: string, chat: SidebarChatEntry) => { - setLatestChatSessions(previous => { - const next = [ - { ...chat, agentRef }, - ...previous.filter(item => item.agentRef !== agentRef || item.id !== chat.id), - ] - return next.sort(byUpdatedDesc) - }) + const upsertLatestChatSession = useCallback( + (agentRef: string, chat: SidebarChatEntry) => { + if ( + isHostAccessBlocked(agentRef) || + deletedChatIdsByAgentRef.current.get(agentRef)?.has(chat.id) + ) + return + setLatestChatSessions(previous => { + if (deletedChatIdsByAgentRef.current.get(agentRef)?.has(chat.id)) return previous + const next = [ + { ...chat, agentRef }, + ...previous.filter(item => item.agentRef !== agentRef || item.id !== chat.id), + ] + return next.sort(byUpdatedDesc) + }) + }, + [isHostAccessBlocked] + ) + + const hideAgent = useCallback((agentRef: string) => { + setLatestChatSessions(previous => previous.filter(chat => chat.agentRef !== agentRef)) }, []) const removeLatestChatSession = useCallback((agentRef: string, chatId: string) => { @@ -285,6 +485,15 @@ export function useChatListController({ ) }, []) + // A catalog request can resolve with a pre-delete snapshot just as its owning + // component commit lands. Tombstones are imperative truth, so reproject them + // onto the published list after every confirmed destructive mutation. + useEffect(() => { + setLatestChatSessions(previous => + previous.filter(item => !deletedChatIdsByAgentRef.current.get(item.agentRef)?.has(item.id)) + ) + }, [latestCatalogMutationRevision]) + // ─── chatList loader (agent-scoped) ─── const loadChatListOnce = useCallback( @@ -292,29 +501,67 @@ export function useChatListController({ agentRef: string, requestGeneration: number ): Promise<{ index: ChatIndex; merged: SidebarChatEntry[] }> => { + const authorityScopeGeneration = authorityScopeGenerationRef.current + const authorityScopeAtRequest = currentAuthorityScopeRef.current + const hostAuthorityEpoch = getHostAuthorityEpoch(agentRef) chatListNextCursorByAgentRef.current[agentRef] = null if (selectedAgentRef.current === agentRef) { setChatListHasMoreRemoteSessions(false) setChatListMoreLoading(chatListLoadingMoreByAgentRef.current.has(agentRef)) } const index = await chatStore.getIndex(agentRef) - const merged = [...index.chats].sort(byUpdatedDesc) + if ( + getHostAuthorityEpoch(agentRef) !== hostAuthorityEpoch || + currentAuthorityScopeRef.current !== authorityScopeAtRequest || + isHostAccessBlocked(agentRef) + ) { + return { index, merged: [] } + } if ( selectedAgentRef.current !== agentRef || requestGenerationRef.current !== requestGeneration ) { - return { index, merged } + return { index, merged: [...index.chats].sort(byUpdatedDesc) } } + const deleted = deletedChatIdsByAgentRef.current.get(agentRef) ?? new Set() + for (const chatId of deletedChatIdsForScope(index, authorityScope)) deleted.add(chatId) + deletedChatIdsByAgentRef.current.set(agentRef, deleted) + const merged = index.chats.filter(chat => !deleted.has(chat.id)).sort(byUpdatedDesc) setChatList(merged) const suppressionMarkerAtRequest = suppressAutoSelectionByAgentRef.current.get(agentRef) scheduleAfterFirstPaint(async () => { - const serverResult = await chatStore - .listSessions(agentRef, { agent: agentRef, limit: SESSION_CATALOG_PAGE_LIMIT }) - .catch((): SessionsListResult => ({ items: [] })) + let serverResult: SessionsListResult + try { + serverResult = await readSessionCatalog(chatStore, agentRef, { + agent: agentRef, + limit: SESSION_CATALOG_PAGE_LIMIT, + }) + } catch (error) { + if ( + authorityScopeGenerationRef.current === authorityScopeGeneration && + getHostAuthorityEpoch(agentRef) === hostAuthorityEpoch && + currentAuthorityScopeRef.current === authorityScopeAtRequest && + !isHostAccessBlocked(agentRef) + ) { + if (isConfirmedHostAccessRevoked(error)) onHostAccessRevoked(agentRef) + else if (isAuthorizationError(error)) onHostAuthorityUncertain(agentRef) + if ( + !isConfirmedHostAccessRevoked(error) && + !isAuthorizationError(error) && + isCatalogOutageError(error) + ) + reportCatalogOffline(agentRef) + } + return + } + reportCatalogOnline(agentRef, authorityScopeGeneration) if ( selectedAgentRef.current !== agentRef || - requestGenerationRef.current !== requestGeneration + requestGenerationRef.current !== requestGeneration || + getHostAuthorityEpoch(agentRef) !== hostAuthorityEpoch || + currentAuthorityScopeRef.current !== authorityScopeAtRequest || + isHostAccessBlocked(agentRef) ) { if ( suppressionMarkerAtRequest !== undefined && @@ -329,7 +576,10 @@ export function useChatListController({ // older hosts and malformed proxy responses so another agent's catalog // entry can never leak into the selected agent's sidebar. const serverSessions = serverResult.items - .filter(s => s.agent === agentRef) + .filter( + s => + s.agent === agentRef && !deletedChatIdsByAgentRef.current.get(agentRef)?.has(s.chatId) + ) .sort(byLastActivityDesc) chatListNextCursorByAgentRef.current[agentRef] = serverResult.nextCursor ?? null setChatListHasMoreRemoteSessions(Boolean(serverResult.nextCursor)) @@ -344,45 +594,20 @@ export function useChatListController({ // "Remote ·" label, no isRemote branch; switchToChat's unified path // hydrates them. setChatList(previous => { - const dedupedPrevious = dedupeSidebarChats(previous) - const serverById = new Map(serverSessions.map(s => [s.chatId, s])) - const knownIds = new Set(dedupedPrevious.map(c => c.id)) - // Cases C/D/E/F (§2.2): the server is authoritative for a cached chat's - // title when it reports one AND no local rename is pending; a pending - // rename (in-flight / offline) keeps the local title until it lands. - const reconciled = dedupedPrevious.map(chat => { - const server = serverById.get(chat.id) - if (!server) return chat - const { title } = resolveSessionTitle({ - inCache: true, - localTitle: chat.title, - serverTitle: server.title, - pendingRename: pendingRenameStateFor(agentRef, chat.id), - placeholder: agentChatPlaceholder(chat.id), - }) - return title === chat.title ? chat : { ...chat, title } - }) - // Cases A/B (§2.2): a server-only chat shows the server title when the - // host reports one, else the "Chat " placeholder. - const fromServerOnly: SidebarChatEntry[] = serverSessions - .filter(s => !knownIds.has(s.chatId)) - .map(s => ({ - id: s.chatId, - title: resolveSessionTitle({ - inCache: false, - serverTitle: s.title, - pendingRename: pendingRenameStateFor(agentRef, s.chatId), - placeholder: agentChatPlaceholder(s.chatId), - }).title, - createdAt: s.lastActivityAt, - updatedAt: s.lastActivityAt, - // Older hosts omit messageCount. Keep that unknown value at zero - // instead of fabricating two messages per turn and overstating - // Activity totals when tool/system messages vary by session. - messageCount: knownServerMessageCount(s), - remote: true, - })) - return [...reconciled, ...fromServerOnly].sort(byUpdatedDesc) + if (isHostAccessBlocked(agentRef)) return [] + const visibleSessions = serverSessions.filter( + session => !deletedChatIdsByAgentRef.current.get(agentRef)?.has(session.chatId) + ) + const dedupedPrevious = dedupeSidebarChats(previous).filter( + chat => !deletedChatIdsByAgentRef.current.get(agentRef)?.has(chat.id) + ) + // §2.2 precedence (cases A-F) lives in `mergeCatalogPage`. + return mergeSelectedAgentCatalogPage( + dedupedPrevious, + visibleSessions, + agentRef, + pendingRenameStateFor + ) }) // §2.5: a rename that 404'd (session not yet server-side) retries now that @@ -434,7 +659,16 @@ export function useChatListController({ return { index, merged } }, - [chatStore.getIndex, chatStore.listSessions, chatStore.reconcileServerSessions, fsm] + [ + authorityScope, + chatStore.getIndex, + chatStore.listSessions, + chatStore.reconcileServerSessions, + fsm, + getHostAuthorityEpoch, + isHostAccessBlocked, + onHostAccessRevoked, + ] ) const loadMoreChatSessions = useCallback(async () => { @@ -450,21 +684,55 @@ export function useChatListController({ chatListLoadingMoreByAgentRef.current.add(agentRef) const requestGeneration = requestGenerationRef.current + const authorityScopeGeneration = authorityScopeGenerationRef.current + const authorityScopeAtRequest = currentAuthorityScopeRef.current + const hostAuthorityEpoch = getHostAuthorityEpoch(agentRef) setChatListMoreLoading(true) try { let serverResult: SessionsListResult try { - serverResult = await chatStore.listSessions( + serverResult = await readSessionCatalog( + chatStore, agentRef, { agent: agentRef, limit: SESSION_CATALOG_PAGE_LIMIT, cursor }, { force: true } ) } catch (error) { + // The sidebar's "Load more" state belongs to whichever agent is selected + // now; a late rejection for another agent may only touch its own cursor. + const stillShowingRequest = () => + selectedAgentRef.current === agentRef && + requestGenerationRef.current === requestGeneration + if ( + authorityScopeGenerationRef.current === authorityScopeGeneration && + getHostAuthorityEpoch(agentRef) === hostAuthorityEpoch && + currentAuthorityScopeRef.current === authorityScopeAtRequest && + !isHostAccessBlocked(agentRef) + ) { + if (isConfirmedHostAccessRevoked(error)) onHostAccessRevoked(agentRef) + else if (isAuthorizationError(error)) onHostAuthorityUncertain(agentRef) + else if (isTerminalCatalogCursorError(error)) { + chatListNextCursorByAgentRef.current[agentRef] = null + if (stillShowingRequest()) setChatListHasMoreRemoteSessions(false) + } else if (stillShowingRequest() && host.current) { + // 429, 5xx, a waking Host or a transport failure: the cursor stays + // valid and the user retries with the same button. No automatic + // retry is scheduled. + const currentHost = host.current + currentHost.pushToast( + `Couldn't load more chats for ${currentHost.agentDisplayName(agentRef)}. Try again shortly.`, + 'info' + ) + } + return + } if ( requestGenerationRef.current === requestGeneration && + getHostAuthorityEpoch(agentRef) === hostAuthorityEpoch && + currentAuthorityScopeRef.current === authorityScopeAtRequest && selectedAgentRef.current === agentRef ) { - if (isRecoverableCatalogCursorError(error)) { + if (isTerminalCatalogCursorError(error)) { chatListNextCursorByAgentRef.current[agentRef] = null setChatListHasMoreRemoteSessions(false) } else { @@ -473,49 +741,45 @@ export function useChatListController({ } return } - if (requestGenerationRef.current !== requestGeneration) return + reportCatalogOnline(agentRef, authorityScopeGeneration) + if ( + requestGenerationRef.current !== requestGeneration || + getHostAuthorityEpoch(agentRef) !== hostAuthorityEpoch || + currentAuthorityScopeRef.current !== authorityScopeAtRequest + ) { + return + } - const serverSessions = serverResult.items.filter(s => s.agent === agentRef) - if (selectedAgentRef.current !== agentRef) return + const serverSessions = serverResult.items.filter( + s => s.agent === agentRef && !deletedChatIdsByAgentRef.current.get(agentRef)?.has(s.chatId) + ) + if ( + selectedAgentRef.current !== agentRef || + isHostAccessBlocked(agentRef) || + getHostAuthorityEpoch(agentRef) !== hostAuthorityEpoch || + currentAuthorityScopeRef.current !== authorityScopeAtRequest + ) { + return + } chatListNextCursorByAgentRef.current[agentRef] = serverResult.nextCursor ?? null setChatListHasMoreRemoteSessions(Boolean(serverResult.nextCursor)) seedSessionSnapshots(fsm, agentRef, serverSessions) setChatList(previous => { - const dedupedPrevious = dedupeSidebarChats(previous) - const serverById = new Map(serverSessions.map(s => [s.chatId, s])) - const knownIds = new Set(dedupedPrevious.map(c => c.id)) - // Cases C/D/E/F (§2.2): reconcile the title of any cached chat this page - // also reports; server wins when it has a title and no rename is pending. - const reconciled = dedupedPrevious.map(chat => { - const server = serverById.get(chat.id) - if (!server) return chat - const { title } = resolveSessionTitle({ - inCache: true, - localTitle: chat.title, - serverTitle: server.title, - pendingRename: pendingRenameStateFor(agentRef, chat.id), - placeholder: agentChatPlaceholder(chat.id), - }) - return title === chat.title ? chat : { ...chat, title } - }) - // Cases A/B (§2.2): server-only page entries. - const fromServerOnly: SidebarChatEntry[] = serverSessions - .filter(s => !knownIds.has(s.chatId)) - .map(s => ({ - id: s.chatId, - title: resolveSessionTitle({ - inCache: false, - serverTitle: s.title, - pendingRename: pendingRenameStateFor(agentRef, s.chatId), - placeholder: agentChatPlaceholder(s.chatId), - }).title, - createdAt: s.lastActivityAt, - updatedAt: s.lastActivityAt, - messageCount: knownServerMessageCount(s), - remote: true, - })) - return [...reconciled, ...fromServerOnly].sort(byUpdatedDesc) + if (isHostAccessBlocked(agentRef)) return [] + const visibleSessions = serverSessions.filter( + session => !deletedChatIdsByAgentRef.current.get(agentRef)?.has(session.chatId) + ) + const dedupedPrevious = dedupeSidebarChats(previous).filter( + chat => !deletedChatIdsByAgentRef.current.get(agentRef)?.has(chat.id) + ) + // §2.2 precedence (cases A-F) lives in `mergeCatalogPage`. + return mergeSelectedAgentCatalogPage( + dedupedPrevious, + visibleSessions, + agentRef, + pendingRenameStateFor + ) }) try { @@ -534,7 +798,14 @@ export function useChatListController({ setChatListMoreLoading(false) } } - }, [chatStore.listSessions, chatStore.reconcileServerSessions, fsm]) + }, [ + chatStore.listSessions, + chatStore.reconcileServerSessions, + fsm, + getHostAuthorityEpoch, + isHostAccessBlocked, + onHostAccessRevoked, + ]) const loadChatList = useCallback( async (agentRef: string): Promise<{ index: ChatIndex; merged: SidebarChatEntry[] } | null> => { @@ -642,10 +913,14 @@ export function useChatListController({ * just refresh the title on the existing entry. */ const upsertHydratedEntry = useCallback((meta: ChatMetadata, title: string) => { + const agentRef = selectedAgentRef.current + if (agentRef && deletedChatIdsByAgentRef.current.get(agentRef)?.has(meta.id)) return setChatList(prev => - prev.some(c => c.id === meta.id) - ? prev.map(c => (c.id === meta.id ? { ...c, ...meta, title, remote: false } : c)) - : [...prev, { ...meta, title }] + agentRef && deletedChatIdsByAgentRef.current.get(agentRef)?.has(meta.id) + ? prev + : prev.some(c => c.id === meta.id) + ? prev.map(c => (c.id === meta.id ? { ...c, ...meta, title, remote: false } : c)) + : [...prev, { ...meta, title }] ) }, []) @@ -679,7 +954,9 @@ export function useChatListController({ /** Append a freshly-created chat to both lists (create / send auto-create). */ const appendNewEntry = useCallback( (agentRef: string, meta: ChatMetadata) => { + if (deletedChatIdsByAgentRef.current.get(agentRef)?.has(meta.id)) return setChatList(prev => { + if (deletedChatIdsByAgentRef.current.get(agentRef)?.has(meta.id)) return prev const next = prev.some(chat => chat.id === meta.id) ? prev : [...prev, meta] return dedupeSidebarChats(next) }) @@ -713,6 +990,11 @@ export function useChatListController({ const clearPendingSelection = useCallback((agentName: string) => { delete pendingChatSelectionByAgentRef.current[agentName] }, []) + const isChatDeleted = useCallback( + (agentRef: string, chatId: string) => + deletedChatIdsByAgentRef.current.get(agentRef)?.has(chatId) ?? false, + [] + ) // ─── Chat CRUD ─── @@ -744,10 +1026,17 @@ export function useChatListController({ */ const applyLocalTitleOnly = useCallback( async (agentRef: string, chatId: string, newTitle: string) => { - if (!agentRef) return + if (!agentRef || !isAuthenticated) return const requestGeneration = requestGenerationRef.current + const authorityScopeGeneration = authorityScopeGenerationRef.current + const authorityScopeAtDelete = currentAuthorityScopeRef.current + const stillOwned = () => + authorityScopeGenerationRef.current === authorityScopeGeneration && + currentAuthorityScopeRef.current === authorityScopeAtDelete + const bindingGeneration = await chatStore.getBindingGeneration() + if (!stillOwned()) return const updatedAt = new Date().toISOString() - await chatStore.renameChat(agentRef, chatId, newTitle) + await chatStore.renameChat(agentRef, chatId, newTitle, bindingGeneration) if (requestGenerationRef.current !== requestGeneration) return setLatestChatSessions(prev => prev @@ -795,6 +1084,11 @@ export function useChatListController({ if (entry.sending) return if (pendingRenamesRef.current.get(key) !== entry) return entry.sending = true + const authorityScopeGeneration = authorityScopeGenerationRef.current + const hostAuthorityEpoch = getHostAuthorityEpoch(entry.agentRef) + const requestStillCurrent = () => + authorityScopeGenerationRef.current === authorityScopeGeneration && + getHostAuthorityEpoch(entry.agentRef) === hostAuthorityEpoch try { await chatStore.renameSession(entry.agentRef, entry.chatId, entry.title) // Only clear if this attempt still owns the key (a stale 200 must never @@ -803,6 +1097,73 @@ export function useChatListController({ pendingRenamesRef.current.delete(key) } } catch (error) { + if ( + isConfirmedHostAccessRevoked(error) && + requestStillCurrent() && + !isHostAccessBlocked(entry.agentRef) + ) { + onHostAccessRevoked(entry.agentRef) + if (pendingRenamesRef.current.get(key) === entry) { + pendingRenamesRef.current.delete(key) + } + return + } + if (isAuthorizationError(error)) { + if ( + !requestStillCurrent() || + isHostAccessBlocked(entry.agentRef) || + pendingRenamesRef.current.get(key) !== entry + ) { + return + } + let readSucceeded = false + try { + await chatStore.listSessions( + entry.agentRef, + { agent: entry.agentRef, limit: 1 }, + { force: true } + ) + readSucceeded = true + } catch (readError) { + if (!requestStillCurrent() || isHostAccessBlocked(entry.agentRef)) return + if (isConfirmedHostAccessRevoked(readError)) { + onHostAccessRevoked(entry.agentRef) + if (pendingRenamesRef.current.get(key) === entry) { + pendingRenamesRef.current.delete(key) + } + return + } + } + if ( + !requestStillCurrent() || + isHostAccessBlocked(entry.agentRef) || + pendingRenamesRef.current.get(key) !== entry + ) { + return + } + if (readSucceeded) { + if (pendingRenamesRef.current.get(key) === entry) { + pendingRenamesRef.current.delete(key) + if (entry.previousTitle) { + await applyLocalTitleOnly(entry.agentRef, entry.chatId, entry.previousTitle) + } + } + host.current?.pushToast('You do not have permission to rename this chat.', 'error') + return + } + // Only the structured host denial proves revocation. A generic 403 + // may be a stale/missing write capability. If the read check also + // fails, preserve the existing uncertain-authority handling. + onHostAuthorityUncertain(entry.agentRef) + if (pendingRenamesRef.current.get(key) === entry) { + entry.state = 'offline' + entry.failureCount += 1 + if (entry.failureCount >= MAX_RENAME_SYNC_ATTEMPTS) { + entry.retriesExhausted = true + } + } + return + } // A newer rename superseded this one: leave the key entirely to it — no // rollback, no re-mark (FIX 2). if (pendingRenamesRef.current.get(key) !== entry) return @@ -844,7 +1205,15 @@ export function useChatListController({ entry.sending = false } }, - [chatStore, applyLocalTitleOnly, host] + [ + chatStore, + applyLocalTitleOnly, + getHostAuthorityEpoch, + host, + isHostAccessBlocked, + onHostAccessRevoked, + onHostAuthorityUncertain, + ] ) /** @@ -861,6 +1230,7 @@ export function useChatListController({ // FIX 1: skip a key with a PATCH already in flight (attemptRenameRpc // guards this too, but skipping avoids spawning a no-op). if (entry.sending) continue + if (isHostAccessBlocked(entry.agentRef)) continue // R1-M1: skip an entry whose retry budget is spent — its optimistic title // stays, but it no longer PATCHes on polls/online. Only 'offline' entries // can ever reach this state (in-flight never counts toward the budget). @@ -879,7 +1249,7 @@ export function useChatListController({ await attemptRenameRpc(entry) } }, - [attemptRenameRpc] + [attemptRenameRpc, isHostAccessBlocked] ) /** @@ -890,6 +1260,7 @@ export function useChatListController({ const handleRenameChatForAgent = useCallback( async (agentRef: string, chatId: string, newTitle: string) => { if (!agentRef) return + const authUserKeyAtRequest = authUserKeyRef.current const key = pendingRenameKey(agentRef, chatId) // FIX 2: the rollback target is the title BEFORE the pending chain began. // If a rename is already pending for this key, preserve its `previousTitle` @@ -901,9 +1272,11 @@ export function useChatListController({ previousTitle = existing.previousTitle } else { const index = await chatStore.getIndex(agentRef) + if (authUserKeyRef.current !== authUserKeyAtRequest) return previousTitle = index.chats.find(c => c.id === chatId)?.title ?? '' } await applyLocalTitleOnly(agentRef, chatId, newTitle) + if (authUserKeyRef.current !== authUserKeyAtRequest) return // Replace any prior entry: this is now the current rename for the key. A // still-in-flight older attempt will no-op on completion (identity guard). const entry = { @@ -919,7 +1292,7 @@ export function useChatListController({ pendingRenamesRef.current.set(key, entry) await attemptRenameRpc(entry) }, - [chatStore, applyLocalTitleOnly, attemptRenameRpc] + [authUserKey, chatStore, applyLocalTitleOnly, attemptRenameRpc] ) const handleRenameChat = useCallback( @@ -947,10 +1320,57 @@ export function useChatListController({ } }, [flushPendingRenames]) + const captureChatDeleteFence = useCallback( + async ( + agentRef: string + ): Promise<{ agentRef: string; fence: ChatDeleteFence; hostAuthorityEpoch: number }> => { + if (!agentRef || isHostAccessBlocked(agentRef)) { + throw new Error('Chat deletion is unavailable while host access is blocked') + } + const hostAuthorityEpoch = getHostAuthorityEpoch(agentRef) + try { + const fence = await chatStore.captureDeleteFence(authorityScope) + if ( + !sameAuthorityScope(fence.authorityScope, authorityScope) || + hostAuthorityEpoch !== getHostAuthorityEpoch(agentRef) || + isHostAccessBlocked(agentRef) + ) { + throw new Error('Chat deletion authority changed before confirmation') + } + return { agentRef, fence, hostAuthorityEpoch } + } catch (error) { + host.current?.pushToast('Could not confirm chat access. Please try again.', 'error') + throw error + } + }, + [authorityScope, chatStore.captureDeleteFence, getHostAuthorityEpoch, host, isHostAccessBlocked] + ) + const handleDeleteChatForAgent = useCallback( - async (agentRef: string, chatId: string) => { + async ( + agentRef: string, + chatId: string, + deletion: { agentRef: string; fence: ChatDeleteFence; hostAuthorityEpoch: number } + ) => { if (!agentRef) return + if ( + deletion.agentRef !== agentRef || + !sameAuthorityScope(deletion.fence.authorityScope, authorityScope) || + deletion.hostAuthorityEpoch !== getHostAuthorityEpoch(agentRef) || + isHostAccessBlocked(agentRef) + ) { + host.current?.pushToast( + 'Delete cancelled because chat access changed. Please try again.', + 'error' + ) + return + } const requestGeneration = requestGenerationRef.current + const authorityScopeGeneration = authorityScopeGenerationRef.current + const authorityScopeAtDelete = currentAuthorityScopeRef.current + const stillOwned = () => + authorityScopeGenerationRef.current === authorityScopeGeneration && + currentAuthorityScopeRef.current === authorityScopeAtDelete // Stop following any in-flight task for this chat first: ack tears down the // SSE + connect/watchdog timers, so a later terminal can't fire onTerminal // and resurrect the just-deleted chat file via appendAssistantMessage. @@ -960,11 +1380,28 @@ export function useChatListController({ // fire onTerminal and resurrect the just-deleted chat file. Dispatched // FIRST (before the delete), preserving the ack-before-delete ordering. host.current?.dispatchSession(deletedKey, { type: 'CHAT_DELETED' }) - await chatStore.deleteChat(agentRef, chatId) + let result: Awaited> + try { + result = await chatStore.deleteChat(agentRef, chatId, deletion.fence) + } catch { + host.current?.pushToast('Could not delete the chat. Please try again.', 'error') + return + } + if (!stillOwned()) return + const deleted = deletedChatIdsByAgentRef.current.get(agentRef) ?? new Set() + deleted.add(chatId) + deletedChatIdsByAgentRef.current.set(agentRef, deleted) + setLatestCatalogMutationRevision(value => value + 1) chatStore.clearCachedRemoteData() - if (requestGenerationRef.current !== requestGeneration) return host.current?.clearComposerDraft(chatId) removeLatestChatSession(agentRef, chatId) + if (result.cleanupPending) { + host.current?.pushToast( + 'Chat removed. Local transcript cleanup will retry when this team is active.', + 'info' + ) + } + if (requestGenerationRef.current !== requestGeneration) return // Post-await guards read the LIVE committed values (selectedAgentRef / // getActiveChatId), intentionally — if the user switched agent during the // delete IPC we must not yank a reselection into the agent they just left. @@ -986,121 +1423,231 @@ export function useChatListController({ } } }, - [chatList, chatStore, removeLatestChatSession, handleCreateChat, host] + [ + authorityScope, + chatList, + chatStore, + getHostAuthorityEpoch, + isHostAccessBlocked, + removeLatestChatSession, + handleCreateChat, + host, + isAuthenticated, + ] ) const handleDeleteChat = useCallback( - async (chatId: string) => { - const agentRef = selectedAgentRef.current - if (!agentRef) return - await handleDeleteChatForAgent(agentRef, chatId) + async ( + chatId: string, + deletion: { agentRef: string; fence: ChatDeleteFence; hostAuthorityEpoch: number } + ) => { + await handleDeleteChatForAgent(deletion.agentRef, chatId, deletion) }, [handleDeleteChatForAgent] ) + // A hold is enforced synchronously (the published list below filters held + // Hosts, and every in-flight read compares its Host's epoch), so holding a + // Host needs no reload. Releasing one does: its sessions were dropped while + // it was held, so the cross-agent list reloads when a held Host comes back. + const [latestReloadRevision, setLatestReloadRevision] = useState(0) + const heldAgentsRef = useRef(new Set()) + useEffect(() => { + const held = new Set(agentNamesRef.current.filter(agentRef => isHostAccessBlocked(agentRef))) + const released = [...heldAgentsRef.current].some( + agentRef => !held.has(agentRef) && agentNamesRef.current.includes(agentRef) + ) + heldAgentsRef.current = held + if (released) setLatestReloadRevision(revision => revision + 1) + }, [agentNamesKey, hostAuthorityRevision, isHostAccessBlocked]) + // ─── Cross-agent latest-sessions loader (seeds badges via SERVER_SNAPSHOT) ─── useEffect(() => { - if (!isAuthenticated || !loadMenuData || !agentNames.length) { + const requestedAgentNames = agentNamesRef.current + if (!isAuthenticated || !loadMenuData || !requestedAgentNames.length) { setLatestChatSessions([]) setLatestChatSessionsLoading(false) return } let cancelled = false + const authorityScopeGeneration = authorityScopeGenerationRef.current + const authorityScopeAtRequest = currentAuthorityScopeRef.current + const hostAuthorityEpochByAgent = new Map( + requestedAgentNames.map(agentRef => [agentRef, getHostAuthorityEpoch(agentRef)]) + ) setLatestChatSessionsLoading(true) ;(async () => { try { const localGroups = await Promise.all( - agentNames.map(async agentRef => { + requestedAgentNames.map(async agentRef => { try { const index = await chatStore.getIndex(agentRef) + if ( + currentAuthorityScopeRef.current !== authorityScopeAtRequest || + getHostAuthorityEpoch(agentRef) !== hostAuthorityEpochByAgent.get(agentRef) || + isHostAccessBlocked(agentRef) + ) { + return { + agentRef, + entries: [] as LatestSidebarChatEntry[], + deletedChatIds: [] as string[], + } + } return { agentRef, entries: index.chats.map(chat => ({ ...chat, agentRef, })), + deletedChatIds: deletedChatIdsForScope(index, authorityScope), } } catch { return { agentRef, entries: [] as LatestSidebarChatEntry[], + deletedChatIds: [] as string[], } } }) ) - if (cancelled) return - setLatestChatSessions(localGroups.flatMap(group => group.entries).sort(byUpdatedDesc)) + if (cancelled || currentAuthorityScopeRef.current !== authorityScopeAtRequest) return + for (const group of localGroups) { + if ( + getHostAuthorityEpoch(group.agentRef) !== + hostAuthorityEpochByAgent.get(group.agentRef) || + isHostAccessBlocked(group.agentRef) + ) { + continue + } + const deleted = deletedChatIdsByAgentRef.current.get(group.agentRef) ?? new Set() + for (const chatId of group.deletedChatIds) deleted.add(chatId) + deletedChatIdsByAgentRef.current.set(group.agentRef, deleted) + } + setLatestChatSessions( + localGroups + .flatMap(group => group.entries) + .filter( + chat => + !isHostAccessBlocked(chat.agentRef) && + !deletedChatIdsByAgentRef.current.get(chat.agentRef)?.has(chat.id) + ) + .sort(byUpdatedDesc) + ) setLatestChatSessionsLoading(false) const sessionGroups = await Promise.all( - agentNames.map(async agentRef => { + requestedAgentNames.map(async agentRef => { // This feeds only the cross-agent sidebar preview. Do not follow // cursors here; the selected-agent session list exposes explicit // on-demand pagination through `loadMoreChatSessions`. - const serverResult: SessionsListResult = await chatStore - .listSessions(agentRef, { + let serverResult: SessionsListResult + try { + serverResult = await readSessionCatalog(chatStore, agentRef, { agent: agentRef, limit: SESSION_CATALOG_PAGE_LIMIT, }) - .catch((): SessionsListResult => ({ items: [] })) + } catch (error) { + if ( + !cancelled && + authorityScopeGenerationRef.current === authorityScopeGeneration && + currentAuthorityScopeRef.current === authorityScopeAtRequest && + getHostAuthorityEpoch(agentRef) === hostAuthorityEpochByAgent.get(agentRef) && + !isHostAccessBlocked(agentRef) + ) { + if (isConfirmedHostAccessRevoked(error)) onHostAccessRevoked(agentRef) + else if (isAuthorizationError(error)) onHostAuthorityUncertain(agentRef) + if ( + !isConfirmedHostAccessRevoked(error) && + !isAuthorizationError(error) && + isCatalogOutageError(error) + ) { + reportCatalogOffline(agentRef) + } + } + return { agentRef, sessions: [] as SessionsListResult['items'] } + } + if ( + cancelled || + authorityScopeGenerationRef.current !== authorityScopeGeneration || + currentAuthorityScopeRef.current !== authorityScopeAtRequest || + getHostAuthorityEpoch(agentRef) !== hostAuthorityEpochByAgent.get(agentRef) || + isHostAccessBlocked(agentRef) + ) { + return { agentRef, sessions: [] as SessionsListResult['items'] } + } + reportCatalogOnline(agentRef, authorityScopeGeneration) return { agentRef, - sessions: serverResult.items.filter(session => session.agent === agentRef), + sessions: serverResult.items.filter( + session => + session.agent === agentRef && + !isHostAccessBlocked(agentRef) && + !deletedChatIdsByAgentRef.current.get(agentRef)?.has(session.chatId) + ), } }) ) - if (cancelled) return + if (cancelled || currentAuthorityScopeRef.current !== authorityScopeAtRequest) return for (const group of sessionGroups) { - seedSessionSnapshots(fsm, group.agentRef, group.sessions) + if ( + getHostAuthorityEpoch(group.agentRef) !== + hostAuthorityEpochByAgent.get(group.agentRef) || + isHostAccessBlocked(group.agentRef) + ) + continue + seedSessionSnapshots( + fsm, + group.agentRef, + group.sessions.filter( + session => !deletedChatIdsByAgentRef.current.get(group.agentRef)?.has(session.chatId) + ) + ) } setLatestChatSessions(previous => { - const serverByKey = new Map() + const sessions: LatestCatalogSession[] = [] for (const group of sessionGroups) { + if ( + getHostAuthorityEpoch(group.agentRef) !== + hostAuthorityEpochByAgent.get(group.agentRef) || + isHostAccessBlocked(group.agentRef) + ) + continue for (const session of group.sessions) { - serverByKey.set(`${group.agentRef}:${session.chatId}`, session) + if (deletedChatIdsByAgentRef.current.get(group.agentRef)?.has(session.chatId)) + continue + sessions.push({ ...session, agentRef: group.agentRef }) } } - // Cases C/D/E/F (§2.2): server-authoritative title for cached entries, - // unless a local rename is pending for that (agentRef, chatId). - const reconciled = previous.map(item => { - const server = serverByKey.get(`${item.agentRef}:${item.id}`) - if (!server) return item - const { title } = resolveSessionTitle({ - inCache: true, - localTitle: item.title, - serverTitle: server.title, - pendingRename: pendingRenameStateFor(item.agentRef, item.id), - placeholder: remotePlaceholder(item.id), - }) - return title === item.title ? item : { ...item, title } + const cached = previous.filter( + item => + getHostAuthorityEpoch(item.agentRef) === + hostAuthorityEpochByAgent.get(item.agentRef) && + !isHostAccessBlocked(item.agentRef) && + !deletedChatIdsByAgentRef.current.get(item.agentRef)?.has(item.id) + ) + // §2.2 precedence (cases A-F) lives in `mergeCatalogPage`; Latest keys + // entries by (agentRef, chatId) and uses the "Remote · " placeholder. + return mergeCatalogPage({ + cached, + sessions, + entryKey: item => `${item.agentRef}:${item.id}`, + sessionKey: session => `${session.agentRef}:${session.chatId}`, + entryPendingRename: item => pendingRenameStateFor(item.agentRef, item.id), + sessionPendingRename: session => + pendingRenameStateFor(session.agentRef, session.chatId), + placeholderFor: remotePlaceholder, + serverOnlyEntry: (session, title) => ({ + id: session.chatId, + title, + createdAt: session.lastActivityAt, + updatedAt: session.lastActivityAt, + messageCount: knownServerMessageCount(session), + remote: true, + agentRef: session.agentRef, + }), }) - const knownKeys = new Set(previous.map(item => `${item.agentRef}:${item.id}`)) - const remoteOnly: LatestSidebarChatEntry[] = [] - for (const group of sessionGroups) { - for (const session of group.sessions) { - const key = `${group.agentRef}:${session.chatId}` - if (knownKeys.has(key)) continue - knownKeys.add(key) - // Cases A/B (§2.2): server title, else "Remote · " placeholder. - remoteOnly.push({ - id: session.chatId, - title: resolveSessionTitle({ - inCache: false, - serverTitle: session.title, - pendingRename: pendingRenameStateFor(group.agentRef, session.chatId), - placeholder: remotePlaceholder(session.chatId), - }).title, - createdAt: session.lastActivityAt, - updatedAt: session.lastActivityAt, - messageCount: knownServerMessageCount(session), - remote: true, - agentRef: group.agentRef, - }) - } - } - return [...reconciled, ...remoteOnly].sort(byUpdatedDesc) }) // §2.5: retry any rename that 404'd, now that this cross-agent poll @@ -1108,7 +1655,14 @@ export function useChatListController({ flushPendingRenamesRef.current( new Set( sessionGroups.flatMap(group => - group.sessions.map(session => `${group.agentRef}:${session.chatId}`) + isHostAccessBlocked(group.agentRef) + ? [] + : group.sessions + .filter( + session => + !deletedChatIdsByAgentRef.current.get(group.agentRef)?.has(session.chatId) + ) + .map(session => `${group.agentRef}:${session.chatId}`) ) ) ) @@ -1123,28 +1677,44 @@ export function useChatListController({ cancelled = true } }, [ - agentNames, + agentNamesKey, chatStore.getIndex, chatStore.listSessions, isAuthenticated, loadMenuData, scopeKey, fsm, + getHostAuthorityEpoch, + isHostAccessBlocked, + onHostAccessRevoked, + latestReloadRevision, ]) + // Stable across renders whose inputs did not change, so consumers keyed on the + // list identity do not recompute on every parent render (R1-M10). + const visibleLatestChatSessions = useMemo( + () => latestChatSessions.filter(session => !isHostAccessBlocked(session.agentRef)), + // `isHostAccessBlocked` reads mutable authority state; the revision is what + // changes when its answers do. + // eslint-disable-next-line react-hooks/exhaustive-deps + [latestChatSessions, isHostAccessBlocked, hostAuthorityRevision] + ) + return { // State (public contract, re-exported unchanged by the parent). - chatList, + chatList: selectedAgent && isHostAccessBlocked(selectedAgent) ? [] : chatList, chatListLoading, chatListMoreLoading, chatListHasMoreRemoteSessions, - latestChatSessions, + latestChatSessions: visibleLatestChatSessions, latestChatSessionsLoading, + hideAgent, // CRUD (public contract). handleCreateChat, handleRenameChat, handleRenameChatForAgent, applyLocalTitleOnly, + captureChatDeleteFence, handleDeleteChat, handleDeleteChatForAgent, // Loader + list-loading control (parent agent-selection effect). @@ -1166,5 +1736,6 @@ export function useChatListController({ readPendingSelection, writePendingSelection, clearPendingSelection, + isChatDeleted, } } diff --git a/desktop-app/ui/src/hooks/useAgentChatActionsValue.ts b/desktop-app/ui/src/hooks/useAgentChatActionsValue.ts index 739d9c2dcb..49a6826ea8 100644 --- a/desktop-app/ui/src/hooks/useAgentChatActionsValue.ts +++ b/desktop-app/ui/src/hooks/useAgentChatActionsValue.ts @@ -18,6 +18,7 @@ export function useAgentChatActionsValue(vm: AppVm): AgentChatActionsContextValu const handleCreateChat = useStableCallback(vm.handleCreateChat) const handleRenameChat = useStableCallback(vm.handleRenameChat) const handleRenameChatForAgent = useStableCallback(vm.handleRenameChatForAgent) + const captureChatDeleteFence = useStableCallback(vm.captureChatDeleteFence) const handleDeleteChat = useStableCallback(vm.handleDeleteChat) const handleDeleteChatForAgent = useStableCallback(vm.handleDeleteChatForAgent) const handleSelectChat = useStableCallback(vm.handleSelectChat) @@ -47,6 +48,7 @@ export function useAgentChatActionsValue(vm: AppVm): AgentChatActionsContextValu handleCreateChat, handleRenameChat, handleRenameChatForAgent, + captureChatDeleteFence, handleDeleteChat, handleDeleteChatForAgent, handleSelectChat, @@ -67,6 +69,7 @@ export function useAgentChatActionsValue(vm: AppVm): AgentChatActionsContextValu handleCreateChat, handleRenameChat, handleRenameChatForAgent, + captureChatDeleteFence, handleDeleteChat, handleDeleteChatForAgent, handleSelectChat, diff --git a/desktop-app/ui/src/hooks/useAppController.ts b/desktop-app/ui/src/hooks/useAppController.ts index 99965d61fc..57b371f678 100644 --- a/desktop-app/ui/src/hooks/useAppController.ts +++ b/desktop-app/ui/src/hooks/useAppController.ts @@ -11,6 +11,11 @@ import type { import { AGENT_WORKSPACE_ROUTES, DESKTOP_ROUTES } from '../constants/navigation' import { pickLatestAgent } from '../lib/agents' import { toPrettyJson } from '../lib/format' +import { + type HostAuthorityHoldKind, + type HostAuthorityStore, + createHostAuthorityStore, +} from '../lib/hostAuthorityStore' import { selectUnauthenticatedView } from '../lib/unauthenticatedView' import { summarizeWorkflowResource } from '../lib/workflows' import type { @@ -53,6 +58,7 @@ import { createResetWorkflowSelection, } from './domain/useWorkflowController.types' import { scheduleAfterFirstPaint } from './scheduleAfterFirstPaint' +import { useChatStore } from './useChatStore' // U5 cold-start buffer: how long an OAuth completion whose sessions are not yet // seeded is retried against later snapshots before it is discarded. Long enough @@ -212,7 +218,12 @@ export function useAppController() { // here (see refreshAuthenticatedData / handleLogout / switchTeamForWorkspace). const connectorsData = useConnectorsController() const teamsData = useTeamsDataController() - const currentTeamId = auth.me?.teamId || teamsData.currentTeamId || '' + // The authenticated principal's own team: the ONE place `me.teamId` is read. + // The chat-list cache scope (`currentTeamId`) may fall back to the display + // directory when it is empty; the delete fence (`chatAuthorityTeamId`) never + // does, because a directory team was never proven to be the session's. + const principalTeamId = auth.me?.teamId ?? '' + const currentTeamId = principalTeamId || teamsData.currentTeamId || '' const availableTeamIds = useMemo(() => teamsData.teams.map(team => team.id), [teamsData.teams]) const authenticatedPrincipalIdentity = auth.isAuthenticated && auth.me @@ -306,11 +317,73 @@ export function useAppController() { (agentName: string) => agentsData.agentDisplayByName[agentName] ?? agentName, [agentsData.agentDisplayByName] ) + const authorityScope = `${auth.runtimeConfigState?.envKey ?? ''}:${auth.isAuthenticated}:${authenticatedPrincipalIdentity ?? ''}:${currentTeamId}` + const authorityScopeRef = useRef(authorityScope) + authorityScopeRef.current = authorityScope + const navigationIntentEpochRef = useRef(0) + // Lazy-init: the factory must run once per mount, like the FSM store. + const hostAuthorityRef = useRef(null) + if (!hostAuthorityRef.current) hostAuthorityRef.current = createHostAuthorityStore() + const hostAuthority = hostAuthorityRef.current + const [hostAuthorityRevision, setHostAuthorityRevision] = useState(0) + const selectedAgentRef = useRef(nav.selectedAgent) + selectedAgentRef.current = nav.selectedAgent + const isHostAccessBlocked = useCallback( + (agentRef: string) => hostAuthority.isBlocked(agentRef), + [hostAuthority] + ) + const getHostAuthorityEpoch = useCallback( + (agentRef: string) => hostAuthority.getEpoch(agentRef), + [hostAuthority] + ) + useEffect(() => { + hostAuthority.reset() + setHostAuthorityRevision(revision => revision + 1) + }, [authorityScope, hostAuthority]) + const { invalidateSessionCatalog } = useChatStore() + const blockHostAccess = useCallback( + (agentRef: string, kind: HostAuthorityHoldKind) => { + // F4: a catalog request issued before the hold must not be shared with a + // reader that starts after `verifyHostAccess` releases it. + invalidateSessionCatalog(agentRef) + if (!hostAuthority.hold(agentRef, kind)) return + setHostAuthorityRevision(revision => revision + 1) + if (selectedAgentRef.current === agentRef) nav.setSelectedAgent(null) + void agentsData.refresh() + }, + [agentsData.refresh, hostAuthority, invalidateSessionCatalog, nav.setSelectedAgent] + ) + const onHostAccessRevoked = useCallback( + (agentRef: string) => blockHostAccess(agentRef, 'revoked'), + [blockHostAccess] + ) + const onHostAuthorityUncertain = useCallback( + (agentRef: string) => blockHostAccess(agentRef, 'uncertain'), + [blockHostAccess] + ) + const verifyHostAccess = useCallback( + async (agentRef: string): Promise => { + const heldAtEpoch = hostAuthority.heldAtEpoch(agentRef) + if (heldAtEpoch === undefined) return true + const scope = authorityScopeRef.current + try { + await window.clerum.rpc.listSessions(agentRef, undefined, { agent: agentRef, limit: 1 }) + } catch { + return false + } + if (authorityScopeRef.current !== scope || !hostAuthority.release(agentRef, heldAtEpoch)) + return false + setHostAuthorityRevision(revision => revision + 1) + return true + }, + [hostAuthority] + ) const chat = useAgentChatController({ selectedAgent: nav.selectedAgent, agentNames: agentsData.agentNames, currentUserId: auth.me?.id, currentTeamId, + currentEnvironmentKey: auth.runtimeConfigState?.envKey ?? '', currentTeamName, isAuthenticated: auth.isAuthenticated, loadMenuData: postPaintDataReady, @@ -322,6 +395,12 @@ export function useAppController() { showDesktopNotification: notificationSettings.showDesktopNotification, openAgentConversationFromNotification, decideApprovalFromNotification, + onHostAccessRevoked, + onHostAuthorityUncertain, + isHostAccessBlocked, + getHostAuthorityEpoch, + hostAuthorityRevision, + chatAuthorityTeamId: principalTeamId, }) // §4.7.4: the ONE central approval-decision function, bound to the chat @@ -597,7 +676,10 @@ export function useAppController() { useEffect(() => { if (!auth.isAuthenticated) return if (agentsData.accessCatalog && nav.selectedAgent) { - if (!agentsData.agentNames.includes(nav.selectedAgent)) { + if ( + !agentsData.agentNames.includes(nav.selectedAgent) || + isHostAccessBlocked(nav.selectedAgent) + ) { nav.setSelectedAgent(null) } } @@ -607,6 +689,7 @@ export function useAppController() { auth.isAuthenticated, nav.selectedAgent, nav.setSelectedAgent, + isHostAccessBlocked, ]) const switchTeamForWorkspace = useCallback( @@ -859,7 +942,7 @@ export function useAppController() { ]) // ─── Cross-domain: handleOpenAgentWorkspace ─── - const handleOpenAgentWorkspace = useCallback( + const openAgentWorkspace = useCallback( (agentName: string, route: AgentWorkspaceRoute = AGENT_WORKSPACE_ROUTES.connectors) => { if (!agentName) return chat.setPendingChatSelection(agentName, null) @@ -878,9 +961,30 @@ export function useAppController() { nav.setSelectedAgentRoute, ] ) + const handleOpenAgentWorkspace = useCallback( + (agentName: string, route: AgentWorkspaceRoute = AGENT_WORKSPACE_ROUTES.connectors) => { + if (!agentName) return + const navigationIntentEpoch = ++navigationIntentEpochRef.current + if (!isHostAccessBlocked(agentName)) { + openAgentWorkspace(agentName, route) + return + } + const scope = authorityScopeRef.current + void verifyHostAccess(agentName).then(verified => { + if ( + verified && + authorityScopeRef.current === scope && + navigationIntentEpochRef.current === navigationIntentEpoch + ) { + openAgentWorkspace(agentName, route) + } + }) + }, + [isHostAccessBlocked, openAgentWorkspace, verifyHostAccess] + ) // ─── Cross-domain: handleSelectChatAgent (Chat page agent picker) ─── - const handleSelectChatAgent = useCallback( + const selectChatAgent = useCallback( ( agentName: string, options: { @@ -981,6 +1085,27 @@ export function useAppController() { nav.setSelectedAgentRoute, ] ) + const handleSelectChatAgent = useCallback( + (agentName: string, options: Parameters[1] = {}) => { + if (!agentName) return + const navigationIntentEpoch = ++navigationIntentEpochRef.current + if (!isHostAccessBlocked(agentName)) { + selectChatAgent(agentName, options) + return + } + const scope = authorityScopeRef.current + void verifyHostAccess(agentName).then(verified => { + if ( + verified && + authorityScopeRef.current === scope && + navigationIntentEpochRef.current === navigationIntentEpoch + ) { + selectChatAgent(agentName, options) + } + }) + }, + [isHostAccessBlocked, selectChatAgent, verifyHostAccess] + ) // ─── Cross-domain: handleNavSelect (extended) ─── const handleNavSelect = useCallback( @@ -1037,6 +1162,7 @@ export function useAppController() { if (requiresTeamSwitch) { await ensureTeamContext({ teamId: targetTeamId }) } + if (isHostAccessBlocked(targetAgent) && !(await verifyHostAccess(targetAgent))) return if (stayInDrawer) { // handleSelectChatAgent(keepNavItem) sets the active chat without @@ -1090,6 +1216,8 @@ export function useAppController() { ensureTeamContext, fullSetStatus, handleSelectChatAgent, + isHostAccessBlocked, + verifyHostAccess, nav.activateChatTab, nav.navItem, nav.selectedAgent, @@ -1336,6 +1464,8 @@ export function useAppController() { // Navigation navItem: nav.navItem, selectedAgent: nav.selectedAgent, + isHostAccessBlocked, + hostAuthorityRevision, selectedAgentRoute: nav.selectedAgentRoute, setSelectedAgent: nav.setSelectedAgent, handleNavSelect, @@ -1419,6 +1549,7 @@ export function useAppController() { handleCreateChat: chat.handleCreateChat, handleRenameChat: chat.handleRenameChat, handleRenameChatForAgent: chat.handleRenameChatForAgent, + captureChatDeleteFence: chat.captureChatDeleteFence, handleDeleteChat: chat.handleDeleteChat, handleDeleteChatForAgent: chat.handleDeleteChatForAgent, handleSelectChat: chat.handleSelectChat, diff --git a/desktop-app/ui/src/hooks/useChatStore.ts b/desktop-app/ui/src/hooks/useChatStore.ts index 4569da3cf3..f8e7c82b00 100644 --- a/desktop-app/ui/src/hooks/useChatStore.ts +++ b/desktop-app/ui/src/hooks/useChatStore.ts @@ -1,5 +1,7 @@ import { useCallback } from 'react' import type { + ChatAuthorityScope, + ChatDeleteFence, ChatMessage, ReplaceChatMessagesOptions, SessionMessagesQuery, @@ -38,6 +40,8 @@ const SESSION_CATALOG_TTL_MS = 5_000 type CachedRequest = { expiresAt: number + /** Host the request was issued against; the key's scope segment may contain ':'. */ + hostRef: string promise: Promise } @@ -64,12 +68,19 @@ export function useChatStore() { [] ) const renameChat = useCallback( - (agentRef: string, chatId: string, title: string) => - window.clerum.chat.rename(agentRef, chatId, title), + (agentRef: string, chatId: string, title: string, bindingGeneration: number) => + window.clerum.chat.rename(agentRef, chatId, title, bindingGeneration), + [] + ) + const getBindingGeneration = useCallback(() => window.clerum.chat.getBindingGeneration(), []) + const captureDeleteFence = useCallback( + (expectedAuthorityScope: ChatAuthorityScope) => + window.clerum.chat.captureDeleteFence(expectedAuthorityScope), [] ) const deleteChat = useCallback( - (agentRef: string, chatId: string) => window.clerum.chat.delete(agentRef, chatId), + (agentRef: string, chatId: string, fence: ChatDeleteFence) => + window.clerum.chat.delete(agentRef, chatId, fence), [] ) const loadMessages = useCallback( @@ -82,6 +93,11 @@ export function useChatStore() { window.clerum.chat.appendMessages(agentRef, chatId, messages), [] ) + const upsertMessages = useCallback( + (agentRef: string, chatId: string, messages: ChatMessage[]) => + window.clerum.chat.upsertMessages(agentRef, chatId, messages), + [] + ) const backfillCounters = useCallback( (agentRef: string, chatId: string, messages: ChatMessage[]) => { const backfill = window.clerum.chat.backfillCounters @@ -148,15 +164,18 @@ export function useChatStore() { const cached = sessionCatalogRequests.get(key) if (!options.force && cached && cached.expiresAt > Date.now()) return cached.promise - const promise = source(hostRef, undefined, query).catch(error => { - sessionCatalogRequests.delete(key) - throw error - }) - sessionCatalogRequests.set(key, { + const entry: CachedRequest = { expiresAt: Date.now() + SESSION_CATALOG_TTL_MS, - promise, - }) - return promise + hostRef, + promise: source(hostRef, undefined, query).catch(error => { + // An invalidation may have replaced this entry with a newer request + // under the same key; only this request's own entry is evicted. + if (sessionCatalogRequests.get(key) === entry) sessionCatalogRequests.delete(key) + throw error + }), + } + sessionCatalogRequests.set(key, entry) + return entry.promise }, [] ) @@ -196,6 +215,14 @@ export function useChatStore() { sessionCatalogRequests.clear() sessionCatalogSource = null }, []) + // F4: a catalog request issued before a Host hold carries the authority of + // that moment. Dropping it when the Host is held keeps a reader that starts + // after `verifyHostAccess` released the hold from sharing it. + const invalidateSessionCatalog = useCallback((hostRef: string) => { + for (const [key, cached] of sessionCatalogRequests) { + if (cached.hostRef === hostRef) sessionCatalogRequests.delete(key) + } + }, []) const setRemoteCacheScope = useCallback((scope: string) => { if (remoteCacheScope === scope) return remoteCacheScope = scope @@ -244,9 +271,12 @@ export function useChatStore() { listChats, createChat, renameChat, + getBindingGeneration, + captureDeleteFence, deleteChat, loadMessages, appendMessages, + upsertMessages, replaceMessages, backfillCounters, markUnreadTerminal, @@ -263,6 +293,7 @@ export function useChatStore() { setHostModel, renameSession, clearCachedRemoteData, + invalidateSessionCatalog, setRemoteCacheScope, setPendingModel, getPendingModel, diff --git a/desktop-app/ui/src/hooks/useComposerDraft.ts b/desktop-app/ui/src/hooks/useComposerDraft.ts index dffeeaea53..f5973edb7c 100644 --- a/desktop-app/ui/src/hooks/useComposerDraft.ts +++ b/desktop-app/ui/src/hooks/useComposerDraft.ts @@ -11,13 +11,17 @@ import { * not. Returns a `[draft, setDraft]` pair scoped to `chatId`. */ export function useComposerDraft( - chatId: string | null + chatId: string | null, + agentRef?: string ): readonly [string, (value: string) => void] { const subscribe = useCallback( - (listener: () => void) => subscribeComposerDraft(chatId, listener), - [chatId] + (listener: () => void) => subscribeComposerDraft(chatId, listener, agentRef), + [agentRef, chatId] + ) + const draft = useSyncExternalStore(subscribe, () => getComposerDraft(chatId, agentRef)) + const setDraft = useCallback( + (value: string) => setComposerDraft(chatId, value, agentRef), + [agentRef, chatId] ) - const draft = useSyncExternalStore(subscribe, () => getComposerDraft(chatId)) - const setDraft = useCallback((value: string) => setComposerDraft(chatId, value), [chatId]) return [draft, setDraft] as const } diff --git a/desktop-app/ui/src/lib/__tests__/catalogMerge.prop.test.ts b/desktop-app/ui/src/lib/__tests__/catalogMerge.prop.test.ts new file mode 100644 index 0000000000..bb0cd63d8d --- /dev/null +++ b/desktop-app/ui/src/lib/__tests__/catalogMerge.prop.test.ts @@ -0,0 +1,287 @@ +import { describe, expect, it } from 'vitest' +import fc from 'fast-check' +import { byUpdatedDesc, mergeCatalogPage } from '../catalogMerge' +import type { PendingRename } from '../resolveSessionTitle' + +/** + * R1-M9 — property suite for the single sidebar/catalog merge precedence rule + * (spec 15 §2.2). The expected titles are written out from the case table, not + * computed by calling `resolveSessionTitle`, so a regression in either the + * merge or the resolver fails here. + */ + +interface Entry { + id: string + title: string + updatedAt: string + /** A field the merge must never touch. */ + messageCount: number +} + +interface Session { + chatId: string + title?: string + lastActivityAt: string +} + +const CHAT_IDS = ['a', 'b', 'c', 'd', 'e', 'f'] as const +const PENDING: PendingRename[] = ['none', 'in-flight', 'offline'] + +const placeholderFor = (chatId: string) => `Chat ${chatId}` + +const timestampArb = fc.oneof( + fc + .integer({ min: Date.UTC(2020, 0, 1), max: Date.UTC(2030, 0, 1) }) + .map(ms => new Date(ms).toISOString()), + fc.constant('not-a-date') +) +const titleArb = fc.constantFrom('', 'Local plan', 'Server plan', 'Renamed') + +const entriesArb = fc.uniqueArray( + fc.record({ + id: fc.constantFrom(...CHAT_IDS), + title: titleArb, + updatedAt: timestampArb, + messageCount: fc.nat({ max: 50 }), + }), + { selector: entry => entry.id, maxLength: CHAT_IDS.length } +) +const sessionsArb = fc.array( + fc.record( + { + chatId: fc.constantFrom(...CHAT_IDS), + title: titleArb, + lastActivityAt: timestampArb, + }, + { requiredKeys: ['chatId', 'lastActivityAt'] } + ), + { maxLength: 10 } +) +const pendingArb = fc.dictionary(fc.constantFrom(...CHAT_IDS), fc.constantFrom(...PENDING)) + +function merge(cached: Entry[], sessions: Session[], pending: Record) { + const pendingFor = (chatId: string): PendingRename => pending[chatId] ?? 'none' + return mergeCatalogPage({ + cached, + sessions, + entryKey: entry => entry.id, + sessionKey: session => session.chatId, + entryPendingRename: entry => pendingFor(entry.id), + sessionPendingRename: session => pendingFor(session.chatId), + placeholderFor, + serverOnlyEntry: (session, title) => ({ + id: session.chatId, + title, + updatedAt: session.lastActivityAt, + messageCount: 0, + }), + }) +} + +/** §2.2 cases C/D/E/F for a cached entry the page also reports. */ +function expectedCachedTitle(local: string, server: string | undefined, pending: PendingRename) { + if (pending !== 'none' && local !== '') return local // E/F + if (server) return server // C + if (local !== '') return local // D + return 'placeholder' +} + +/** §2.2 cases A/B for a session the cache does not hold. */ +function expectedServerOnlyTitle(server: string | undefined, chatId: string) { + return server ? server : placeholderFor(chatId) +} + +const firstReport = (sessions: Session[], chatId: string) => + sessions.find(session => session.chatId === chatId) + +describe('mergeCatalogPage', () => { + it('holds every cached entry and every reported chat exactly once', () => { + fc.assert( + fc.property(entriesArb, sessionsArb, pendingArb, (cached, sessions, pending) => { + const ids = merge(cached, sessions, pending).map(entry => entry.id) + const expected = new Set([ + ...cached.map(entry => entry.id), + ...sessions.map(session => session.chatId), + ]) + expect(ids).toHaveLength(expected.size) + expect(new Set(ids)).toEqual(expected) + }) + ) + }) + + it('titles a cached entry by the §2.2 precedence and leaves its other fields alone', () => { + fc.assert( + fc.property(entriesArb, sessionsArb, pendingArb, (cached, sessions, pending) => { + const merged = merge(cached, sessions, pending) + for (const entry of cached) { + const out = merged.find(candidate => candidate.id === entry.id) + const session = firstReport(sessions, entry.id) + if (!session) { + expect(out).toBe(entry) + continue + } + const expected = expectedCachedTitle( + entry.title, + session.title, + pending[entry.id] ?? 'none' + ) + expect(out).toEqual({ + ...entry, + title: expected === 'placeholder' ? placeholderFor(entry.id) : expected, + }) + if (out?.title === entry.title) expect(out).toBe(entry) + } + }) + ) + }) + + it('builds one server-only entry from the first report of each new chat', () => { + fc.assert( + fc.property(entriesArb, sessionsArb, pendingArb, (cached, sessions, pending) => { + const merged = merge(cached, sessions, pending) + const cachedIds = new Set(cached.map(entry => entry.id)) + for (const chatId of new Set(sessions.map(session => session.chatId))) { + if (cachedIds.has(chatId)) continue + const first = firstReport(sessions, chatId) as Session + expect(merged.find(entry => entry.id === chatId)).toEqual({ + id: chatId, + title: expectedServerOnlyTitle(first.title, chatId), + updatedAt: first.lastActivityAt, + messageCount: 0, + }) + } + }) + ) + }) + + it('orders the result newest first', () => { + fc.assert( + fc.property(entriesArb, sessionsArb, pendingArb, (cached, sessions, pending) => { + const merged = merge(cached, sessions, pending) + for (let i = 1; i < merged.length; i += 1) { + expect(byUpdatedDesc(merged[i - 1] as Entry, merged[i] as Entry)).toBeLessThanOrEqual(0) + } + }) + ) + }) + + it('is idempotent: merging the same page again changes nothing', () => { + fc.assert( + fc.property(entriesArb, sessionsArb, pendingArb, (cached, sessions, pending) => { + const once = merge(cached, sessions, pending) + expect(merge(once, sessions, pending)).toEqual(once) + }) + ) + }) + + // Cursor pages report disjoint chats, so merging page A and then page B must + // equal one merge of both. Overlapping pages do not compose by design: a + // later report updates a chat's title but its `updatedAt` stays from the + // report that created the entry. + it('composes over disjoint pages: A then B equals one merge of A and B', () => { + fc.assert( + fc.property( + entriesArb, + sessionsArb, + fc.subarray([...CHAT_IDS]), + pendingArb, + (cached, sessions, pageAIds, pending) => { + const inPageA = new Set(pageAIds) + const pageA = sessions.filter(session => inPageA.has(session.chatId)) + const pageB = sessions.filter(session => !inPageA.has(session.chatId)) + + const sequential = merge(merge(cached, pageA, pending), pageB, pending) + + // Witness: both pages reached the result, so a merge that ignores + // its sessions cannot satisfy the equality below. + expect(new Set(sequential.map(entry => entry.id))).toEqual( + new Set([...cached.map(entry => entry.id), ...sessions.map(s => s.chatId)]) + ) + expect(sequential).toEqual(merge(cached, [...pageA, ...pageB], pending)) + } + ) + ) + }) + + it('orders known dates newest first with an unparseable date as the epoch (fixed oracle)', () => { + // The property test above checks order with the production comparator, so an + // inverted comparator or a different fallback for NaN passes it. This case + // states the expected order literally. + const cached: Entry[] = [ + { id: 'jan', title: 'Jan', updatedAt: '2026-01-01T00:00:00.000Z', messageCount: 0 }, + { id: 'invalid', title: 'Invalid', updatedAt: 'no-fecha', messageCount: 0 }, + { id: 'pre-epoch', title: 'Pre', updatedAt: '1969-06-01T00:00:00.000Z', messageCount: 0 }, + { id: 'mar', title: 'Mar', updatedAt: '2026-03-01T00:00:00.000Z', messageCount: 0 }, + ] + const sessions: Session[] = [ + { chatId: 'feb', title: 'Feb', lastActivityAt: '2026-02-01T00:00:00.000Z' }, + ] + expect(merge(cached, sessions, {}).map(entry => entry.id)).toEqual([ + 'mar', + 'feb', + 'jan', + 'invalid', + 'pre-epoch', + ]) + }) + + it('names the placeholder after the cached entry id, not the reported chat id', () => { + // Entry and session share a key but not an id, so the two placeholder calls + // are distinguishable. The cached title is empty and the server has none, so + // the placeholder is what the entry ends up titled. + const calls: string[] = [] + const merged = mergeCatalogPage({ + cached: [ + { id: 'cached-id', title: '', updatedAt: '2026-01-01T00:00:00.000Z', messageCount: 0 }, + ], + sessions: [{ chatId: 'reported-chat-id', lastActivityAt: '2026-02-01T00:00:00.000Z' }], + entryKey: () => 'shared-key', + sessionKey: () => 'shared-key', + entryPendingRename: () => 'none', + sessionPendingRename: () => 'none', + placeholderFor: chatId => { + calls.push(chatId) + return `Chat ${chatId}` + }, + serverOnlyEntry: (session, title) => ({ + id: session.chatId, + title, + updatedAt: session.lastActivityAt, + messageCount: 0, + }), + }) + // Liveness witness: the merge reached the placeholder path for the shared key. + expect(calls).toEqual(['cached-id']) + expect(merged.map(entry => entry.title)).toEqual(['Chat cached-id']) + }) + + it('names a server-only session placeholder after its chat id', () => { + const merged = mergeCatalogPage({ + cached: [], + sessions: [{ chatId: 'reported-chat-id', lastActivityAt: '2026-02-01T00:00:00.000Z' }], + entryKey: entry => entry.id, + sessionKey: session => session.chatId, + entryPendingRename: () => 'none', + sessionPendingRename: () => 'none', + placeholderFor, + serverOnlyEntry: (session, title) => ({ + id: session.chatId, + title, + updatedAt: session.lastActivityAt, + messageCount: 0, + }), + }) + expect(merged.map(entry => entry.title)).toEqual(['Chat reported-chat-id']) + }) + + it('keeps a pending local rename over a server title (case E)', () => { + const cached: Entry[] = [ + { id: 'a', title: 'Renamed', updatedAt: '2026-09-12T00:00:00.000Z', messageCount: 3 }, + ] + const sessions: Session[] = [ + { chatId: 'a', title: 'Server plan', lastActivityAt: '2026-09-13T00:00:00.000Z' }, + ] + expect(merge(cached, sessions, { a: 'in-flight' })[0]?.title).toBe('Renamed') + expect(merge(cached, sessions, {})[0]?.title).toBe('Server plan') + }) +}) diff --git a/desktop-app/ui/src/lib/__tests__/classifyErrorKind.test.ts b/desktop-app/ui/src/lib/__tests__/classifyErrorKind.test.ts index e128b53ef1..5a643c3bd5 100644 --- a/desktop-app/ui/src/lib/__tests__/classifyErrorKind.test.ts +++ b/desktop-app/ui/src/lib/__tests__/classifyErrorKind.test.ts @@ -5,7 +5,7 @@ describe('classifyErrorKind — host availability codes', () => { it('classifies host_waking messages as waking, even wrapped by Electron IPC', () => { expect( classifyErrorKind( - 'Error invoking remote method rpc:invokeHostMessage: ' + + "Error invoking remote method 'rpc:invokeHostMessage': " + 'Error: host_waking: agent host "chatllm" is waking up — retry shortly' ) ).toBe('waking') diff --git a/desktop-app/ui/src/lib/__tests__/composerDraftStore.test.ts b/desktop-app/ui/src/lib/__tests__/composerDraftStore.test.ts index b2ac9e1757..5e9b0dcdda 100644 --- a/desktop-app/ui/src/lib/__tests__/composerDraftStore.test.ts +++ b/desktop-app/ui/src/lib/__tests__/composerDraftStore.test.ts @@ -95,6 +95,21 @@ describe('composerDraftStore', () => { expect(getComposerDraft('chat-1')).toBe('') }) + it('isolates no-chat drafts by agent while preserving the legacy unscoped bucket', () => { + setComposerDraft(null, 'agent-a draft', 'agent-a') + setComposerDraft(null, 'agent-b draft', 'agent-b') + setComposerDraft(null, 'legacy draft') + + expect(getComposerDraft(null, 'agent-a')).toBe('agent-a draft') + expect(getComposerDraft(null, 'agent-b')).toBe('agent-b draft') + expect(getComposerDraft(null)).toBe('legacy draft') + + clearComposerDraftAfterSend(null, 'agent-a') + expect(getComposerDraft(null, 'agent-a')).toBe('') + expect(getComposerDraft(null, 'agent-b')).toBe('agent-b draft') + expect(getComposerDraft(null)).toBe('legacy draft') + }) + it('resetComposerDraftStore wipes all drafts and listeners', () => { const listener = vi.fn() subscribeComposerDraft('chat-1', listener) diff --git a/desktop-app/ui/src/lib/__tests__/hostAuthorityStore.test.ts b/desktop-app/ui/src/lib/__tests__/hostAuthorityStore.test.ts new file mode 100644 index 0000000000..54fe9fbc30 --- /dev/null +++ b/desktop-app/ui/src/lib/__tests__/hostAuthorityStore.test.ts @@ -0,0 +1,54 @@ +import { describe, expect, it } from 'vitest' +import { createHostAuthorityStore } from '../hostAuthorityStore' + +describe('createHostAuthorityStore', () => { + it('keeps Host X authority unchanged when Host Y is held', () => { + const store = createHostAuthorityStore() + const hostXEpoch = store.getEpoch('host-x') + + expect(store.hold('host-y', 'revoked')).toBe(true) + + expect(store.getEpoch('host-x')).toBe(hostXEpoch) + expect(store.getEpoch('host-y')).not.toBe(hostXEpoch) + expect(store.isBlocked('host-y')).toBe(true) + expect(store.isBlocked('host-x')).toBe(false) + }) + + it('never downgrades a confirmed revocation to an uncertain hold', () => { + const store = createHostAuthorityStore() + store.hold('host-y', 'revoked') + const heldAt = store.heldAtEpoch('host-y') + + expect(store.hold('host-y', 'uncertain')).toBe(false) + expect(store.heldAtEpoch('host-y')).toBe(heldAt) + }) + + it('refuses a stale release and moves the epoch on a valid one', () => { + const store = createHostAuthorityStore() + store.hold('host-y', 'uncertain') + const firstHold = store.heldAtEpoch('host-y') + if (firstHold === undefined) throw new Error('host-y must be held') + store.hold('host-y', 'revoked') + + expect(store.release('host-y', firstHold)).toBe(false) + expect(store.isBlocked('host-y')).toBe(true) + + const currentHold = store.heldAtEpoch('host-y') + if (currentHold === undefined) throw new Error('host-y must still be held') + const beforeRelease = store.getEpoch('host-y') + expect(store.release('host-y', currentHold)).toBe(true) + expect(store.isBlocked('host-y')).toBe(false) + expect(store.getEpoch('host-y')).not.toBe(beforeRelease) + }) + + it('reset clears every hold and changes the epoch of a previously held Host', () => { + const store = createHostAuthorityStore() + store.hold('host-y', 'revoked') + const heldEpoch = store.getEpoch('host-y') + + store.reset() + + expect(store.isBlocked('host-y')).toBe(false) + expect(store.getEpoch('host-y')).not.toBe(heldEpoch) + }) +}) diff --git a/desktop-app/ui/src/lib/__tests__/httpErrorStatus.test.ts b/desktop-app/ui/src/lib/__tests__/httpErrorStatus.test.ts new file mode 100644 index 0000000000..85f61fbc8f --- /dev/null +++ b/desktop-app/ui/src/lib/__tests__/httpErrorStatus.test.ts @@ -0,0 +1,51 @@ +import { describe, expect, it } from 'vitest' +import { httpErrorStatus, isAuthorizationError } from '../format' + +describe('httpErrorStatus labeled-status boundary (NEW-dui-4)', () => { + it.each([ + ['request to status401 failed', undefined], + ['upstream support-http500 refused the call', undefined], + ['status 503 from upstream', 503], + ['upstream returned status: 429', 429], + ['request failed (status=404)', 404], + ['upstream http status 502', 502], + ['proxy error, status code 504', 504], + ])('parses %j as %s', (message, expected) => { + expect(httpErrorStatus(new Error(message))).toBe(expected) + }) + + it('never reads a status out of a waking or draining Host reference', () => { + const waking = new Error('host_waking: agent host "status401" is waking up — retry shortly') + const draining = new Error('host_draining: agent host "status403" is draining — retry shortly') + + expect(httpErrorStatus(waking)).toBeUndefined() + expect(isAuthorizationError(waking)).toBe(false) + expect(httpErrorStatus(draining)).toBeUndefined() + // Liveness witness: the same Host reference outside the availability + // projection still reaches the parser, which reads the real status. + expect(httpErrorStatus(new Error('403 Forbidden: agent host "status401"'))).toBe(403) + }) +}) + +describe('httpErrorStatus bounded parsing (NEW-sec-2)', () => { + it('finishes a 40k whitespace run after a status label quickly', () => { + const hostile = new Error(`upstream status${' '.repeat(40_000)}x`) + + const startedAt = performance.now() + const status = httpErrorStatus(hostile) + const elapsedMs = performance.now() - startedAt + + expect(status).toBeUndefined() + expect(elapsedMs).toBeLessThan(200) + // Liveness witness: an ordinary labeled message still parses. + expect(httpErrorStatus(new Error('upstream status: 503'))).toBe(503) + }) + + it('reads only the leading 2 KB of a message', () => { + const padding = 'x'.repeat(3000) + + expect(httpErrorStatus(new Error(`${padding} status 500`))).toBeUndefined() + // Liveness witness: the same label inside the bound is parsed. + expect(httpErrorStatus(new Error(`${'x'.repeat(100)} status 500`))).toBe(500) + }) +}) diff --git a/desktop-app/ui/src/lib/catalogMerge.ts b/desktop-app/ui/src/lib/catalogMerge.ts new file mode 100644 index 0000000000..1061b84bd5 --- /dev/null +++ b/desktop-app/ui/src/lib/catalogMerge.ts @@ -0,0 +1,97 @@ +import { type PendingRename, resolveSessionTitle } from '@lib/resolveSessionTitle' + +/** + * Merge precedence between the sidebar's cached chat entries and one page of the + * server session catalog (spec 15 §2.2). The same rule set used to be written + * out three times in `useChatListController` (the selected agent's first page, + * its load-more pages, and the cross-agent Latest list); this is the single + * definition. + * + * - Every cached entry survives, with every field but `title` untouched. Its + * title follows `resolveSessionTitle` (cases C/D/E/F): the server title wins + * when the Host reports one and no local rename is pending; a pending rename + * keeps the local title. An entry whose title is unchanged keeps its object + * identity. + * - A server session with no cached entry becomes one new entry (cases A/B), + * titled by the server or by the placeholder. A key reported twice by the + * page yields one entry, and the first report decides it. + * - The result is ordered by `updatedAt`, newest first; an unparseable + * timestamp sorts as the epoch. + * + * Callers own everything that is not precedence: authority epochs, deletion + * tombstones and cached-entry de-duplication happen before this runs. + */ + +export interface CatalogMergeEntry { + id: string + title: string + updatedAt: string +} + +export interface CatalogMergeSession { + chatId: string + title?: string + lastActivityAt: string +} + +export interface CatalogMergeInput { + cached: readonly E[] + sessions: readonly S[] + /** Identity shared by an entry and the session that describes it. */ + entryKey: (entry: E) => string + sessionKey: (session: S) => string + entryPendingRename: (entry: E) => PendingRename + sessionPendingRename: (session: S) => PendingRename + placeholderFor: (chatId: string) => string + /** Builds the entry for a server-only session from its resolved title. */ + serverOnlyEntry: (session: S, title: string) => E +} + +function sortableTimestamp(value: string): number { + const timestamp = Date.parse(value) + return Number.isFinite(timestamp) ? timestamp : 0 +} + +export function byUpdatedDesc(a: { updatedAt: string }, b: { updatedAt: string }): number { + return sortableTimestamp(b.updatedAt) - sortableTimestamp(a.updatedAt) +} + +export function mergeCatalogPage( + input: CatalogMergeInput +): E[] { + const sessionByKey = new Map() + for (const session of input.sessions) { + const key = input.sessionKey(session) + if (!sessionByKey.has(key)) sessionByKey.set(key, session) + } + + const cachedKeys = new Set() + const reconciled = input.cached.map(entry => { + const key = input.entryKey(entry) + cachedKeys.add(key) + const session = sessionByKey.get(key) + if (!session) return entry + const { title } = resolveSessionTitle({ + inCache: true, + localTitle: entry.title, + serverTitle: session.title, + pendingRename: input.entryPendingRename(entry), + placeholder: input.placeholderFor(entry.id), + }) + return title === entry.title ? entry : { ...entry, title } + }) + + const serverOnly: E[] = [] + for (const [key, session] of sessionByKey) { + if (cachedKeys.has(key)) continue + const { title } = resolveSessionTitle({ + inCache: false, + serverTitle: session.title, + pendingRename: input.sessionPendingRename(session), + placeholder: input.placeholderFor(session.chatId), + }) + serverOnly.push(input.serverOnlyEntry(session, title)) + } + + return [...reconciled, ...serverOnly].sort(byUpdatedDesc) +} diff --git a/desktop-app/ui/src/lib/composerDraftStore.ts b/desktop-app/ui/src/lib/composerDraftStore.ts index 8fce8e0310..a75b38c07a 100644 --- a/desktop-app/ui/src/lib/composerDraftStore.ts +++ b/desktop-app/ui/src/lib/composerDraftStore.ts @@ -8,13 +8,7 @@ type Listener = () => void -/** - * Draft key used while no chat is active yet (before the first send creates one). - * This bucket is shared, not per-agent: it assumes at most one "no active chat" - * composer is mounted at a time (true today — the inline composer renders only when - * `!activeChatId`). A future split-view that showed two no-chat composers at once - * would make them share this draft. - */ +/** Legacy no-chat key retained for callers that do not yet provide an agent. */ const NO_CHAT_DRAFT_KEY = '__no_chat__' const drafts = new Map() @@ -22,8 +16,10 @@ let revision = 0 const revisions = new Map() const listenersByKey = new Map>() -function keyFor(chatId: string | null): string { - return chatId ?? NO_CHAT_DRAFT_KEY +function keyFor(chatId: string | null, agentRef?: string): string { + if (chatId) return chatId + const agent = agentRef?.trim() + return agent ? `${NO_CHAT_DRAFT_KEY}:${agent}` : NO_CHAT_DRAFT_KEY } function emit(key: string): void { @@ -32,16 +28,16 @@ function emit(key: string): void { for (const listener of listeners) listener() } -export function getComposerDraftRevision(chatId: string | null): number { - return revisions.get(keyFor(chatId)) ?? 0 +export function getComposerDraftRevision(chatId: string | null, agentRef?: string): number { + return revisions.get(keyFor(chatId, agentRef)) ?? 0 } -export function getComposerDraft(chatId: string | null): string { - return drafts.get(keyFor(chatId)) ?? '' +export function getComposerDraft(chatId: string | null, agentRef?: string): string { + return drafts.get(keyFor(chatId, agentRef)) ?? '' } -export function setComposerDraft(chatId: string | null, value: string): void { - const key = keyFor(chatId) +export function setComposerDraft(chatId: string | null, value: string, agentRef?: string): void { + const key = keyFor(chatId, agentRef) if ((drafts.get(key) ?? '') === value) return revisions.set(key, ++revision) if (value) drafts.set(key, value) @@ -49,18 +45,31 @@ export function setComposerDraft(chatId: string | null, value: string): void { emit(key) } -export function clearComposerDraft(chatId: string | null): void { - setComposerDraft(chatId, '') +export function clearComposerDraft(chatId: string | null, agentRef?: string): void { + setComposerDraft(chatId, '', agentRef) } /** Clears the draft for a chat and the "no chat yet" bucket (used on send). */ -export function clearComposerDraftAfterSend(chatId: string | null): void { - clearComposerDraft(null) +export function clearComposerDraftAfterSend(chatId: string | null, agentRef?: string): void { + clearComposerDraft(null, agentRef) if (chatId) clearComposerDraft(chatId) } -export function subscribeComposerDraft(chatId: string | null, listener: Listener): () => void { - const key = keyFor(chatId) +/** Drop drafts on logout or principal/team change and notify mounted composers. */ +export function clearAllComposerDrafts(): void { + for (const key of [...drafts.keys()]) { + drafts.delete(key) + revisions.set(key, ++revision) + emit(key) + } +} + +export function subscribeComposerDraft( + chatId: string | null, + listener: Listener, + agentRef?: string +): () => void { + const key = keyFor(chatId, agentRef) let listeners = listenersByKey.get(key) if (!listeners) { listeners = new Set() diff --git a/desktop-app/ui/src/lib/format.ts b/desktop-app/ui/src/lib/format.ts index 37942b9ae0..5f6efc41a5 100644 --- a/desktop-app/ui/src/lib/format.ts +++ b/desktop-app/ui/src/lib/format.ts @@ -1,5 +1,6 @@ import type { HostMessageResponse, SessionTokensLite } from '../../../src/types' import type { AppErrorKind } from '../uiTypes' +import { stripIpcWrapper } from './gfsGrantErrors' export function toPrettyJson(value: unknown): string { return JSON.stringify(value, null, 2) @@ -159,22 +160,98 @@ function errorText(err: unknown): string { } /** - * True when an error reflects an HTTP 404 (the server doesn't know a chat the - * local cache references). Used by the D.4 reconcile to evict a stale local - * chat — which is destructive, so we require the 404 status token specifically - * rather than a generic "not found" substring (a "host/agent not found" - * transport error must NOT evict the user's cache). + * Every producer puts the response status in the first few dozen characters + * (`NNN StatusText: …`, `List sessions failed (NNN): …`). Parsing a bounded + * prefix keeps the cost of a status read independent of an upstream body that + * an error message may echo. */ +const HTTP_STATUS_PARSE_MAX_CHARS = 2048 + +/** + * Labeled statuses must be introduced by the start of the text, whitespace or + * `(`, and separated from the digits by `:`/`=` or whitespace, so a Host named + * `status401` or `support-http500` never yields a status. Each quantifier is + * followed by a disjoint token, which keeps a long whitespace run linear. + */ +const HTTP_STATUS_PATTERN = + /^(?:http\s+)?(?[1-5]\d{2})\b|(?:^|[\s(])(?:http(?:\s+status)?|status(?:\s+code)?)(?:\s*[:=]\s*|\s+)(?[1-5]\d{2})\b|\((?[1-5]\d{2})\)/i + +/** IPC preserves error messages but not always custom status fields. */ +export function httpErrorStatus(err: unknown): number | undefined { + if (err && typeof err === 'object') { + const status = (err as { status?: unknown; cause?: { status?: unknown } }).status + if (typeof status === 'number') return status + const causeStatus = (err as { cause?: { status?: unknown } }).cause?.status + if (typeof causeStatus === 'number') return causeStatus + } + const text = stripIpcWrapper(errorText(err).slice(0, HTTP_STATUS_PARSE_MAX_CHARS)) + // AppService's Host-availability projection carries no status, only the + // Host reference, whose digits are never a response status. + if (text.startsWith('host_waking:') || text.startsWith('host_draining:')) return undefined + // Parse only labeled HTTP statuses so digits in a host name or body aren't + // mistaken for the response status. + const match = HTTP_STATUS_PATTERN.exec(text) + const status = match?.groups?.leading ?? match?.groups?.labeled ?? match?.groups?.parenthesized + return status ? Number(status) : undefined +} + +const HOST_AVAILABILITY_BODY_PATTERN = /"code"\s*:\s*"host_(?:waking|draining)"/ + +/** + * The Host is waking or draining: an expected transition, not an outage. Invoke + * failures arrive as AppService's `host_waking:`/`host_draining:` projection; + * catalog reads arrive as the rpc-proxy 503 whose body carries the same code. + */ +export function isHostAvailabilityError(err: unknown): boolean { + const text = stripIpcWrapper(errorText(err).slice(0, HTTP_STATUS_PARSE_MAX_CHARS)) + if (text.startsWith('host_waking:') || text.startsWith('host_draining:')) return true + return httpErrorStatus(err) === 503 && HOST_AVAILABILITY_BODY_PATTERN.test(text) +} + +/** True when the parsed response status is in the server-error range. */ +export function isHttpServerError(err: unknown): boolean { + const status = httpErrorStatus(err) + return status !== undefined && status >= 500 && status < 600 +} + +/** A transcript 404 is ambiguous: absence, authority and wake failure share it. */ export function isHttp404(err: unknown): boolean { - const value = errorText(err) + return httpErrorStatus(err) === 404 +} + +/** Only an explicit authorization status can revoke the visible chat scope. */ +export function isAuthorizationError(err: unknown): boolean { + const status = httpErrorStatus(err) + return status === 401 || status === 403 +} + +/** + * The main process refused a sessions cursor before issuing any request + * (`sanitizeSessionsListQuery` in `ipc.ts`). That refusal carries no HTTP + * status, only its fixed message behind the IPC wrapper. + */ +export function isInvalidSessionsCursorError(err: unknown): boolean { + return ( + httpErrorStatus(err) === undefined && + /(?:^|error: )invalid sessions cursor$/.test(errorText(err)) + ) +} + +export function isHttp403(err: unknown): boolean { + return httpErrorStatus(err) === 403 +} + +/** Exact Host-wide denial projected by the trusted Desktop RPC client. */ +export function isConfirmedHostAccessRevoked(err: unknown): boolean { return ( - /\b404\b/.test(value) || value.includes('chat not found') || value.includes('session not found') + httpErrorStatus(err) === 403 && + /(?:^|error: )403 forbidden: host_access_revoked$/.test(errorText(err)) ) } /** - * True when an error reflects a transport/connectivity failure rather than a - * server-side rejection. The D.4 reconcile stays in offline mode on these. + * True when a transcript read failed without an authority decision. The + * reconcile keeps the cached conversation while these errors recover. */ export function isNetworkError(err: unknown): boolean { const value = errorText(err) @@ -184,7 +261,8 @@ export function isNetworkError(err: unknown): boolean { value.includes('network') || value.includes('fetch failed') || value.includes('econnrefused') || - value.includes('failed to fetch') + value.includes('failed to fetch') || + [502, 503, 504].includes(httpErrorStatus(err) ?? 0) ) } diff --git a/desktop-app/ui/src/lib/gfsGrantErrors.ts b/desktop-app/ui/src/lib/gfsGrantErrors.ts index 9b5d88bada..ff602a9e13 100644 --- a/desktop-app/ui/src/lib/gfsGrantErrors.ts +++ b/desktop-app/ui/src/lib/gfsGrantErrors.ts @@ -36,8 +36,13 @@ const GFS_GRANT_ERROR_MESSAGES: Record = { * is an implementation detail of our own main/renderer split; the server * verdict that follows it is the part a user can act on. This strips exactly * the wrapper and leaves the verdict intact. + * + * The quoted channel is mandatory: Electron's renderer always formats the + * rejection as `Error invoking remote method '${channel}': ${error}`, so a + * prefix without it is not an IPC wrapper and must stay visible to the + * classifiers rather than be treated as one. */ -const IPC_WRAPPER_PREFIX = /^Error invoking remote method '[^']*':\s*(?:[A-Za-z]*Error:\s*)?/ +const IPC_WRAPPER_PREFIX = /^Error invoking remote method '[^']*':\s*(?:[A-Za-z]*Error:\s*)?/i export function stripIpcWrapper(message: string): string { return message.replace(IPC_WRAPPER_PREFIX, '') diff --git a/desktop-app/ui/src/lib/hostAuthorityStore.ts b/desktop-app/ui/src/lib/hostAuthorityStore.ts new file mode 100644 index 0000000000..45d9aef037 --- /dev/null +++ b/desktop-app/ui/src/lib/hostAuthorityStore.ts @@ -0,0 +1,67 @@ +/** + * Per-Host authority state for the renderer: which Hosts are held after an + * authorization failure, and an epoch per Host that changes whenever that + * Host's authority changes. + * + * Catalog and transcript reads snapshot `getEpoch(agentRef)` before an await + * and discard the result when it moved. The epoch is per Host on purpose: a + * revocation of Host Y must never invalidate an in-flight read of Host X, so + * an agent that never changed authority reads as epoch 0, never as a global + * counter. + * + * `useAppController` owns the production instance; the controller test + * harnesses build theirs from this same factory so the epoch path under test + * is the production one. + */ + +export type HostAuthorityHoldKind = 'revoked' | 'uncertain' + +export interface HostAuthorityStore { + isBlocked: (agentRef: string) => boolean + getEpoch: (agentRef: string) => number + /** The epoch at which the Host was held, or undefined when it is not held. */ + heldAtEpoch: (agentRef: string) => number | undefined + /** + * Hold the Host. Returns false when nothing changed: a confirmed revocation + * is never downgraded to an uncertain hold. + */ + hold: (agentRef: string, kind: HostAuthorityHoldKind) => boolean + /** + * Release a hold after a successful verification. Returns false when the + * hold changed since `heldAtEpoch` was read (the verification is stale). + */ + release: (agentRef: string, heldAtEpoch: number) => boolean + /** Forget every hold and epoch (a new authority scope). */ + reset: () => void +} + +export function createHostAuthorityStore(): HostAuthorityStore { + let epoch = 0 + const epochByAgent = new Map() + const holds = new Map() + + return { + isBlocked: agentRef => holds.has(agentRef), + getEpoch: agentRef => epochByAgent.get(agentRef) ?? 0, + heldAtEpoch: agentRef => holds.get(agentRef)?.epoch, + hold: (agentRef, kind) => { + if (holds.get(agentRef)?.kind === 'revoked' && kind === 'uncertain') return false + epoch += 1 + holds.set(agentRef, { kind, epoch }) + epochByAgent.set(agentRef, epoch) + return true + }, + release: (agentRef, heldAtEpoch) => { + if (holds.get(agentRef)?.epoch !== heldAtEpoch) return false + holds.delete(agentRef) + epoch += 1 + epochByAgent.set(agentRef, epoch) + return true + }, + reset: () => { + holds.clear() + epoch += 1 + epochByAgent.clear() + }, + } +} diff --git a/desktop-app/vitest.config.ts b/desktop-app/vitest.config.ts index f4901843e9..3d48864104 100644 --- a/desktop-app/vitest.config.ts +++ b/desktop-app/vitest.config.ts @@ -29,6 +29,9 @@ export default defineConfig({ // which serves an out-of-root setup file under `/@fs/` and cannot resolve // it from a path relative to this package. setupFiles: [path.resolve(__dirname, '../scripts/testing/bind-loopback-in-tests.mjs')], + // Owns the temporary root of the renderer chat fixture's real ChatStore + // directories and removes it once every test file has finished. + globalSetup: ['ui/src/hooks/domain/__tests__/__fixtures__/mockClerumStoreRoot.ts'], fileParallelism: false, teardownTimeout: 60_000, include: [ diff --git a/docs/testing/local-measurements.md b/docs/testing/local-measurements.md index 56b5a62369..6292ab95b0 100644 --- a/docs/testing/local-measurements.md +++ b/docs/testing/local-measurements.md @@ -37,11 +37,33 @@ and unsupported matrix shapes, and runs before `make test-unit-all` executes packages. Each package keeps its `npm test` semantics; real PostgreSQL tests remain opt-in and are not T1 evidence in this unit lane. +## AutoResearch hard-gate entrypoints + +Use `scripts/dev/autoresearch-benchmark.sh -- ` as the +AutoResearch benchmark command. It runs the measurement intake first, so a +missing or stale `origin/dev`, default detached HEAD, or merge conflict blocks +the benchmark before it starts. A passing intake emits +`AUTORESEARCH_GATE branch_freshness=pass`. + +Configure `make autoresearch-checks` as AutoResearch's `--checks-command`. +It runs the CI/Makefile service-matrix parity check and emits +`AUTORESEARCH_GATE service_matrix_parity=pass`. The Ralph adapter maps a +non-zero checks command to the packet's `tests_pass` hard gate, so service +drift cannot be kept. The captured gate lines make the two reasons explicit +in packet evidence. + +These entrypoints use only the current local checkout and locally recorded Git +refs. They do not fetch, push, create commits, change Minikube/Kubernetes +state, or publish AutoResearch session files; the ignored `autoresearch.*` +files remain local unless deliberately copied elsewhere. + CI runs these contract checks in the `shell-syntax` job; locally (no cluster required): ```bash node scripts/tests/run-node-test-files.mjs scripts/tests/test-test-services.cjs scripts/tests/test-measurement-base.cjs bash scripts/tests/test-repo-intake-packet.sh +bash scripts/tests/test-autoresearch-gates.sh make test-service-matrix +make autoresearch-checks ``` diff --git a/external-rest-api/src/releaseManifest.ts b/external-rest-api/src/releaseManifest.ts index 338f898a4b..f159b47b75 100644 --- a/external-rest-api/src/releaseManifest.ts +++ b/external-rest-api/src/releaseManifest.ts @@ -9,7 +9,7 @@ export type ReleaseManifest = { export const releaseManifest: ReleaseManifest = { releaseId: 'v0.10.0', externalRestApiVersion: '0.1.85', - rpcProxyVersion: '0.1.94', + rpcProxyVersion: '0.1.111', desktopVersion: '0.10.0', minimumDesktopVersion: '0.1.252', } diff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.json index 961897acbc..bb3c7fe0c4 100644 --- a/host-context-controller/package-lock.json +++ b/host-context-controller/package-lock.json @@ -1,12 +1,12 @@ { "name": "clerum-host-context-controller", - "version": "0.1.352", + "version": "0.1.375", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "clerum-host-context-controller", - "version": "0.1.352", + "version": "0.1.375", "license": "SEE LICENSE IN LICENSE", "dependencies": { "@clerum/codex-catalog-projection": "file:../packages/codex-catalog-projection", diff --git a/host-context-controller/package.json b/host-context-controller/package.json index e0a845ff60..dde8a1bec4 100644 --- a/host-context-controller/package.json +++ b/host-context-controller/package.json @@ -1,6 +1,6 @@ { "name": "clerum-host-context-controller", - "version": "0.1.352", + "version": "0.1.375", "description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API", "main": "dist/main.js", "engines": { diff --git a/host-context-controller/src/__tests__/hostReconciler.mounts.test.ts b/host-context-controller/src/__tests__/hostReconciler.mounts.test.ts index 7f6f24ed41..4865643b97 100644 --- a/host-context-controller/src/__tests__/hostReconciler.mounts.test.ts +++ b/host-context-controller/src/__tests__/hostReconciler.mounts.test.ts @@ -172,6 +172,8 @@ function runtimeSecretAnnotations(host = makeHost(), refreshBefore = '2999-01-01 'clerum.io/gfs-token-host-uid': host.uid!, 'clerum.io/runtime-token-refresh-before': refreshBefore, 'clerum.io/gfs-token-refresh-before': '2999-01-01T00:00:00.000Z', + 'clerum.io/runtime-token-has-channel-ingress': 'false', + 'clerum.io/runtime-token-fronts-oauth-server': 'false', } } @@ -233,6 +235,7 @@ function runtimeSecretReconciler(opts: { const reconciler = new HostReconciler(makeStubKc(), { coreApi: coreApi as unknown as k8s.CoreV1Api, appsApi: appsApi as unknown as k8s.AppsV1Api, + isCommunicationChannelCacheSynced: () => true, }) return { reconciler, coreApi, appsApi } } @@ -1256,6 +1259,7 @@ describe('HostReconciler.reconcile — uses resolveContextMounts', () => { const stubKc = makeStubKc() const reconciler = new HostReconciler(stubKc, { resolveContextMounts, + isCommunicationChannelCacheSynced: () => true, // Stub APIs to capture the Deployment body and ack everything. coreApi: { readNamespacedSecret: readSecretWithChannelReaderRuntimeAuthLabels(), diff --git a/host-context-controller/src/hostReconciler.codexScopeProvenance.test.ts b/host-context-controller/src/hostReconciler.codexScopeProvenance.test.ts index cde5f50353..37493ee32a 100644 --- a/host-context-controller/src/hostReconciler.codexScopeProvenance.test.ts +++ b/host-context-controller/src/hostReconciler.codexScopeProvenance.test.ts @@ -149,6 +149,7 @@ function createReconciler() { coreApi: asCoreApi(coreApi), networkingApi: asNetworkingApi(networkingApi), rbacApi: asRbacApi(rbacApi), + isCommunicationChannelCacheSynced: () => true, }) return { reconciler, appsApi, coreApi, networkingApi, rbacApi } } diff --git a/host-context-controller/src/hostReconciler.grokHostRuntime.test.ts b/host-context-controller/src/hostReconciler.grokHostRuntime.test.ts index f9a5de2727..9811f0da92 100644 --- a/host-context-controller/src/hostReconciler.grokHostRuntime.test.ts +++ b/host-context-controller/src/hostReconciler.grokHostRuntime.test.ts @@ -168,6 +168,7 @@ function createReconciler() { coreApi: asCoreApi(coreApi), networkingApi: asNetworkingApi(networkingApi), rbacApi: asRbacApi(rbacApi), + isCommunicationChannelCacheSynced: () => true, }) return { reconciler, appsApi, coreApi, networkingApi, rbacApi } } diff --git a/host-context-controller/src/hostReconciler.rbacPvcReadFirst.test.ts b/host-context-controller/src/hostReconciler.rbacPvcReadFirst.test.ts index 75a2c3d9a5..379060db4e 100644 --- a/host-context-controller/src/hostReconciler.rbacPvcReadFirst.test.ts +++ b/host-context-controller/src/hostReconciler.rbacPvcReadFirst.test.ts @@ -308,7 +308,7 @@ describe('Host read-first RBAC and PVC contracts', () => { }, spec: { volumeName: 'bound-volume' }, }) - await (f.reconciler as any).ensurePvc(host) + await expect((f.reconciler as any).ensurePvc(host)).resolves.toBe(true) expect(f.core.readNamespacedPersistentVolumeClaim).toHaveBeenCalledOnce() expect(f.core.createNamespacedPersistentVolumeClaim).not.toHaveBeenCalled() expect(f.core.replaceNamespacedPersistentVolumeClaim).not.toHaveBeenCalled() @@ -317,7 +317,7 @@ describe('Host read-first RBAC and PVC contracts', () => { it('updates an unbound Host PVC once with its observed resourceVersion', async () => { const f = fixture() - await (f.reconciler as any).ensurePvc(host) + await expect((f.reconciler as any).ensurePvc(host)).resolves.toBe(true) expect(f.core.readNamespacedPersistentVolumeClaim).toHaveBeenCalledOnce() expect(f.core.createNamespacedPersistentVolumeClaim).not.toHaveBeenCalled() expect(f.core.replaceNamespacedPersistentVolumeClaim).toHaveBeenCalledOnce() @@ -327,6 +327,28 @@ describe('Host read-first RBAC and PVC contracts', () => { expect(await count('PersistentVolumeClaim', 'skipped')).toBe(1) }) + it('counts a Host PVC create race as successful after fresh read convergence', async () => { + const f = fixture() + f.core.readNamespacedPersistentVolumeClaim + .mockRejectedValueOnce({ code: 404 }) + .mockResolvedValueOnce({ + metadata: { + name: 'rbac-host-workspace', + namespace: host.namespace, + uid: 'pvc-uid', + resourceVersion: '17', + }, + }) + f.core.createNamespacedPersistentVolumeClaim.mockRejectedValueOnce({ code: 409 }) + + await expect((f.reconciler as any).ensurePvc(host)).resolves.toBe(true) + + expect(f.core.readNamespacedPersistentVolumeClaim).toHaveBeenCalledTimes(2) + expect(f.core.createNamespacedPersistentVolumeClaim).toHaveBeenCalledOnce() + expect(f.core.replaceNamespacedPersistentVolumeClaim).toHaveBeenCalledOnce() + expect(await count('PersistentVolumeClaim', 'conflict')).toBe(1) + }) + it('propagates a Host PVC initial GET failure outside retained write catches', async () => { const f = fixture() const failure = { code: 403 } @@ -342,7 +364,7 @@ describe('Host read-first RBAC and PVC contracts', () => { const f = fixture() f.core.readNamespacedPersistentVolumeClaim.mockRejectedValueOnce({ code: 404 }) f.core.createNamespacedPersistentVolumeClaim.mockRejectedValueOnce({ code: 503 }) - await expect((f.reconciler as any).ensurePvc(host)).resolves.toBeUndefined() + await expect((f.reconciler as any).ensurePvc(host)).resolves.toBe(false) expect(f.core.readNamespacedPersistentVolumeClaim).toHaveBeenCalledOnce() expect(f.core.createNamespacedPersistentVolumeClaim).toHaveBeenCalledOnce() expect(f.core.replaceNamespacedPersistentVolumeClaim).not.toHaveBeenCalled() @@ -353,7 +375,7 @@ describe('Host read-first RBAC and PVC contracts', () => { it.each([409, 503])('retains Host PVC PUT%d without retrying or recording a skip', async code => { const f = fixture() f.core.replaceNamespacedPersistentVolumeClaim.mockRejectedValueOnce({ code }) - await expect((f.reconciler as any).ensurePvc(host)).resolves.toBeUndefined() + await expect((f.reconciler as any).ensurePvc(host)).resolves.toBe(false) expect(f.core.readNamespacedPersistentVolumeClaim).toHaveBeenCalledOnce() expect(f.core.replaceNamespacedPersistentVolumeClaim).toHaveBeenCalledOnce() expect(f.core.createNamespacedPersistentVolumeClaim).not.toHaveBeenCalled() diff --git a/host-context-controller/src/hostReconciler.readFirst.test.ts b/host-context-controller/src/hostReconciler.readFirst.test.ts index 01095d6654..0601f9730e 100644 --- a/host-context-controller/src/hostReconciler.readFirst.test.ts +++ b/host-context-controller/src/hostReconciler.readFirst.test.ts @@ -214,7 +214,7 @@ describe('Host read-first Service and Deployment contracts', () => { const { core, reconciler } = fixture() core.readNamespacedService.mockRejectedValueOnce({ code: 404 }) core.createNamespacedService.mockRejectedValueOnce({ code: 503 }) - await expect((reconciler as any).ensureService(host)).resolves.toBeUndefined() + await expect((reconciler as any).ensureService(host)).resolves.toBe(false) expect(core.readNamespacedService).toHaveBeenCalledOnce() expect(core.createNamespacedService).toHaveBeenCalledOnce() expect(core.replaceNamespacedService).not.toHaveBeenCalled() @@ -225,7 +225,7 @@ describe('Host read-first Service and Deployment contracts', () => { it('retains a Host Service convergence failure without recording a successful skip', async () => { const { core, reconciler } = fixture() core.replaceNamespacedService.mockRejectedValueOnce({ code: 503 }) - await expect((reconciler as any).ensureService(host)).resolves.toBeUndefined() + await expect((reconciler as any).ensureService(host)).resolves.toBe(false) expect(core.readNamespacedService).toHaveBeenCalledOnce() expect(core.createNamespacedService).not.toHaveBeenCalled() expect(core.replaceNamespacedService).toHaveBeenCalledOnce() @@ -238,7 +238,7 @@ describe('Host read-first Service and Deployment contracts', () => { const desired = (reconciler as any).buildService(host) as k8s.V1Service core.readNamespacedService.mockResolvedValue(asApiserverService(desired)) core.createNamespacedService.mockRejectedValue({ code: 409 }) - await (reconciler as any).ensureService(host) + await expect((reconciler as any).ensureService(host)).resolves.toBe(true) expect(core.readNamespacedService).toHaveBeenCalledOnce() expect(core.createNamespacedService).not.toHaveBeenCalled() expect(core.replaceNamespacedService).not.toHaveBeenCalled() @@ -325,7 +325,7 @@ describe('Host read-first Service and Deployment contracts', () => { const { core, reconciler } = fixture() core.readNamespacedService.mockRejectedValueOnce({ code: 404 }) core.createNamespacedService.mockRejectedValueOnce({ code: 409 }) - await (reconciler as any).ensureService(host) + await expect((reconciler as any).ensureService(host)).resolves.toBe(true) expect(core.readNamespacedService).toHaveBeenCalledTimes(2) expect(core.createNamespacedService).toHaveBeenCalledOnce() expect(core.replaceNamespacedService).toHaveBeenCalledOnce() diff --git a/host-context-controller/src/hostReconciler.test.ts b/host-context-controller/src/hostReconciler.test.ts index b185932a94..f7a2e31c91 100644 --- a/host-context-controller/src/hostReconciler.test.ts +++ b/host-context-controller/src/hostReconciler.test.ts @@ -1056,12 +1056,12 @@ describe('HostReconciler Host inventory mutation authority', () => { function stubPreDeploymentReconcileEffects(reconciler: HostReconciler): void { vi.spyOn(reconciler as any, 'validateHostSecret').mockResolvedValue({ ok: true }) + vi.spyOn(reconciler as any, 'ensurePvc').mockResolvedValue(true) + vi.spyOn(reconciler as any, 'ensureService').mockResolvedValue(true) for (const method of [ 'ensureHostServiceAccount', 'ensureHostRole', 'ensureHostRoleBinding', - 'ensurePvc', - 'ensureService', 'ensureMcpHostIngressNetworkPolicy', 'ensureMcpHostGfsEgressNetworkPolicy', 'ensureWorkflowApprovalReaderMcpHostIngressNetworkPolicy', @@ -1071,7 +1071,7 @@ describe('HostReconciler Host inventory mutation authority', () => { 'ensureChannelReaderEgressNetworkPolicy', 'ensureWorkflowApprovalReaderHostEgressNetworkPolicy', ]) { - vi.spyOn(reconciler as any, method).mockResolvedValue(undefined) + vi.spyOn(reconciler as any, method).mockResolvedValue('up_to_date') } } @@ -1420,9 +1420,7 @@ describe('HostReconciler Host inventory mutation authority', () => { await releaseTokenProvision.promise return { revision: 'stale-revision', scopeHash: 'stale-scope' } }) - const ensureDeployment = vi - .spyOn(reconciler as any, 'ensureDeployment') - .mockResolvedValue(undefined) + const ensureDeployment = vi.spyOn(reconciler as any, 'ensureDeployment').mockResolvedValue(true) const pending = reconciler.reconcile(host) await tokenProvisionStarted.promise diff --git a/host-context-controller/src/hostReconciler.ts b/host-context-controller/src/hostReconciler.ts index b84ae5337a..f262dd1cfd 100644 --- a/host-context-controller/src/hostReconciler.ts +++ b/host-context-controller/src/hostReconciler.ts @@ -78,6 +78,7 @@ import { HostWorkflowControlSpec, } from './types' import { + type ResourceApplyResult, applyNetworkPolicy, canonicalStringify, canonicalizeValue, @@ -382,6 +383,7 @@ const GFS_TOKEN_REFRESH_BEFORE_ANNOTATION = 'clerum.io/gfs-token-refresh-before' const GFS_TOKEN_EXPECTED_SUBJECT_ANNOTATION = 'clerum.io/gfs-token-expected-subject' const GFS_TOKEN_CAPABILITY_SET_HASH_ANNOTATION = 'clerum.io/gfs-token-capability-set-hash' const GFS_TOKEN_HOST_UID_ANNOTATION = 'clerum.io/gfs-token-host-uid' +const HOST_UID_ANNOTATION = 'clerum.io/host-uid' const GFS_TOKEN_HOST_GENERATION_ANNOTATION = 'clerum.io/gfs-token-host-generation' const RUNTIME_TOKEN_HOST_BINDING_HASH_ANNOTATION = 'clerum.io/runtime-token-host-binding-hash' const RUNTIME_TOKEN_SCOPE_HASH_ANNOTATION = 'clerum.io/runtime-token-scope-hash' @@ -390,6 +392,12 @@ const RUNTIME_TOKEN_AUDIENCE_ANNOTATION = 'clerum.io/runtime-token-audience' const RUNTIME_TOKEN_SCHEMA_VERSION_ANNOTATION = 'clerum.io/runtime-token-schema-version' const RUNTIME_TOKEN_BOOTSTRAP_STATE_ANNOTATION = 'clerum.io/runtime-token-bootstrap-state' const RUNTIME_TOKEN_ROLLOUT_REQUIRED_ANNOTATION = 'clerum.io/runtime-token-rollout-required' +const RUNTIME_TOKEN_BOOTSTRAP_DEPLOYMENT_UID_ANNOTATION = + 'clerum.io/runtime-token-bootstrap-deployment-uid' +const RUNTIME_TOKEN_BOOTSTRAP_APPLIED_REVISION_ANNOTATION = + 'clerum.io/runtime-token-bootstrap-applied-revision' +const RUNTIME_TOKEN_HAS_CHANNEL_INGRESS_ANNOTATION = 'clerum.io/runtime-token-has-channel-ingress' +const RUNTIME_TOKEN_FRONTS_OAUTH_ANNOTATION = 'clerum.io/runtime-token-fronts-oauth-server' const RUNTIME_TOKEN_ISSUER = 'control-api' const RUNTIME_TOKEN_AUDIENCE = 'host-context-controller,workflow-approvals' // v3 extends the existing first-party access/refresh token material with the @@ -496,6 +504,8 @@ type HostSecretValidationResult = type BootstrapOptions = { forceFreshForWake?: boolean targetSuspended?: boolean + refreshGfsOnly?: boolean + preserveDeploymentTemplateOnWake?: boolean } type RuntimeTokenProvision = { @@ -532,6 +542,8 @@ export class HostReconciler { private codexSnapshot: CodexCatalogSnapshot = { flagEnabled: false } private lastCodexConfigMap: k8s.V1ConfigMap | undefined private readonly readinessTimers: Map> = new Map() + private readonly reconcileGenerations = new Map() + private readonly statusGenerations = new Map() /** * host.name → image whose pull-policy refusal was already error-logged. * The periodic resync rebuilds the Deployment forever; the operator needs @@ -696,18 +708,27 @@ export class HostReconciler { /** * Whether this Host fronts an enabled `auth.type: oauth` mcp-server, gating - * the derive-only `oauth:user-token` runtime scope. Fails CLOSED (false) when - * the cross-CRD read throws: an uncertain read must never grant the scope. + * the derive-only `oauth:user-token` runtime scope. A read that failed after + * its retries is reported as unobserved instead of collapsing to `false`: + * "scope unknown" is not "scope revoked". Callers decide what an unknown + * answer means — the issuance path fails closed only where that cannot strip + * a grant the running pod holds, and the deployment drift guard + * (`currentScopeMatchesProvision`) never forces a re-mint on it. An uncertain + * read never grants the scope. * * Each call is an INDEPENDENT live point-in-time read (mirrors * `hasChannelIngress`); the value is deliberately not cached across a - * reconcile. Callers that need one consistent value within a unit of work - * resolve it once and thread the same bool (see the issuance path, which - * threads it into the mint-scope derive and the drift hash so they never - * diverge); the deployment drift guard re-reads it on purpose to catch a - * scope change mid-reconcile. + * reconcile. The issuance path resolves it once and threads the same bool + * into the mint-scope derive and the drift hash so they never diverge; the + * deployment drift guard re-reads it on purpose to catch a scope change + * mid-reconcile. */ - private async frontsOAuthServer(host: HostCRD): Promise { + private async observeFrontsOAuthServer( + host: HostCRD + ): Promise< + | { observed: true; frontsOAuthServer: boolean } + | { observed: false; attempts: number; error: unknown } + > { // Bounded retry that absorbs a TRANSIENT apiserver blip without touching the // authoritative-`false` path. Contract of the underlying resolver: a real // scope change (the server genuinely stopped fronting an oauth mcp-server) @@ -724,7 +745,7 @@ export class HostReconciler { let lastErr: unknown for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) { try { - return await this.hostFrontsOAuthServerFn(host) + return { observed: true, frontsOAuthServer: await this.hostFrontsOAuthServerFn(host) } } catch (err) { lastErr = err if (attempt < MAX_ATTEMPTS) { @@ -733,21 +754,7 @@ export class HostReconciler { } } } - // All attempts threw. Residual honesty: a SUSTAINED apiserver outage (every - // attempt throws) still fails CLOSED here and can flip the scope off. That is - // correct — a sustained outage is a genuine "scope unknown" situation, not a - // blip — the retry only smooths over transient blips, never a real outage. - log.warn( - 'failed to resolve whether Host fronts an oauth mcp-server after retries; failing closed (no oauth:user-token scope)', - { - host: host.name, - namespace: host.namespace, - contextRef: host.spec.contextRef, - attempts: MAX_ATTEMPTS, - err: lastErr instanceof Error ? lastErr.message : String(lastErr), - } - ) - return false + return { observed: false, attempts: MAX_ATTEMPTS, error: lastErr } } /** @@ -984,7 +991,7 @@ export class HostReconciler { } } - private isHccOwnedHostResource( + private static isHccOwnedHostResource( resource: { metadata?: { labels?: Record } }, hostName: string ): boolean { @@ -1009,7 +1016,7 @@ export class HostReconciler { throw error } - if (!this.isHccOwnedHostResource(resource, hostName)) { + if (!HostReconciler.isHccOwnedHostResource(resource, hostName)) { log.warn('Skipping Host resource delete - not HCC-owned', { kind, name, host: hostName }) return } @@ -1554,6 +1561,81 @@ export class HostReconciler { return (deployment?.status?.readyReplicas ?? 0) > 0 } + /** + * The binding covers the Deployment UID and its runtime-token-revision + * annotation only. It does not bind the rest of the pod template, so a + * template edit that keeps both values still matches. + * + * This is deliberate: there is no pod-template hash. The record and the + * Deployment belong to the same Host, and any principal that can edit the + * pod template can already read the runtime-token Secret, so a template hash + * would add no isolation. A hash would also have to be computed on the + * apiserver-defaulted object, which never equals the applied body, so it + * would fail to match and re-mint the credentials on every reconcile. + */ + private static bootstrapBindingMatchesDeployment( + record: Pick, + deployment: k8s.V1Deployment | null + ): boolean { + const deploymentIdentity = deployment?.metadata?.uid + const boundDeploymentIdentity = + record.metadata?.annotations?.[RUNTIME_TOKEN_BOOTSTRAP_DEPLOYMENT_UID_ANNOTATION] + const appliedRevision = HostReconciler.deploymentRuntimeTokenRevision(deployment) + const boundAppliedRevision = + record.metadata?.annotations?.[RUNTIME_TOKEN_BOOTSTRAP_APPLIED_REVISION_ANNOTATION] + return ( + !!deploymentIdentity && + boundDeploymentIdentity === deploymentIdentity && + appliedRevision !== '' && + boundAppliedRevision === appliedRevision + ) + } + + private static credentialRecordBelongsToHost( + record: { metadata?: { labels?: Record } }, + host: HostCRD + ): boolean { + return HostReconciler.isHccOwnedHostResource(record, host.name) + } + + private static runtimeRecordIsComplete( + record: Pick + ): boolean { + return HostReconciler.runtimeTokenSecretData(record) !== null + } + + private static retainedBoolean(record: k8s.V1Secret, key: string): boolean | null { + const value = record.metadata?.annotations?.[key] + return value === 'true' ? true : value === 'false' ? false : null + } + + private static deploymentBelongsToHost( + deployment: k8s.V1Deployment | null, + host: HostCRD + ): boolean { + return ( + !!deployment && + HostReconciler.credentialRecordBelongsToHost(deployment, host) && + deployment.metadata?.name === host.name && + deployment.metadata?.namespace === host.namespace && + !!deployment.metadata?.uid && + !!deployment.metadata?.resourceVersion && + !deployment.metadata?.deletionTimestamp && + !!deployment.spec?.template?.spec && + !!host.uid && + deployment.metadata?.annotations?.[HOST_UID_ANNOTATION] === host.uid + ) + } + + private static credentialRenewalWindowReached( + record: k8s.V1Secret, + nowMs: number, + annotation: string + ): boolean { + const refreshBefore = Date.parse(record.metadata?.annotations?.[annotation] ?? '') + return !Number.isFinite(refreshBefore) || nowMs >= refreshBefore + } + /** * True when this reconcile will (re)start a pod that reads the * mcp-host-runtime-token Secret fresh at boot -- a missing Deployment, a @@ -1570,11 +1652,16 @@ export class HostReconciler { * readiness 503s forever, and the pod never becomes Ready. Reuse is only * safe for a pod that stays up and never re-reads the Secret (it uses its * in-memory rotated tokens), which is exactly deploymentReady && replicas>=1. + * + * A wake bootstrap already bound to this exact Deployment UID and applied + * runtime revision is the one startup exception: its replica-only wake starts + * that Deployment with the newly mounted material. Any identity or revision + * change must mint again. */ private static deploymentNeedsFreshBootstrap( deployment: k8s.V1Deployment | null, currentRevision: string, - bootstrapIsFresh: boolean, + credentialRecord: k8s.V1Secret | null, options: BootstrapOptions ): boolean { if (options.forceFreshForWake || !deployment) return true @@ -1583,8 +1670,16 @@ export class HostReconciler { if (options.targetSuspended === true) return false if ((deployment.spec?.replicas ?? 1) === 0) return true if (HostReconciler.deploymentReady(deployment)) return false + if ( + credentialRecord?.metadata?.annotations?.[RUNTIME_TOKEN_BOOTSTRAP_STATE_ANNOTATION] === + RUNTIME_TOKEN_BOOTSTRAP_STATE_FRESH && + HostReconciler.bootstrapBindingMatchesDeployment(credentialRecord, deployment) + ) { + return false + } return ( - !bootstrapIsFresh || + credentialRecord?.metadata?.annotations?.[RUNTIME_TOKEN_BOOTSTRAP_STATE_ANNOTATION] !== + RUNTIME_TOKEN_BOOTSTRAP_STATE_FRESH || HostReconciler.deploymentRuntimeTokenRevision(deployment) !== currentRevision ) } @@ -1710,6 +1805,28 @@ export class HostReconciler { ) } + /** + * Deployment drift guard: does the provisioned credential still match the + * Host's current scope contract? An OAuth read that failed after its retries + * is "scope unknown", so it matches a provision minted under either OAuth + * answer instead of collapsing to `false` and forcing a re-mint that would + * strip oauth:user-token and roll the Deployment. The issuance path owns the + * decision to narrow the grant (see `ensureMcpHostRuntimeTokenSecret`). + */ + private currentScopeMatchesProvision( + host: HostCRD, + provision: RuntimeTokenProvision, + live: Awaited> + ): boolean { + if (live.observed) { + return provision.scopeHash === this.runtimeScopeHashFor(host, live.frontsOAuthServer) + } + return ( + provision.scopeHash === this.runtimeScopeHashFor(host, true) || + provision.scopeHash === this.runtimeScopeHashFor(host, false) + ) + } + private async refreshCodexSnapshot(): Promise { try { const cm = await this.coreApi.readNamespacedConfigMap({ @@ -1755,9 +1872,13 @@ export class HostReconciler { private async ensureMcpHostRuntimeTokenSecret( host: HostCRD, options: BootstrapOptions = {} - ): Promise { + ): Promise { let lastErr: unknown = null - for (let attempt = 1; attempt <= 3; attempt++) { + // The held GFS-only renewal runs inside a reconcile that keeps a Ready + // runtime in place; it gets one attempt so a failing renewal never blocks + // that reconcile on backoff. The next hold pass retries it. + const maxAttempts = options.refreshGfsOnly ? 1 : 3 + for (let attempt = 1; attempt <= maxAttempts; attempt++) { try { const expectedGfsSubject = makeExpectedHostGfsSubject(host.namespace, host.name) if (!expectedGfsSubject) { @@ -1776,89 +1897,313 @@ export class HostReconciler { const existingRevision = existing ? HostReconciler.runtimeTokenSecretRevisionFromSecret(existing) : null + const nowMs = Date.now() + const preserveWakeTemplate = + options.preserveDeploymentTemplateOnWake === true && + !!deployment && + HostReconciler.deploymentBelongsToHost(deployment, host) && + (deployment.spec?.replicas ?? 1) === 0 && + HostReconciler.deploymentRuntimeTokenRevision(deployment) !== '' + if (options.refreshGfsOnly) { + const trustedDeployment = + HostReconciler.deploymentBelongsToHost(deployment, host) && + (deployment?.spec?.replicas ?? 0) > 0 + if ( + !existing || + !HostReconciler.credentialRecordBelongsToHost(existing, host) || + !host.uid || + existing.metadata?.annotations?.[GFS_TOKEN_HOST_UID_ANNOTATION] !== host.uid || + !HostReconciler.runtimeRecordIsComplete(existing) || + !trustedDeployment + ) { + log.warn( + 'deferring held-runtime credential renewal because runtime identity is not verified', + { + host: host.name, + namespace: host.namespace, + } + ) + return null + } + // A fresh bootstrap is consumed here when it is bound to this + // Deployment (a wake bootstrap) or when a Ready Deployment already + // runs its revision (a normal mint whose pod booted before channel + // authority was lost). The pod rotates the single-use refresh token + // it booted with, so leaving the record fresh would later roll a new + // pod onto that revoked token. The pass returns after the one write; + // GFS renewal follows on the next hold pass. + if ( + existing.metadata?.annotations?.[RUNTIME_TOKEN_BOOTSTRAP_STATE_ANNOTATION] === + RUNTIME_TOKEN_BOOTSTRAP_STATE_FRESH && + (HostReconciler.bootstrapBindingMatchesDeployment(existing, deployment) || + (HostReconciler.deploymentReady(deployment) && + HostReconciler.deploymentRuntimeTokenRevision(deployment) === existingRevision)) + ) { + const appliedRevision = HostReconciler.deploymentRuntimeTokenRevision(deployment) + if (HostReconciler.deploymentReady(deployment)) { + await this.coreApi.replaceNamespacedSecret({ + name, + namespace: host.namespace, + body: { + ...existing, + metadata: { + ...existing.metadata, + annotations: { + ...(existing.metadata?.annotations ?? {}), + [RUNTIME_TOKEN_BOOTSTRAP_STATE_ANNOTATION]: + RUNTIME_TOKEN_BOOTSTRAP_STATE_CONSUMED, + [RUNTIME_TOKEN_ROLLOUT_REQUIRED_ANNOTATION]: 'false', + }, + }, + }, + }) + } + return { + revision: appliedRevision || existingRevision || '', + scopeHash: + existing.metadata?.annotations?.[RUNTIME_TOKEN_SCOPE_HASH_ANNOTATION] ?? '', + } + } + if (!HostReconciler.deploymentReady(deployment)) return null + if ( + !HostReconciler.credentialRenewalWindowReached( + existing, + nowMs, + GFS_TOKEN_REFRESH_BEFORE_ANNOTATION + ) + ) { + this.recordGfsLifecycleEvidence(host, 'reused') + return { + revision: + HostReconciler.deploymentRuntimeTokenRevision(deployment) || existingRevision || '', + scopeHash: + existing.metadata?.annotations?.[RUNTIME_TOKEN_SCOPE_HASH_ANNOTATION] ?? '', + } + } + + const gfs = await mintHostGfsToken({ name: host.name, namespace: host.namespace }) + const data = { ...(HostReconciler.runtimeTokenSecretData(existing) ?? {}) } + data[MCP_HOST_GFS_TOKEN_SECRET_KEY] = Buffer.from(gfs.token).toString('base64') + const gfsExpiresAtMs = nowMs + Math.max(0, gfs.expiresInSeconds) * 1000 + const gfsRefreshBeforeSec = HostReconciler.effectiveBootstrapRefreshBeforeSec( + Math.max(0, gfs.expiresInSeconds) + ) + const nextRevision = HostReconciler.runtimeTokenSecretRevision(data) + await this.coreApi.replaceNamespacedSecret({ + name, + namespace: host.namespace, + body: { + ...existing, + data, + metadata: { + ...existing.metadata, + annotations: { + ...(existing.metadata?.annotations ?? {}), + [GFS_TOKEN_EXPECTED_SUBJECT_ANNOTATION]: + makeExpectedHostGfsSubject(host.namespace, host.name) ?? '', + [GFS_TOKEN_CAPABILITY_SET_HASH_ANNOTATION]: HostReconciler.gfsCapabilitySetHash(), + [GFS_TOKEN_EXPIRES_AT_ANNOTATION]: new Date(gfsExpiresAtMs).toISOString(), + [GFS_TOKEN_REFRESH_BEFORE_ANNOTATION]: new Date( + gfsExpiresAtMs - gfsRefreshBeforeSec * 1000 + ).toISOString(), + [RUNTIME_TOKEN_SECRET_REVISION_ANNOTATION]: nextRevision, + }, + }, + }, + }) + this.recordGfsLifecycleEvidence(host, 'rotated') + return { + revision: + HostReconciler.deploymentRuntimeTokenRevision(deployment) || existingRevision || '', + scopeHash: existing.metadata?.annotations?.[RUNTIME_TOKEN_SCOPE_HASH_ANNOTATION] ?? '', + } + } const bootstrapIsFresh = existing?.metadata?.annotations?.[RUNTIME_TOKEN_BOOTSTRAP_STATE_ANNOTATION] === RUNTIME_TOKEN_BOOTSTRAP_STATE_FRESH - const nowMs = Date.now() - const hasChannelIngress = this.hasChannelIngress(host) - // Resolve ONCE per issuance and thread the SAME frontsOAuthServer bool and - // codex projection into the mint-scope derive, the refresh decision, the - // scope hash and the stored annotation, so the minted token and the drift - // hash never diverge. - const frontsOAuthServer = await this.frontsOAuthServer(host) + const retainedAnnotations = existing?.metadata?.annotations ?? {} + const cacheSynced = this.ccCacheSyncedFn() + const retainedScopeIsTrusted = + !!existing && + HostReconciler.credentialRecordBelongsToHost(existing, host) && + !!host.uid && + retainedAnnotations[GFS_TOKEN_HOST_UID_ANNOTATION] === host.uid + const retainedChannelIngress = + retainedScopeIsTrusted && existing + ? HostReconciler.retainedBoolean(existing, RUNTIME_TOKEN_HAS_CHANNEL_INGRESS_ANNOTATION) + : null + const retainedFrontsOAuth = + retainedScopeIsTrusted && existing + ? HostReconciler.retainedBoolean(existing, RUNTIME_TOKEN_FRONTS_OAUTH_ANNOTATION) + : null + const hasChannelIngress = cacheSynced + ? this.hasChannelIngress(host) + : retainedChannelIngress === true + if (!cacheSynced && retainedChannelIngress === null) { + log.warn( + 'skipping runtime token mint during channel cache loss without retained scope observation', + { + host: host.name, + namespace: host.namespace, + observation: 'hasChannelIngress', + } + ) + return null + } const projection = this.projectCodexForHost(host) const grokProjection = this.projectGrokForHost(host) - const scopeHash = HostReconciler.runtimeTokenScopeHash( - host, - hasChannelIngress, - frontsOAuthServer, - projection, - grokProjection - ) - let decision: { + const decideRefresh = ( + frontsOAuth: boolean + ): { refresh: boolean rolloutRequired: boolean reason: string refreshTokenExpMs?: number - } = existing - ? HostReconciler.runtimeTokenRefreshDecision( - host, + } => { + let decided: { + refresh: boolean + rolloutRequired: boolean + reason: string + refreshTokenExpMs?: number + } = existing + ? HostReconciler.runtimeTokenRefreshDecision( + host, + existing, + nowMs, + hasChannelIngress, + frontsOAuth, + projection, + grokProjection + ) + : { refresh: true, rolloutRequired: false, reason: 'missing_secret' } + if ( + existing && + HostReconciler.deploymentRestartedAfterRuntimeSecretIssued(existing, deployment) + ) { + // A manual restart after the runtime Secret was issued starts pods with + // env-var bootstrap tokens from that older Secret. Force a re-issue and + // rollout so every restarted pod observes a fresh, unconsumed token pair. + decided = { + refresh: true, + rolloutRequired: true, + reason: 'deployment_restarted_after_secret_issued', + } + } + + // Revoked-on-wake guard: the mcp-host runtime refresh token is + // single-use-rotating (control-api revokes the prior JTI on every + // refresh). A pod that is about to boot -- wake (scale 0->1), missing + // Deployment, or a not-Ready (re)starting pod -- reads its bootstrap + // refresh token from THIS Secret. If the pre-suspend pod already + // rotated, the copy in the Secret is REVOKED, so an exp-based "reuse" + // hands the booting pod a dead token and readiness 503s forever. When + // a new pod will consume the Secret, force a fresh mint + rollout even + // though the decoded exp is far out. Reuse stays correct only for a + // running, Ready pod that never re-reads the Secret (churn fix). + if ( + existingRevision && + decided.refresh === false && + HostReconciler.deploymentNeedsFreshBootstrap( + deployment, + existingRevision, existing, - nowMs, - hasChannelIngress, - frontsOAuthServer, - projection, - grokProjection + options ) - : { refresh: true, rolloutRequired: false, reason: 'missing_secret' } - if ( - existing && - HostReconciler.deploymentRestartedAfterRuntimeSecretIssued(existing, deployment) - ) { - // A manual restart after the runtime Secret was issued starts pods with - // env-var bootstrap tokens from that older Secret. Force a re-issue and - // rollout so every restarted pod observes a fresh, unconsumed token pair. - decision = { - refresh: true, - rolloutRequired: true, - reason: 'deployment_restarted_after_secret_issued', + ) { + decided = { + refresh: true, + rolloutRequired: true, + reason: 'fresh_mint_for_booting_pod', + ...(decided.refreshTokenExpMs !== undefined + ? { refreshTokenExpMs: decided.refreshTokenExpMs } + : {}), + } } + return decided } - // Revoked-on-wake guard: the mcp-host runtime refresh token is - // single-use-rotating (control-api revokes the prior JTI on every - // refresh). A pod that is about to boot -- wake (scale 0->1), missing - // Deployment, or a not-Ready (re)starting pod -- reads its bootstrap - // refresh token from THIS Secret. If the pre-suspend pod already - // rotated, the copy in the Secret is REVOKED, so an exp-based "reuse" - // hands the booting pod a dead token and readiness 503s forever. When - // a new pod will consume the Secret, force a fresh mint + rollout even - // though the decoded exp is far out. Reuse stays correct only for a - // running, Ready pod that never re-reads the Secret (churn fix). - if ( - existingRevision && - decision.refresh === false && - HostReconciler.deploymentNeedsFreshBootstrap( - deployment, - existingRevision, - bootstrapIsFresh, - options - ) - ) { - decision = { - refresh: true, - rolloutRequired: true, - reason: 'fresh_mint_for_booting_pod', - ...(decision.refreshTokenExpMs !== undefined - ? { refreshTokenExpMs: decision.refreshTokenExpMs } - : {}), + // Resolve ONCE per issuance and thread the SAME frontsOAuthServer bool and + // codex projection into the mint-scope derive, the refresh decision, the + // scope hash and the stored annotation, so the minted token and the drift + // hash never diverge. + // + // The OAuth observation reads the McpServer cache, which does not depend + // on the CommunicationChannel cache, so it stays live during channel + // cache loss: an admin who removed the OAuth mcp-server while the channel + // watch was down must not have `oauth:user-token` re-granted from the + // retained annotation on a held wake. + // + // A live read that failed after its retries is "scope unknown", not + // "scope revoked". It resolves to `false` (fail closed) only when that + // cannot narrow a grant the running pod holds: the retained value is + // `false`, or there is no trusted retained value while channel authority + // is synced. A retained `true` under synced authority defers the decision + // while the credential needs no renewal, so an McpServer relist does not + // strip oauth:user-token and roll the Deployment; once the credential must + // be renewed, the renewal mints without the unobserved scope. Under channel + // cache loss the mint is skipped instead. + const liveFrontsOAuth = await this.observeFrontsOAuthServer(host) + let observedFrontsOAuth: boolean + if (liveFrontsOAuth.observed) { + observedFrontsOAuth = liveFrontsOAuth.frontsOAuthServer + } else { + const observationFailure = { + host: host.name, + namespace: host.namespace, + observation: 'frontsOAuthServer', + attempts: liveFrontsOAuth.attempts, + err: + liveFrontsOAuth.error instanceof Error + ? liveFrontsOAuth.error.message + : String(liveFrontsOAuth.error), + } + if (retainedFrontsOAuth === false || (cacheSynced && retainedFrontsOAuth !== true)) { + log.warn( + 'failed to resolve whether Host fronts an oauth mcp-server after retries; failing closed (no oauth:user-token scope)', + observationFailure + ) + observedFrontsOAuth = false + } else if (cacheSynced) { + if (!decideRefresh(true).refresh) { + log.warn( + 'deferring runtime token decision: OAuth observation unavailable and retained scope grants oauth:user-token', + observationFailure + ) + return null + } + log.warn( + 'renewing runtime token without oauth:user-token: OAuth observation unavailable when the credential must be renewed', + observationFailure + ) + observedFrontsOAuth = false + } else { + log.warn( + 'skipping runtime token mint during channel cache loss without an authoritative OAuth observation', + observationFailure + ) + return null } } + const scopeHash = HostReconciler.runtimeTokenScopeHash( + host, + hasChannelIngress, + observedFrontsOAuth, + projection, + grokProjection + ) + const decision = decideRefresh(observedFrontsOAuth) if (existing && existingRevision && !decision.refresh) { const deploymentRevision = HostReconciler.deploymentRuntimeTokenRevision(deployment) const rolloutMarker = existing.metadata?.annotations?.[RUNTIME_TOKEN_ROLLOUT_REQUIRED_ANNOTATION] + const bootstrapBoundToAppliedDeployment = + HostReconciler.bootstrapBindingMatchesDeployment(existing, deployment) + const bootstrapPendingForAppliedDeployment = + bootstrapIsFresh && bootstrapBoundToAppliedDeployment + const bootstrapConsumedForAppliedDeployment = + existing.metadata?.annotations?.[RUNTIME_TOKEN_BOOTSTRAP_STATE_ANNOTATION] === + RUNTIME_TOKEN_BOOTSTRAP_STATE_CONSUMED && bootstrapBoundToAppliedDeployment const rolloutPending = rolloutMarker === 'true' || (rolloutMarker === undefined && @@ -1866,6 +2211,19 @@ export class HostReconciler { deploymentRevision !== '' && deploymentRevision !== existingRevision) const annotationUpdates: Record = {} + if (cacheSynced) { + const currentIngress = + existing.metadata?.annotations?.[RUNTIME_TOKEN_HAS_CHANNEL_INGRESS_ANNOTATION] + if (currentIngress !== String(hasChannelIngress)) { + annotationUpdates[RUNTIME_TOKEN_HAS_CHANNEL_INGRESS_ANNOTATION] = + String(hasChannelIngress) + } + const currentOAuth = + existing.metadata?.annotations?.[RUNTIME_TOKEN_FRONTS_OAUTH_ANNOTATION] + if (currentOAuth !== String(observedFrontsOAuth)) { + annotationUpdates[RUNTIME_TOKEN_FRONTS_OAUTH_ANNOTATION] = String(observedFrontsOAuth) + } + } if ( host.generation !== undefined && existing.metadata?.annotations?.[GFS_TOKEN_HOST_GENERATION_ANNOTATION] !== @@ -1873,6 +2231,11 @@ export class HostReconciler { ) { annotationUpdates[GFS_TOKEN_HOST_GENERATION_ANNOTATION] = String(host.generation) } + if (bootstrapPendingForAppliedDeployment && HostReconciler.deploymentReady(deployment)) { + annotationUpdates[RUNTIME_TOKEN_BOOTSTRAP_STATE_ANNOTATION] = + RUNTIME_TOKEN_BOOTSTRAP_STATE_CONSUMED + annotationUpdates[RUNTIME_TOKEN_ROLLOUT_REQUIRED_ANNOTATION] = 'false' + } if ( bootstrapIsFresh && HostReconciler.deploymentReady(deployment) && @@ -1910,7 +2273,10 @@ export class HostReconciler { }) this.recordGfsLifecycleEvidence(host, 'reused') const selectedRevision = - rolloutPending || HostReconciler.shouldRollForRuntimeSecret(deployment, false) + rolloutPending || + (!bootstrapPendingForAppliedDeployment && + !bootstrapConsumedForAppliedDeployment && + HostReconciler.shouldRollForRuntimeSecret(deployment, false)) ? existingRevision : deploymentRevision || existingRevision return { @@ -1946,7 +2312,7 @@ export class HostReconciler { const tokens = await issueMcpHostRuntimeTokens( host.name, host.uid ?? '', - this.resolveEffectiveControlScopesForHost(host, frontsOAuthServer, hasChannelIngress) + this.resolveEffectiveControlScopesForHost(host, observedFrontsOAuth, hasChannelIngress) ) const gfs = await mintHostGfsToken({ name: host.name, namespace: host.namespace }) const body = buildMcpHostRuntimeTokenSecret( @@ -1961,8 +2327,9 @@ export class HostReconciler { throw new Error('mcp-host-runtime-token Secret body missing required credential keys') } const rolloutRequired = - !existing || - HostReconciler.shouldRollForRuntimeSecret(deployment, decision.rolloutRequired) + !preserveWakeTemplate && + (!existing || + HostReconciler.shouldRollForRuntimeSecret(deployment, decision.rolloutRequired)) body.metadata = { ...body.metadata, annotations: { @@ -1974,13 +2341,23 @@ export class HostReconciler { nowMs, gfs.expiresInSeconds, hasChannelIngress, - frontsOAuthServer, + observedFrontsOAuth, existing?.metadata?.annotations?.[GFS_TOKEN_HOST_UID_ANNOTATION], projection, grokProjection ), [RUNTIME_TOKEN_BOOTSTRAP_STATE_ANNOTATION]: RUNTIME_TOKEN_BOOTSTRAP_STATE_FRESH, [RUNTIME_TOKEN_ROLLOUT_REQUIRED_ANNOTATION]: rolloutRequired ? 'true' : 'false', + ...(preserveWakeTemplate && deployment + ? { + [RUNTIME_TOKEN_BOOTSTRAP_DEPLOYMENT_UID_ANNOTATION]: + deployment.metadata?.uid ?? '', + [RUNTIME_TOKEN_BOOTSTRAP_APPLIED_REVISION_ANNOTATION]: + HostReconciler.deploymentRuntimeTokenRevision(deployment), + } + : {}), + [RUNTIME_TOKEN_HAS_CHANNEL_INGRESS_ANNOTATION]: String(hasChannelIngress), + [RUNTIME_TOKEN_FRONTS_OAUTH_ANNOTATION]: String(observedFrontsOAuth), }, } @@ -1995,7 +2372,12 @@ export class HostReconciler { resourceName: name, reason: decision.reason, }) - return { revision, scopeHash } + return { + revision: preserveWakeTemplate + ? HostReconciler.deploymentRuntimeTokenRevision(deployment) || revision + : revision, + scopeHash, + } } const replaceBody = { @@ -2019,9 +2401,11 @@ export class HostReconciler { rolloutRequired, }) return { - revision: rolloutRequired - ? revision - : HostReconciler.deploymentRuntimeTokenRevision(deployment) || revision, + revision: preserveWakeTemplate + ? HostReconciler.deploymentRuntimeTokenRevision(deployment) || revision + : rolloutRequired + ? revision + : HostReconciler.deploymentRuntimeTokenRevision(deployment) || revision, scopeHash, } } catch (err) { @@ -2035,23 +2419,23 @@ export class HostReconciler { host: host.name, namespace: host.namespace, attempt, - maxAttempts: 3, + maxAttempts, error: String(err), }) - if (attempt < 3) { + if (attempt < maxAttempts) { await new Promise(resolve => setTimeout(resolve, delayMs)) } } } throw new Error( - `Failed to ensure mcp-host-runtime-token Secret for host "${host.name}" after 3 attempts: ${String(lastErr)}` + `Failed to ensure mcp-host-runtime-token Secret for host "${host.name}" after ${maxAttempts} attempt${maxAttempts === 1 ? '' : 's'}: ${String(lastErr)}` ) } private async provisionRuntimeTokenRevision( host: HostCRD, options: BootstrapOptions - ): Promise { + ): Promise { try { return await this.ensureMcpHostRuntimeTokenSecret(host, options) } catch (err) { @@ -2181,10 +2565,19 @@ export class HostReconciler { private setStatus(name: string, status: HostRuntimeStatus): void { this.statusMap.set(name, status) + this.statusGenerations.set(name, this.reconcileGenerations.get(name) ?? 0) + } + + private advanceReconcileGeneration(name: string): number { + const generation = (this.reconcileGenerations.get(name) ?? 0) + 1 + this.reconcileGenerations.set(name, generation) + return generation } private clearStatus(name: string): void { this.statusMap.delete(name) + this.reconcileGenerations.delete(name) + this.statusGenerations.delete(name) this.lifecycle.clearHost(name) const timer = this.readinessTimers.get(name) if (timer) { @@ -3182,6 +3575,7 @@ export class HostReconciler { name: host.name, namespace: host.namespace, labels, + ...(host.uid ? { annotations: { [HOST_UID_ANNOTATION]: host.uid } } : {}), }, spec: { // A suspended stateless Host scales to 0 on EVERY reconcile path @@ -3474,6 +3868,7 @@ export class HostReconciler { intervalMs = 5000, maxAttempts = 12 ): void { + const pollGeneration = this.reconcileGenerations.get(name) ?? 0 const existing = this.readinessTimers.get(name) if (existing) { clearTimeout(existing) @@ -3486,6 +3881,13 @@ export class HostReconciler { attempts++ const ready = await this.checkDeploymentReady(name, namespace) if (ready) { + if ( + (this.reconcileGenerations.get(name) ?? 0) !== pollGeneration || + (this.statusGenerations.get(name) ?? 0) > pollGeneration + ) { + this.readinessTimers.delete(name) + return + } this.setStatus(name, { deployed: true, ready: true, message: 'Running' }) this.readinessTimers.delete(name) return @@ -3502,16 +3904,17 @@ export class HostReconciler { this.readinessTimers.set(name, timer) } - private async ensurePvc(host: HostCRD, revalidate?: () => void): Promise { + private async ensurePvc(host: HostCRD, revalidate?: () => void): Promise { const pvc = this.buildPvc(host) const name = this.pvcName(host) + let createFailed = false const mutationAllowed = () => { revalidate?.() return true } // Initial GET errors must propagate; retain only the established // non-throwing POST and convergence failures of this PVC writer. - await ensureResource({ + const result = await ensureResource({ mutationAllowed, read: () => observeExistenceRead('PersistentVolumeClaim', () => @@ -3527,6 +3930,7 @@ export class HostReconciler { ) } catch (error) { if (error != null && getErrorCode(error) === 409) throw error + createFailed = true log.error('Failed to create Host PVC', { host: host.name, err: error }) } }, @@ -3549,17 +3953,19 @@ export class HostReconciler { }, onSkipped: () => createsTotal.inc({ kind: 'PersistentVolumeClaim', outcome: 'skipped' }), }) + return !createFailed && this.resourceApplySucceeded(result) } - private async ensureService(host: HostCRD, revalidate?: () => void): Promise { + private async ensureService(host: HostCRD, revalidate?: () => void): Promise { const service = this.buildService(host) + let createFailed = false const mutationAllowed = () => { revalidate?.() return true } // Initial read errors propagate. Only the existing POST/convergence paths // retain their historical non-throwing failures. - await ensureResource({ + const result = await ensureResource({ mutationAllowed, read: () => observeExistenceRead('Service', () => @@ -3572,6 +3978,7 @@ export class HostReconciler { ) } catch (error) { if (error != null && getErrorCode(error) === 409) throw error + createFailed = true log.error('Failed to create Host Service', { host: host.name, err: error }) } }, @@ -3600,23 +4007,61 @@ export class HostReconciler { } }, }) + return !createFailed && this.resourceApplySucceeded(result) + } + + private resourceApplySucceeded(result: ResourceApplyResult): boolean { + return result === 'created' || result === 'replaced' || result === 'up_to_date' } private async ensureDeployment( host: HostCRD, mounts: ResolvedSfsMount[], - runtimeTokenRevision: string, + runtimeTokenRevision: string | undefined, lifecycle?: EffectiveHostLifecycle, resolveStateBeforeMutation?: () => Promise, revalidate?: () => void - ): Promise { - const buildDesiredDeployment = async (): Promise => { + ): Promise { + let observedDeployment: k8s.V1Deployment | undefined + let holdingTemplate = false + let applied = true + const buildDesiredDeployment = async (): Promise => { const state = resolveStateBeforeMutation ? await resolveStateBeforeMutation() : null + const effective = state?.lifecycle ?? lifecycle + holdingTemplate = effective?.suspensionBlocked === true + if (holdingTemplate) { + if (!observedDeployment) return null + const existing = observedDeployment + if ( + !HostReconciler.isHccOwnedHostResource(existing, host.name) || + existing.metadata?.name !== host.name || + existing.metadata?.namespace !== host.namespace || + !existing.metadata?.uid || + !existing.metadata?.resourceVersion || + existing.metadata.deletionTimestamp || + !existing.spec?.template?.spec + ) { + throw new Error(`Cannot preserve an unverified Deployment for Host "${host.name}"`) + } + const annotatedHostUid = existing.metadata.annotations?.[HOST_UID_ANNOTATION] + const hostUidVerified = !!host.uid && annotatedHostUid === host.uid + if (!hostUidVerified) { + throw new Error(`Cannot preserve an unverified Deployment for Host "${host.name}"`) + } + // Preserve UID/resourceVersion and every applied field. Conflicts + // re-read this object before retrying; replicas is the only field that + // may change. An unverified legacy object must never be scaled. + const replicas = effective?.allowScaleUpDuringHold ? 1 : (existing.spec.replicas ?? 1) + return { + ...existing, + spec: { ...existing.spec, replicas }, + } + } return this.buildDeployment( host, mounts, state?.runtimeTokenRevision ?? runtimeTokenRevision, - state?.lifecycle ?? lifecycle, + effective, state?.grokExecutionEnabled ?? this.hostDerivesGrokExecution(host) ) } @@ -3624,16 +4069,22 @@ export class HostReconciler { revalidate?.() return true } - await ensureResource({ + const result = await ensureResource({ mutationAllowed, - read: () => - observeExistenceRead('Deployment', () => + read: async () => { + observedDeployment = await observeExistenceRead('Deployment', () => this.appsApi.readNamespacedDeployment({ namespace: host.namespace, name: host.name }) - ), + ) + return observedDeployment + }, create: async () => { // The initial GET may outlive the lifecycle or token-scope observation. const deployment = await buildDesiredDeployment() revalidate?.() + if (!deployment) { + applied = false + return + } return observeCreate('Deployment', () => this.appsApi.createNamespacedDeployment({ namespace: host.namespace, body: deployment }) ) @@ -3643,8 +4094,13 @@ export class HostReconciler { replaceWithConflictRetry({ description: `Deployment "${host.name}"`, logPrefix: '[HostReconciler]', - resolveBody: buildDesiredDeployment, - mergeExisting: preserveHostDeploymentAnnotations, + resolveBody: async () => { + const desired = await buildDesiredDeployment() + if (!desired) throw new Error('Deployment disappeared during preservation') + return desired + }, + mergeExisting: (desired, existing) => + holdingTemplate ? desired : preserveHostDeploymentAnnotations(desired, existing), isUpToDate: deploymentMatchesDesired, mutationAllowed, read, @@ -3656,6 +4112,7 @@ export class HostReconciler { }), }), }) + return applied && this.resourceApplySucceeded(result) } private async deleteRuntimeResources(name: string, namespace: string): Promise { @@ -3822,9 +4279,11 @@ export class HostReconciler { } } - private async ensureDesktopNetworkPolicy(host: HostCRD): Promise { + private async ensureDesktopNetworkPolicy( + host: HostCRD + ): Promise<'not-required' | ResourceApplyResult> { const isDesktop = !!(host.spec.desktop?.browser || host.spec.desktop?.x11) - if (!isDesktop) return + if (!isDesktop) return 'not-required' const policyName = `allow-rpc-proxy-desktop-${host.name}` const policy: k8s.V1NetworkPolicy = { @@ -3858,17 +4317,13 @@ export class HostReconciler { }, } - try { - await applyNetworkPolicy( - this.networkingApi, - policyName, - host.namespace, - policy, - '[HostReconciler]' - ) - } catch (error) { - log.error('Failed to ensure desktop NetworkPolicy', { policy: policyName, err: error }) - } + return applyNetworkPolicy( + this.networkingApi, + policyName, + host.namespace, + policy, + '[HostReconciler]' + ) } /** @@ -3877,7 +4332,7 @@ export class HostReconciler { * egress. Desktop hosts also expose the desktop service port through the * same host-scoped egress boundary. */ - private async ensureRpcProxyHostEgressNetworkPolicy(host: HostCRD): Promise { + private async ensureRpcProxyHostEgressNetworkPolicy(host: HostCRD): Promise { const isDesktop = !!(host.spec.desktop?.browser || host.spec.desktop?.x11) const policyName = `rpc-proxy-${host.name}-egress-mcp-host` const ports: k8s.V1NetworkPolicyPort[] = [{ port: config.hostPort, protocol: 'TCP' }] @@ -3920,7 +4375,7 @@ export class HostReconciler { }, } - await applyNetworkPolicy( + return applyNetworkPolicy( this.networkingApi, policyName, config.rpcProxyNamespace, @@ -3934,7 +4389,9 @@ export class HostReconciler { * its bound mcp-host on :8080. Provider approvals and verification continue * through mcp-host; channel-reader must not reach control-api gateways. */ - private async ensureChannelReaderEgressNetworkPolicy(host: HostCRD): Promise { + private async ensureChannelReaderEgressNetworkPolicy( + host: HostCRD + ): Promise { const policyName = `channel-reader-${host.name}-egress` const policy: k8s.V1NetworkPolicy = { apiVersion: 'networking.k8s.io/v1', @@ -3979,7 +4436,7 @@ export class HostReconciler { // channel-reader security boundary. Route-level JWT auth is the other // half, so an NP apply error still MUST surface in HostRuntimeStatus // rather than be silently swallowed. - await applyNetworkPolicy( + return applyNetworkPolicy( this.networkingApi, policyName, config.channelsNamespace, @@ -3993,7 +4450,9 @@ export class HostReconciler { * are verified by control-api against a CommunicationChannel, then forwarded * to that channel's bound Host for the authoritative provider-decision path. */ - private async ensureWorkflowApprovalReaderHostEgressNetworkPolicy(host: HostCRD): Promise { + private async ensureWorkflowApprovalReaderHostEgressNetworkPolicy( + host: HostCRD + ): Promise { const policyName = `workflow-approval-reader-${host.name}-egress-mcp-host` const policy: k8s.V1NetworkPolicy = { apiVersion: 'networking.k8s.io/v1', @@ -4033,7 +4492,7 @@ export class HostReconciler { }, } - await applyNetworkPolicy( + return applyNetworkPolicy( this.networkingApi, policyName, config.channelsNamespace, @@ -4047,7 +4506,7 @@ export class HostReconciler { * pod ingress on :8080 to ONLY accept connections from channel-reader- * pods (label match). */ - private async ensureMcpHostIngressNetworkPolicy(host: HostCRD): Promise { + private async ensureMcpHostIngressNetworkPolicy(host: HostCRD): Promise { const policyName = `mcp-host-${host.name}-ingress-channel-reader` const policy: k8s.V1NetworkPolicy = { apiVersion: 'networking.k8s.io/v1', @@ -4095,7 +4554,7 @@ export class HostReconciler { // channel-reader↔mcp-host security boundary. Route-level JWT auth is // the other half, so an apply error MUST surface in HostRuntimeStatus // rather than be silently swallowed. - await applyNetworkPolicy( + return applyNetworkPolicy( this.networkingApi, policyName, host.namespace, @@ -4111,7 +4570,7 @@ export class HostReconciler { */ private async ensureWorkflowApprovalReaderMcpHostIngressNetworkPolicy( host: HostCRD - ): Promise { + ): Promise { const policyName = `mcp-host-${host.name}-ingress-workflow-approval-reader` const policy: k8s.V1NetworkPolicy = { apiVersion: 'networking.k8s.io/v1', @@ -4153,7 +4612,7 @@ export class HostReconciler { }, } - await applyNetworkPolicy( + return applyNetworkPolicy( this.networkingApi, policyName, host.namespace, @@ -4167,7 +4626,9 @@ export class HostReconciler { * namespace-wide mcp-host ingress rule for rpc-proxy. Desktop hosts include * the desktop port, but only for the selected Host pod. */ - private async ensureRpcProxyMcpHostIngressNetworkPolicy(host: HostCRD): Promise { + private async ensureRpcProxyMcpHostIngressNetworkPolicy( + host: HostCRD + ): Promise { const isDesktop = !!(host.spec.desktop?.browser || host.spec.desktop?.x11) const policyName = `mcp-host-${host.name}-ingress-rpc-proxy` const ports: k8s.V1NetworkPolicyPort[] = [{ port: config.hostPort, protocol: 'TCP' }] @@ -4210,7 +4671,7 @@ export class HostReconciler { }, } - await applyNetworkPolicy( + return applyNetworkPolicy( this.networkingApi, policyName, host.namespace, @@ -4224,7 +4685,7 @@ export class HostReconciler { * NetworkPolicy only opens the required transport lane from the selected Host * pod to the governed GFS controller service. */ - private async ensureMcpHostGfsEgressNetworkPolicy(host: HostCRD): Promise { + private async ensureMcpHostGfsEgressNetworkPolicy(host: HostCRD): Promise { const target = { namespace: config.gfsNamespace, port: config.gfscPort } const policyName = `mcp-host-${host.name}-egress-gfs` const policy: k8s.V1NetworkPolicy = { @@ -4265,7 +4726,7 @@ export class HostReconciler { }, } - await applyNetworkPolicy( + return applyNetworkPolicy( this.networkingApi, policyName, host.namespace, @@ -4278,7 +4739,9 @@ export class HostReconciler { * Per-Host mcp-host egress to the static Codex LLM proxy runtime listener. * Derived from the same Codex projection that mints `llm:codex:execute`. */ - private async ensureMcpHostCodexProxyEgressNetworkPolicy(host: HostCRD): Promise { + private async ensureMcpHostCodexProxyEgressNetworkPolicy( + host: HostCRD + ): Promise { const policyName = `mcp-host-${host.name}-egress-codex-proxy` const policy: k8s.V1NetworkPolicy = { apiVersion: 'networking.k8s.io/v1', @@ -4318,7 +4781,7 @@ export class HostReconciler { }, } - await applyNetworkPolicy( + return applyNetworkPolicy( this.networkingApi, policyName, host.namespace, @@ -4347,17 +4810,21 @@ export class HostReconciler { ) } - private async reconcileMcpHostCodexProxyEgressNetworkPolicy(host: HostCRD): Promise { + private async reconcileMcpHostCodexProxyEgressNetworkPolicy( + host: HostCRD + ): Promise<'not-required' | ResourceApplyResult> { const projection = this.projectCodexForHost(host) - if (projection.eligibility === 'uncertain') return + if (projection.eligibility === 'uncertain') return 'not-required' if (projection.requiresCodexProxyEgress) { - await this.ensureMcpHostCodexProxyEgressNetworkPolicy(host) - return + return this.ensureMcpHostCodexProxyEgressNetworkPolicy(host) } await this.deleteMcpHostCodexProxyEgressNetworkPolicy(host) + return 'not-required' } - private async ensureMcpHostGrokProxyEgressNetworkPolicy(host: HostCRD): Promise { + private async ensureMcpHostGrokProxyEgressNetworkPolicy( + host: HostCRD + ): Promise { const policyName = `mcp-host-${host.name}-egress-grok-proxy` const policy: k8s.V1NetworkPolicy = { apiVersion: 'networking.k8s.io/v1', @@ -4396,7 +4863,7 @@ export class HostReconciler { ], }, } - await applyNetworkPolicy( + return applyNetworkPolicy( this.networkingApi, policyName, host.namespace, @@ -4425,14 +4892,16 @@ export class HostReconciler { ) } - private async reconcileMcpHostGrokProxyEgressNetworkPolicy(host: HostCRD): Promise { + private async reconcileMcpHostGrokProxyEgressNetworkPolicy( + host: HostCRD + ): Promise<'not-required' | ResourceApplyResult> { const projection = this.projectGrokForHost(host) - if (projection.eligibility === 'uncertain') return + if (projection.eligibility === 'uncertain') return 'not-required' if (projection.requiresGrokProxyEgress) { - await this.ensureMcpHostGrokProxyEgressNetworkPolicy(host) - return + return this.ensureMcpHostGrokProxyEgressNetworkPolicy(host) } await this.deleteMcpHostGrokProxyEgressNetworkPolicy(host) + return 'not-required' } /** @@ -4510,6 +4979,7 @@ export class HostReconciler { const capturedAuthority = this.captureHostMutationAuthority() const capturedDependencies = this.resolveHostMutationDependencies?.(host) return this.lifecycle.serializeByHost(host.name, async () => { + this.advanceReconcileGeneration(host.name) this.requireHostMutationAuthority(`Host "${host.name}" reconcile`, capturedAuthority) const admittedHost = this.resolveCurrentHost ? this.resolveCurrentHost(host.name) : host if (!admittedHost) return @@ -4686,10 +5156,187 @@ export class HostReconciler { }) } + let pvcApplied = true + let serviceApplied = true + const resourceErrors: unknown[] = [] + const npFailures: string[] = [] + const ensureIndependentResource = async ( + kind: 'Host PVC' | 'Host Service', + reconcile: () => Promise + ): Promise => { + try { + return await reconcile() + } catch (error) { + if (isBenignSupersessionError(error)) throw error + resourceErrors.push(error) + log.error('Failed to reconcile Host resource', { + host: host.name, + resource: kind, + err: error, + }) + return false + } + } + const holdAppliedRuntime = async (wakeRequested: boolean): Promise => { + const liveDeployment = await this.readHostDeploymentOrNull(host) + // A legacy Deployment without a runtime-token-revision annotation is not + // a preserved wake target, so it stays at zero replicas until channel + // authority returns and the normal path rewrites its template. + const appliedHostRuntime = + HostReconciler.deploymentBelongsToHost(liveDeployment, host) && + HostReconciler.deploymentRuntimeTokenRevision(liveDeployment) !== '' + const liveReplicas = liveDeployment?.spec?.replicas ?? 1 + const preservedWakeTarget = appliedHostRuntime && liveReplicas === 0 + // A suspension recorded in status whose scale-down never reached the + // Deployment leaves the applied runtime running. A wake on it needs no + // mint and no replica change: it is held like any running runtime. + const wakeOnRunningRuntime = wakeRequested && appliedHostRuntime && liveReplicas > 0 + const runtimeIsReady = + !!liveDeployment && + HostReconciler.deploymentBelongsToHost(liveDeployment, host) && + HostReconciler.deploymentReady(liveDeployment) + const scaleUpForRecovery = lifecycle.effective.state === 'active' && preservedWakeTarget + const allowScaleUp = preservedWakeTarget && (wakeRequested || scaleUpForRecovery) + if ( + !liveDeployment || + ((wakeRequested || scaleUpForRecovery) && !preservedWakeTarget && !wakeOnRunningRuntime) + ) { + this.setStatus(host.name, { + deployed: !!liveDeployment, + ready: false, + message: + 'Waiting for a verified applied runtime before waking during CommunicationChannel inventory loss', + }) + return + } + let provisioningRequired = false + if (allowScaleUp) { + // A pod starting from zero will consume bootstrap credentials. Mint and + // persist them before the replica-only update; a failed mint aborts here. + provisioningRequired = true + } else if ( + (lifecycle.effective.state === 'active' || wakeOnRunningRuntime) && + runtimeIsReady + ) { + let provision: RuntimeTokenProvision | null + try { + provision = await this.provisionRuntimeTokenRevision(host, { + targetSuspended: false, + refreshGfsOnly: true, + }) + } catch (error) { + // The Ready runtime keeps serving on the credential it holds; a failed + // renewal is reported, not turned into an undeployed Host. The next + // hold pass renews again. + log.warn('held runtime credential renewal failed; keeping the Ready runtime', { + host: host.name, + namespace: host.namespace, + err: error instanceof Error ? error.message : String(error), + }) + // Telemetry only: this pass still ends with the runtime deployed and + // Ready, so the administrative outcome it reports is `succeeded`. A + // `failed` outcome here would give one intent both outcomes. + this.enqueueHostTelemetry(host, 'controller_error', 'RuntimeCredentialRenewalFailed', { + status: 'failed', + error_class: error instanceof Error ? error.name : typeof error, + }) + this.setStatus(host.name, { + deployed: true, + ready: runtimeIsReady, + message: 'Held runtime credential renewal failed; runtime kept', + }) + return + } + if (!provision) { + this.setStatus(host.name, { + deployed: true, + ready: false, + message: 'Held runtime credential renewal is not safely available', + }) + return + } + revalidateHostMutationBoundary() + } + if (provisioningRequired) { + const provision = await this.provisionRuntimeTokenRevision(host, { + forceFreshForWake: true, + // This mint exists only to scale the Host up from zero, so the + // target is never suspended. + targetSuspended: false, + preserveDeploymentTemplateOnWake: true, + }) + if (!provision) { + this.setStatus(host.name, { + deployed: !!liveDeployment, + ready: false, + message: 'Waiting for authoritative scope observation', + }) + return + } + revalidateHostMutationBoundary() + } + const applied = await this.ensureDeployment( + host, + mounts, + undefined, + { ...lifecycle.effective, allowScaleUpDuringHold: allowScaleUp }, + undefined, + revalidateHostMutationBoundary + ) + revalidateHostMutationBoundary() + const runtimeExpectedToRun = + lifecycle.effective.state !== 'suspended' || allowScaleUp || wakeOnRunningRuntime + if (runtimeExpectedToRun) { + this.lifecycle.markHostNotSuspended(host.name) + } + const deploymentReady = + (lifecycle.effective.state !== 'suspended' || wakeOnRunningRuntime) && + applied && + (await this.checkDeploymentReady(host.name, host.namespace)) + revalidateHostMutationBoundary() + const ready = deploymentReady && pvcApplied && serviceApplied && npFailures.length === 0 + // PVC/Service read errors were already thrown before the hold; only + // non-throwing convergence failures reach this point. + const failures = [ + ...(!pvcApplied ? ['Host PVC did not converge'] : []), + ...(!serviceApplied ? ['Host Service did not converge'] : []), + ...npFailures, + ] + this.advanceReconcileGeneration(host.name) + const timer = this.readinessTimers.get(host.name) + if (timer) { + clearTimeout(timer) + this.readinessTimers.delete(host.name) + } + this.setStatus(host.name, { + deployed: applied, + ready, + message: !applied + ? 'Waiting for CommunicationChannel inventory before creating runtime' + : failures.length > 0 + ? `degraded — Host runtime boundary incomplete while preserving applied runtime (${failures.join('; ')})` + : !runtimeExpectedToRun + ? 'CommunicationChannel inventory unavailable; preserving suspended Host replicas' + : deploymentReady + ? 'CommunicationChannel inventory unavailable; preserving applied runtime' + : 'CommunicationChannel inventory unavailable; preserved applied runtime is not Ready', + }) + // Mirror the normal path: a runtime expected to run (including a held + // wake from zero) is polled until Ready; a degraded boundary is not, + // because the poll would overwrite the degraded verdict with "Running". + if (applied && !deploymentReady && failures.length === 0 && runtimeExpectedToRun) { + this.pollReadiness(host.name, host.namespace) + } + } + revalidateHostMutationBoundary() - await this.ensurePvc(host, revalidateHostMutationBoundary) + pvcApplied = await ensureIndependentResource('Host PVC', () => + this.ensurePvc(host, revalidateHostMutationBoundary) + ) revalidateHostMutationBoundary() - await this.ensureService(host, revalidateHostMutationBoundary) + serviceApplied = await ensureIndependentResource('Host Service', () => + this.ensureService(host, revalidateHostMutationBoundary) + ) // NetworkPolicies before Deployments. Calico/Cilium evaluate egress and // ingress against the policies that exist when the connection is opened, @@ -4702,45 +5349,54 @@ export class HostReconciler { // Route-level edge context and ownership checks remain the application // boundary. An NP apply failure means the network boundary is missing, so // surface it via HostRuntimeStatus instead of swallowing it. - const npFailures: string[] = [] - try { - revalidateHostMutationBoundary() - await this.ensureMcpHostIngressNetworkPolicy(host) - revalidateHostMutationBoundary() - await this.ensureMcpHostGfsEgressNetworkPolicy(host) - revalidateHostMutationBoundary() - await this.reconcileMcpHostCodexProxyEgressNetworkPolicy(host) - revalidateHostMutationBoundary() - await this.reconcileMcpHostGrokProxyEgressNetworkPolicy(host) - // The mcp-host→llm-hooks egress policy is now owned by LlmHookReconciler - // (per-host, scoped to referenced hook pods — N1/N7); host-delete cleanup - // of `mcp-host--egress-llm-hooks` stays in deleteHostNetworkPolicies. - revalidateHostMutationBoundary() - await this.ensureWorkflowApprovalReaderMcpHostIngressNetworkPolicy(host) - } catch (err) { - log.error('Failed to ensure mcp-host NP', { host: host.name, err }) - npFailures.push(`mcp-host NP: ${(err as Error).message}`) - } - try { - revalidateHostMutationBoundary() - await this.ensureRpcProxyMcpHostIngressNetworkPolicy(host) - revalidateHostMutationBoundary() - await this.ensureRpcProxyHostEgressNetworkPolicy(host) - } catch (err) { - log.error('Failed to ensure rpc-proxy host NP', { host: host.name, err }) - npFailures.push(`rpc-proxy NP: ${(err as Error).message}`) + const ensureNetworkPolicy = async ( + name: string, + reconcile: () => Promise<'not-required' | ResourceApplyResult> + ): Promise => { + try { + revalidateHostMutationBoundary() + const result = await reconcile() + revalidateHostMutationBoundary() + if (result !== 'not-required' && !this.resourceApplySucceeded(result)) { + throw new Error(`apply did not converge (${result})`) + } + } catch (err) { + log.error('Failed to ensure Host NetworkPolicy', { host: host.name, policy: name, err }) + npFailures.push(`${name}: ${err instanceof Error ? err.message : String(err)}`) + } } + await ensureNetworkPolicy('mcp-host ingress', () => + this.ensureMcpHostIngressNetworkPolicy(host) + ) + await ensureNetworkPolicy('mcp-host GFS egress', () => + this.ensureMcpHostGfsEgressNetworkPolicy(host) + ) + await ensureNetworkPolicy('mcp-host Codex proxy egress', () => + this.reconcileMcpHostCodexProxyEgressNetworkPolicy(host) + ) + await ensureNetworkPolicy('mcp-host Grok proxy egress', () => + this.reconcileMcpHostGrokProxyEgressNetworkPolicy(host) + ) + // The mcp-host→llm-hooks egress policy is now owned by LlmHookReconciler + // (per-host, scoped to referenced hook pods — N1/N7); host-delete cleanup + // of `mcp-host--egress-llm-hooks` stays in deleteHostNetworkPolicies. + await ensureNetworkPolicy('workflow approval reader ingress', () => + this.ensureWorkflowApprovalReaderMcpHostIngressNetworkPolicy(host) + ) + await ensureNetworkPolicy('rpc-proxy mcp-host ingress', () => + this.ensureRpcProxyMcpHostIngressNetworkPolicy(host) + ) + await ensureNetworkPolicy('rpc-proxy Host egress', () => + this.ensureRpcProxyHostEgressNetworkPolicy(host) + ) + await ensureNetworkPolicy('desktop ingress', () => this.ensureDesktopNetworkPolicy(host)) + await ensureNetworkPolicy('channel-reader egress', () => + this.ensureChannelReaderEgressNetworkPolicy(host) + ) + await ensureNetworkPolicy('workflow approval reader egress', () => + this.ensureWorkflowApprovalReaderHostEgressNetworkPolicy(host) + ) revalidateHostMutationBoundary() - await this.ensureDesktopNetworkPolicy(host) - try { - revalidateHostMutationBoundary() - await this.ensureChannelReaderEgressNetworkPolicy(host) - revalidateHostMutationBoundary() - await this.ensureWorkflowApprovalReaderHostEgressNetworkPolicy(host) - } catch (err) { - log.error('Failed to ensure channels egress NP', { host: host.name, err }) - npFailures.push(`egress NP: ${(err as Error).message}`) - } // Resolve policy once before bootstrap so token scopes and wake/suspend // decisions use the latest channel state already observed. @@ -4755,6 +5411,21 @@ export class HostReconciler { } revalidateHostMutationBoundary() + // PVC/Service errors abort before any credential or Deployment mutation, + // on the hold path as on the normal path. + if (resourceErrors.length === 1) throw resourceErrors[0] + if (resourceErrors.length > 1) { + throw new AggregateError( + resourceErrors, + `Failed to reconcile Host resources for "${host.name}"` + ) + } + + if (lifecycle.effective.suspensionBlocked) { + await holdAppliedRuntime(forceFreshForWake) + return + } + // Bootstrap captures the scope contract used for issuance. The Deployment // guard below compares that contract with the live channel cache after this // potentially slow I/O and before every create/replace attempt. @@ -4762,6 +5433,23 @@ export class HostReconciler { forceFreshForWake, targetSuspended: lifecycle.effective.stateless && lifecycle.effective.state === 'suspended', }) + if (!runtimeTokenProvision) { + const liveDeployment = await this.readHostDeploymentOrNull(host) + this.setStatus(host.name, { + deployed: !!liveDeployment, + ready: false, + message: 'Waiting for authoritative scope observation', + }) + return + } + const requireRuntimeTokenProvision = (): RuntimeTokenProvision => { + if (!runtimeTokenProvision) { + const error = new Error('Waiting for authoritative scope observation') + error.name = 'RuntimeScopeObservationUnavailableError' + throw error + } + return runtimeTokenProvision + } revalidateHostMutationBoundary() // replaceWithConflictRetry may wait and re-read after a 409, so each body is @@ -4784,20 +5472,32 @@ export class HostReconciler { const ensureCurrentRuntimeTokenScope = async (): Promise => { for (let attempt = 1; attempt <= 3; attempt++) { revalidateHostMutationBoundary() - const currentScopeHash = this.runtimeScopeHashFor(host, await this.frontsOAuthServer(host)) - if (runtimeTokenProvision.scopeHash === currentScopeHash) return + const currentOAuth = await this.observeFrontsOAuthServer(host) + if (this.currentScopeMatchesProvision(host, requireRuntimeTokenProvision(), currentOAuth)) { + return + } runtimeTokenProvision = await this.provisionRuntimeTokenRevision(host, { forceFreshForWake: false, targetSuspended: lifecycle.effective.stateless && lifecycle.effective.state === 'suspended', }) + if (!runtimeTokenProvision) { + const error = new Error('Waiting for authoritative scope observation') + error.name = 'RuntimeScopeObservationUnavailableError' + throw error + } revalidateHostMutationBoundary() - const postProvisionScopeHash = this.runtimeScopeHashFor( - host, - await this.frontsOAuthServer(host) - ) - if (runtimeTokenProvision.scopeHash === postProvisionScopeHash) return + const postProvisionOAuth = await this.observeFrontsOAuthServer(host) + if ( + this.currentScopeMatchesProvision( + host, + requireRuntimeTokenProvision(), + postProvisionOAuth + ) + ) { + return + } log.warn('CommunicationChannel scope contract changed during token provisioning', { host: host.name, @@ -4813,15 +5513,30 @@ export class HostReconciler { const resolveDeploymentState = async (): Promise => { for (let attempt = 1; attempt <= 3; attempt++) { revalidateHostMutationBoundary() - await resolveDeploymentLifecycle() + const beforeScope = await resolveDeploymentLifecycle() + if (beforeScope.suspensionBlocked) { + return { + lifecycle: beforeScope, + runtimeTokenRevision: requireRuntimeTokenProvision().revision, + grokExecutionEnabled: this.hostDerivesGrokExecution(host), + } + } await ensureCurrentRuntimeTokenScope() const effective = await resolveDeploymentLifecycle() - const currentScopeHash = this.runtimeScopeHashFor(host, await this.frontsOAuthServer(host)) - if (runtimeTokenProvision.scopeHash === currentScopeHash) { + const currentOAuth = await this.observeFrontsOAuthServer(host) + // The OAuth lookup can await I/O after the last lifecycle observation. + // Retry assessment rather than committing a now-obsolete mode. + if ( + this.lifecycle.enforceCommunicationChannelPolicyBeforeDeployment(host.name, lifecycle) !== + lifecycle + ) { + continue + } + if (this.currentScopeMatchesProvision(host, requireRuntimeTokenProvision(), currentOAuth)) { revalidateHostMutationBoundary() return { lifecycle: effective, - runtimeTokenRevision: runtimeTokenProvision.revision, + runtimeTokenRevision: requireRuntimeTokenProvision().revision, grokExecutionEnabled: this.hostDerivesGrokExecution(host), } } @@ -4837,16 +5552,39 @@ export class HostReconciler { } revalidateHostMutationBoundary() - await this.ensureDeployment( - host, - mounts, - runtimeTokenProvision.revision, - lifecycle.effective, - resolveDeploymentState, - revalidateHostMutationBoundary - ) + let deploymentApplied: boolean + try { + deploymentApplied = await this.ensureDeployment( + host, + mounts, + runtimeTokenProvision.revision, + lifecycle.effective, + resolveDeploymentState, + revalidateHostMutationBoundary + ) + } catch (error) { + if (error instanceof Error && error.name === 'RuntimeScopeObservationUnavailableError') { + const liveDeployment = await this.readHostDeploymentOrNull(host) + this.setStatus(host.name, { + deployed: !!liveDeployment, + ready: false, + message: 'Waiting for authoritative scope observation', + }) + return + } + throw error + } revalidateHostMutationBoundary() + if (!deploymentApplied) { + this.setStatus(host.name, { + deployed: false, + ready: false, + message: 'Waiting for CommunicationChannel inventory before creating runtime', + }) + return + } + const suspended = lifecycle.effective.stateless && lifecycle.effective.state === 'suspended' if (suspended) { this.lifecycle.recordSuspendedApplied(host.name) @@ -4872,7 +5610,10 @@ export class HostReconciler { ready: ready && npFailures.length === 0, message, }) - if (!ready && !suspended) { + // Do not schedule a readiness poll when NetworkPolicy has degraded the + // runtime: the same-generation poll would overwrite the degraded verdict + // with "Running" once the Deployment alone becomes Ready. + if (!ready && !suspended && npFailures.length === 0) { this.pollReadiness(host.name, host.namespace) } diff --git a/host-context-controller/src/k8sClient.test.ts b/host-context-controller/src/k8sClient.test.ts index 698ca0d488..07c369a65c 100644 --- a/host-context-controller/src/k8sClient.test.ts +++ b/host-context-controller/src/k8sClient.test.ts @@ -7550,7 +7550,7 @@ describe('McpServerWatcher.start ordering (#281 R6-bis)', () => { expect(watcher.isCommunicationChannelCacheSynced()).toBe(false) expect(errorSpy).toHaveBeenCalledWith( '[K8s] CommunicationChannel initial load failed; ccCacheSynced remains false ' + - '(B2 preserves channel-reader replicas and holds stateless lifecycle active):', + '(B2 preserves channel-reader replicas, preserving durable Host lifecycle state):', expect.any(Error) ) @@ -10839,8 +10839,10 @@ describe('McpServerWatcher.hostFrontsOAuthServer (oauth:user-token scope probe)' }) /** Drive the REAL getServerInfosByContext: populate the server cache and answer - * the Context read the producer makes with an allow-list of those servers. */ + * the Context read the producer makes with an allow-list of those servers. + * The cache is an observation only while its LIST -> WATCH pair is live. */ const withServers = (watcher: McpServerWatcher, servers: any[]): void => { + markMcpServerInventoryAuthoritative(watcher) for (const s of servers) (watcher as any).servers.set(s.name, s) mocks.getNamespacedCustomObject.mockImplementation(async ({ plural }: { plural: string }) => plural === 'contexts' @@ -10885,6 +10887,60 @@ describe('McpServerWatcher.hostFrontsOAuthServer (oauth:user-token scope probe)' await expect((watcher as any).hostFrontsOAuthServer(hostFor('ctx'))).resolves.toBe(false) watcher.stop() }) + + /** The resolver McpServerWatcher wires into HostReconciler at construction. */ + const wiredResolver = (watcher: McpServerWatcher): ((host: HostCRD) => Promise) => { + const calls = (watcher.getHostReconciler() as any).setHostFrontsOAuthServer.mock.calls + expect(calls).toHaveLength(1) + return calls[0][0] + } + + it('does not serve a retired McpServer cache that would grant the OAuth scope', async () => { + const watcher = new McpServerWatcher() + withServers(watcher, [serverCRD({ name: 'notion', auth: { type: 'oauth' }, enabled: true })]) + const resolve = wiredResolver(watcher) + // Liveness witness: while the inventory is authoritative, this same cache + // and wired resolver grant the scope. + await expect(resolve(hostFor('ctx'))).resolves.toBe(true) + + // A McpServer watch outage retires the inventory but keeps its last cache, + // which still holds the enabled OAuth server an admin may have disabled. + expect((watcher as any).retireMcpServerWatch()).toBe(true) + expect((watcher as any).servers.get('notion')?.spec.enabled).toBe(true) + + await expect(resolve(hostFor('ctx'))).rejects.toThrow( + 'McpServer inventory is not authoritative; OAuth scope is unobserved' + ) + watcher.stop() + }) + + it('does not serve a McpServer cache that lost authority during the Context read', async () => { + const watcher = new McpServerWatcher() + withServers(watcher, [serverCRD({ name: 'notion', auth: { type: 'oauth' }, enabled: true })]) + const contextRead = deferred() + const contextReads: string[] = [] + mocks.getNamespacedCustomObject.mockImplementation( + async ({ plural, name }: { plural: string; name: string }) => { + contextReads.push(`${plural}/${name}`) + return contextRead.promise + } + ) + + const pending = wiredResolver(watcher)(hostFor('ctx')) + await flushMicrotasks() + // Liveness witness: the observation reached the live Context read. + expect(contextReads).toEqual(['contexts/ctx']) + expect((watcher as any).retireMcpServerWatch()).toBe(true) + contextRead.resolve({ + metadata: { name: 'ctx', namespace: 'mcp-server' }, + spec: { contextId: 'ctx', mcpServers: ['notion'] }, + }) + + await expect(pending).rejects.toThrow( + 'McpServer inventory lost authority during the OAuth scope read' + ) + watcher.stop() + }) }) describe('McpServerWatcher readiness under sustained watch churn (GKE Premature-close regime)', () => { // Regression for the clerum-dev livelock. PR #205 coupled /ready to a safety diff --git a/host-context-controller/src/k8sClient.ts b/host-context-controller/src/k8sClient.ts index 5847b95e85..7b26031449 100644 --- a/host-context-controller/src/k8sClient.ts +++ b/host-context-controller/src/k8sClient.ts @@ -1759,9 +1759,9 @@ export class McpServerWatcher implements McpServerProvider { void this.requestHostFleetReconcile('CommunicationChannel recovery', lifecycleGeneration) return true } catch (error) { - console.error( - '[K8s] CommunicationChannel cache recovery failed; stateless lifecycle remains held active:', - error + hccLogger.error( + '[K8s] CommunicationChannel cache recovery failed; preserving durable Host lifecycle state:', + { err: error } ) return false } @@ -2519,9 +2519,22 @@ export class McpServerWatcher implements McpServerProvider { * already filters to enabled + allowed servers) that mcp-host discovery uses, * so HCC does not need a second cross-CRD read path. HostReconciler wraps this * call in a fail-closed guard, so a thrown read there yields no oauth scope. + * + * A retired McpServer watch keeps its last `servers` cache, so an answer read + * from it is only an observation while the same LIST -> WATCH generation stays + * authoritative across the whole read. Otherwise this throws: a server an + * admin disabled during the outage must not re-grant the scope from the stale + * cache, and HostReconciler treats the throw as an unobserved read. */ private async hostFrontsOAuthServer(host: HostCRD): Promise { + const watchGeneration = this.mcpWatchGeneration + if (!this.hasMcpServerInventoryAuthority(watchGeneration)) { + throw new Error('McpServer inventory is not authoritative; OAuth scope is unobserved') + } const servers = await this.getServerInfosByContext(host.spec.contextRef) + if (!this.hasMcpServerInventoryAuthority(watchGeneration)) { + throw new Error('McpServer inventory lost authority during the OAuth scope read') + } return servers.some(server => server.enabled && server.auth?.type === 'oauth') } @@ -2828,7 +2841,7 @@ export class McpServerWatcher implements McpServerProvider { } catch (error) { console.error( '[K8s] CommunicationChannel initial load failed; ccCacheSynced remains false ' + - '(B2 preserves channel-reader replicas and holds stateless lifecycle active):', + '(B2 preserves channel-reader replicas, preserving durable Host lifecycle state):', error ) } @@ -3957,8 +3970,8 @@ export class McpServerWatcher implements McpServerProvider { if (err) { console.error('[K8s] CommunicationChannel watch error:', err) } - console.log( - '[K8s] CommunicationChannel watch ended; holding stateless lifecycle active until snapshot recovery' + hccLogger.info( + '[K8s] CommunicationChannel watch ended; preserving durable Host lifecycle state until snapshot recovery' ) void this.requestHostFleetReconcile( 'CommunicationChannel watch interruption', diff --git a/host-context-controller/src/statelessLifecycle.types.ts b/host-context-controller/src/statelessLifecycle.types.ts index 70a3d8cff3..ebaa1410fb 100644 --- a/host-context-controller/src/statelessLifecycle.types.ts +++ b/host-context-controller/src/statelessLifecycle.types.ts @@ -8,13 +8,17 @@ import { HostCondition, HostLifecycleState, HostLifecycleStatus } from './types' /** * Effective lifecycle of a Host after the reconcile-time rejection checks. - * `stateless` is false when spec.lifecycle.stateless is off OR a rejection - * applies (legacy force-always-on CommunicationChannel policy, spec.desktop - * present, unsatisfiable SharedFileSystem co-location). + * `stateless` is false when spec.lifecycle.stateless is off OR a confirmed + * incompatibility applies. Cache uncertainty blocks suspension and template + * selection; `stateless` then describes the request, not the applied mode. */ export interface EffectiveHostLifecycle { stateless: boolean state: HostLifecycleState + /** Unknown channel authority: preserve the entire applied Deployment template. */ + suspensionBlocked?: boolean + /** An active recovery or fresh-read-verified wake may scale the held Deployment without a template change. */ + allowScaleUpDuringHold?: boolean } /** Reconcile-time lifecycle assessment: effective mode + durable status + condition. */ diff --git a/host-context-controller/src/statelessLifecycleExecutor.ts b/host-context-controller/src/statelessLifecycleExecutor.ts index f731391ca1..479abe2d8b 100644 --- a/host-context-controller/src/statelessLifecycleExecutor.ts +++ b/host-context-controller/src/statelessLifecycleExecutor.ts @@ -11,6 +11,7 @@ import { config } from './config' import { isHeartbeatManagedLifecycleReason } from './constants' import type { ResolvedSfsMount } from './hostReconciler' import { SFS_LABEL, SFS_NAMESPACE_LABEL, WFC_APP_LABEL } from './k8s/sharedFileSystemFactory' +import { hccLogger } from './logger' import { STATELESS_PULL_POLICY_REJECTED_CONDITION_TYPE, pullPolicyNotApplicableCondition, @@ -21,17 +22,24 @@ import { HostLifecycleAssessment, SuspendFromHeartbeatOutcome, } from './statelessLifecycle.types' -import { HostCRD, HostCondition, HostCrdStatus, HostLifecycleStatus } from './types' +import { + HostCRD, + HostCondition, + HostCrdStatus, + HostLifecycleState, + HostLifecycleStatus, +} from './types' import { getErrorCode } from './utils' // ─── Stateless lifecycle (Stage 2) ───────────────────────────────────────── +const log = hccLogger.child({ module: 'stateless-lifecycle' }) const GROUP = 'clerum.io' const VERSION = 'v1alpha1' const PLURAL_HOSTS = 'hosts' const STATELESS_REJECTED_CONDITION_TYPE = 'StatelessEnableRejected' const COMMUNICATION_CHANNEL_CACHE_UNSYNCED_REASON = 'CommunicationChannelCacheUnsynced' const COMMUNICATION_CHANNEL_CACHE_UNSYNCED_MESSAGE = - 'CommunicationChannel cache is not synchronized; stateless lifecycle is held active' + 'CommunicationChannel cache is not synchronized; stateless lifecycle transitions are held' const ACTIVE_COMMUNICATION_CHANNELS_REASON = 'ActiveCommunicationChannels' /** * Host CR annotation carrying the monotonic wake generation (Stage 4.3). @@ -185,7 +193,7 @@ export class StatelessLifecycleExecutor { return } StatelessLifecycleExecutor.singleReplicaInvariantWarned = true - console.warn( + log.warn( '[HostReconciler] Stateless lifecycle status writes assume HCC runs replicas:1 WITHOUT leader election. serializeByHost orders writers within this process; the resourceVersion precondition + 409-retry closes the cross-writer window — but neither elects a leader. Running >1 replica risks two controllers racing (safe against corruption, but can livelock).' ) } @@ -217,14 +225,18 @@ export class StatelessLifecycleExecutor { if (host.spec.lifecycle?.stateless !== true) { return { stateless: false, state: 'active' } } - const hasCommunicationChannels = this.countCommunicationChannels(host.name) > 0 - if ( - !this.isCommunicationChannelCacheSynced() || - host.spec.desktop !== undefined || - hasCommunicationChannels - ) { + const hasCommunicationChannels = + this.isCommunicationChannelCacheSynced() && this.countCommunicationChannels(host.name) > 0 + if (host.spec.desktop !== undefined || hasCommunicationChannels) { return { stateless: false, state: 'active' } } + if (!this.isCommunicationChannelCacheSynced()) { + return { + stateless: true, + state: host.status?.lifecycle?.state ?? 'active', + suspensionBlocked: true, + } + } return { stateless: true, state: host.status?.lifecycle?.state ?? 'active' } } @@ -234,11 +246,10 @@ export class StatelessLifecycleExecutor { } { const reasons: string[] = [] const messages: string[] = [] - if (!this.isCommunicationChannelCacheSynced()) { - reasons.push(COMMUNICATION_CHANNEL_CACHE_UNSYNCED_REASON) - messages.push(COMMUNICATION_CHANNEL_CACHE_UNSYNCED_MESSAGE) - } - const ccCount = this.countCommunicationChannels(hostName) + // A retained positive entry is no more authoritative than an empty cache. + const ccCount = this.isCommunicationChannelCacheSynced() + ? this.countCommunicationChannels(hostName) + : 0 if (ccCount > 0) { // Addendum 6 (operator visibility): the rejection message names both the // associated channel count AND the recovery action, because control-ui @@ -321,6 +332,13 @@ export class StatelessLifecycleExecutor { } } + if (!this.isCommunicationChannelCacheSynced()) { + return this.holdDurableStateDuringChannelCacheRecovery( + wakeHandledGeneration, + host.status?.lifecycle?.state ?? 'active' + ) + } + // Accepted: preserve the durable state from the CRD status — a suspended // Host must survive HCC restarts and periodic resyncs. const state = host.status?.lifecycle?.state ?? 'active' @@ -351,13 +369,34 @@ export class StatelessLifecycleExecutor { } } + /** Unknown channel inventory requires preserving the applied Deployment. */ + private holdDurableStateDuringChannelCacheRecovery( + wakeHandledGeneration: number, + state: HostLifecycleState = 'active' + ): HostLifecycleAssessment { + return { + effective: { stateless: true, state, suspensionBlocked: true }, + lifecycle: { + state, + wakeHandledGeneration, + reason: COMMUNICATION_CHANNEL_CACHE_UNSYNCED_MESSAGE, + }, + condition: { + type: STATELESS_REJECTED_CONDITION_TYPE, + status: 'False', + reason: COMMUNICATION_CHANNEL_CACHE_UNSYNCED_REASON, + message: COMMUNICATION_CHANNEL_CACHE_UNSYNCED_MESSAGE, + }, + pullPolicyCondition: statelessPullPolicyCondition(config.hostImage), + } + } + /** * Close the interval between the initial lifecycle assessment and the * Deployment write. Reconciliation performs several Kubernetes operations - * in between; if the CommunicationChannel watch ends or a channel starts - * referencing this Host during that interval, an assessment that was safe - * when computed must not still scale the Host to zero or leave it on the - * stateless runtime template. + * in between; if the CommunicationChannel watch ends, retain the applied + * template, whose mode may be stateful. A confirmed channel still applies the + * established incompatible-mode policy. * * This synchronous cache check adds no Kubernetes API calls to steady state. */ @@ -366,11 +405,28 @@ export class StatelessLifecycleExecutor { assessment: HostLifecycleAssessment ): HostLifecycleAssessment { if (!assessment.effective.stateless) { + // Independent desktop/SFS incompatibilities retain their policy. A + // channel-only rejection cannot choose a runtime after authority is lost. + if ( + !this.isCommunicationChannelCacheSynced() && + assessment.condition.reason === ACTIVE_COMMUNICATION_CHANNELS_REASON + ) { + return this.holdDurableStateDuringChannelCacheRecovery( + assessment.lifecycle.wakeHandledGeneration, + assessment.effective.state + ) + } return assessment } const { reasons, messages } = this.communicationChannelPolicyRejection(hostName) if (reasons.length === 0) { + if (!this.isCommunicationChannelCacheSynced() && !assessment.effective.suspensionBlocked) { + return this.holdDurableStateDuringChannelCacheRecovery( + assessment.lifecycle.wakeHandledGeneration, + assessment.effective.state + ) + } return assessment } const message = messages.join('; ') @@ -419,9 +475,9 @@ export class StatelessLifecycleExecutor { } } } catch (err) { - console.warn( + log.warn( `[HostReconciler] Could not resolve wfc node placement for SFS "${m.namespace}/${m.name}" (host "${host.name}"); skipping the co-location rejection check:`, - err + { err } ) return null } @@ -575,44 +631,17 @@ export class StatelessLifecycleExecutor { freshLifecycle?.wakeHandledGeneration ?? 0 ) const cachedState = host.status?.lifecycle?.state ?? 'active' - // A rejection is an INTENDED reason override even when it does not - // change state. The reject/kill-switch path sets - // assessment.lifecycle = { state:'active', reason: } - // with condition.status='True' (StatelessEnableRejected). When the - // Host is ALREADY active a first-ever/steady rejection has - // assessment.state ('active') === cachedState ('active'), so the plain - // state-diff discriminator would route it to the ECHO branch, which - // re-sources reason via isHeartbeatManagedLifecycleReason and DROPS the - // rejection message (not heartbeat-managed) — silently losing the - // rejection explanation the CRD documents. Detect it via the condition - // transitioning to Rejected:True so the rejection reason is stamped on - // the fresh state instead of folded into the heartbeat-managed-only - // echo. (StatelessDisabled kill-switch carries status:'False' and no - // lifecycle reason, so it never trips this branch.) - const assessmentIsRejection = assessment.condition.status === 'True' + // Confirmed rejection requires active even when the cached state was + // already active. Cache uncertainty is different: preserve the fresh + // durable state and block scale/template transitions until authority + // returns. + const assessmentForcesActive = assessment.condition.status === 'True' const assessmentIntendsStateOverride = assessment.lifecycle.state !== cachedState let lifecycle: HostLifecycleStatus - if (assessmentIntendsStateOverride) { - // The assessment DIFFERS from the cached snapshot's state: it - // INTENDED a transition (kill-switch/rejection forcing active over a - // suspended snapshot, or a resolveWakeBeforeScaleDown wake - // transition). That override wins; only wakeHandledGeneration stays - // monotonic against fresh. + if (assessmentIntendsStateOverride || assessmentForcesActive) { + // The assessment requires a transition or forces active by policy. + // Only wakeHandledGeneration is sourced monotonically from fresh. lifecycle = { ...assessment.lifecycle, wakeHandledGeneration } - } else if (assessmentIsRejection) { - // Reject-while-active: state does not change, but the assessment's - // rejection reason is an INTENDED override. Keep state from FRESH - // (never reintroduce the 8th costume — a heartbeat suspend/wake that - // landed since the snapshot must still be preserved) and stamp the - // rejection reason onto it instead of dropping it into the - // heartbeat-managed-only echo. - lifecycle = { - state: freshLifecycle?.state ?? 'active', - wakeHandledGeneration, - ...(assessment.lifecycle.reason !== undefined - ? { reason: assessment.lifecycle.reason } - : {}), - } } else { // The assessment's state was a pass-through ECHO of the cached // snapshot — prefer the FRESH durable state and the heartbeat-managed @@ -653,9 +682,9 @@ export class StatelessLifecycleExecutor { } catch (err) { // Best-effort: a status-write failure (including 409-retry exhaustion) // is logged, not thrown — status writes must not block reconciliation. - console.error( + log.error( `[HostReconciler] Failed to write lifecycle status for "${host.name}" (${getErrorCode(err) ?? 'no code'}):`, - err + { err } ) return false } @@ -672,6 +701,11 @@ export class StatelessLifecycleExecutor { return this.effectiveLifecycleFromCache(host) } + private canSuspendFromCache(host: HostCRD): boolean { + const lifecycle = this.effectiveLifecycleFromCache(host) + return lifecycle.stateless && !lifecycle.suspensionBlocked + } + /** Public view of the clerum.io/wake-requested generation parser. */ getWakeRequestedGeneration(host: HostCRD): number { return this.wakeRequestedGeneration(host) @@ -788,10 +822,7 @@ export class StatelessLifecycleExecutor { // cache trails the API. Re-evaluate the complete effective stateless // policy from every fresh read/retry before honoring old heartbeat // evidence. - if ( - !this.sameHostSpecRevision(host, fresh) || - !this.effectiveLifecycleFromCache(fresh).stateless - ) { + if (!this.sameHostSpecRevision(host, fresh) || !this.canSuspendFromCache(fresh)) { return { skip: true } } const freshLifecycle = fresh.status?.lifecycle @@ -804,7 +835,7 @@ export class StatelessLifecycleExecutor { // newer evidence. A fresh `suspended` returns as a SILENT idempotent // no-op: the drained-report retry path relies on exactly that. if (freshLifecycle?.state !== 'suspended') { - console.log( + log.info( `[HostReconciler] Suspend of stateless Host "${host.name}" skipped — fresh state "${freshLifecycle?.state ?? 'active'}" is not draining (drain decision overturned)` ) } @@ -828,7 +859,7 @@ export class StatelessLifecycleExecutor { // with no reviver. const freshHandled = freshLifecycle.wakeHandledGeneration ?? 0 if (freshHandled > entryWakeHandledGeneration) { - console.log( + log.info( `[StatelessSuspend] host=${host.name} phase=drained_report_stale reason=wake_handled_since entryGeneration=${entryWakeHandledGeneration} freshGeneration=${freshHandled} ts=${this.now().getTime()}` ) return { skip: true } @@ -840,7 +871,7 @@ export class StatelessLifecycleExecutor { // suspended state that must immediately be overturned. const freshRequested = this.wakeRequestedGeneration(fresh) if (freshRequested > freshHandled) { - console.log( + log.info( `[StatelessSuspend] host=${host.name} phase=drained_report_stale reason=wake_pending requestedGeneration=${freshRequested} handledGeneration=${freshHandled} ts=${this.now().getTime()}` ) return { skip: true } @@ -887,7 +918,7 @@ export class StatelessLifecycleExecutor { } this.lastWrittenLifecycleStatus.delete(host.name) this.logSuspendedApplied(host.name) - console.log( + log.info( `[HostReconciler] Suspending stateless Host "${host.name}" (reason: ${reason}) — reconciling to replicas=0` ) // L2: call reconcileCore (not the serialized reconcile) — this method @@ -1057,10 +1088,7 @@ export class StatelessLifecycleExecutor { const result = await this.patchStatusWithPrecondition( host, fresh => { - if ( - !this.sameHostSpecRevision(host, fresh) || - !this.effectiveLifecycleFromCache(fresh).stateless - ) { + if (!this.sameHostSpecRevision(host, fresh) || !this.canSuspendFromCache(fresh)) { return { skip: true } } const freshLifecycle = fresh.status?.lifecycle @@ -1077,7 +1105,7 @@ export class StatelessLifecycleExecutor { // evidence. const freshHandled = freshLifecycle?.wakeHandledGeneration ?? 0 if (freshHandled > entryWakeHandledGeneration) { - console.log( + log.info( `[StatelessSuspend] host=${host.name} phase=draining_write_stale reason=wake_handled_since entryGeneration=${entryWakeHandledGeneration} freshGeneration=${freshHandled} ts=${this.now().getTime()}` ) return { skip: true } @@ -1086,7 +1114,7 @@ export class StatelessLifecycleExecutor { // by definition ("pending wake wins") — never fence over it. const freshRequested = this.wakeRequestedGeneration(fresh) if (freshRequested > freshHandled) { - console.log( + log.info( `[StatelessSuspend] host=${host.name} phase=draining_write_stale reason=wake_pending requestedGeneration=${freshRequested} handledGeneration=${freshHandled} ts=${this.now().getTime()}` ) return { skip: true } @@ -1108,7 +1136,7 @@ export class StatelessLifecycleExecutor { target.status = writtenStatus }) this.lastWrittenLifecycleStatus.delete(host.name) - console.log( + log.info( `[HostReconciler] Marked stateless Host "${host.name}" draining — control-api now answers {drain:true} from Host.status` ) } @@ -1173,7 +1201,7 @@ export class StatelessLifecycleExecutor { revalidate ) if ('skipped' in result || writtenStatus === undefined) { - console.log( + log.info( `[StatelessLifecycle] host=${host.name} cancel-drain skipped — fresh state "${skippedNotDraining ?? 'active'}" is not draining` ) return @@ -1182,7 +1210,7 @@ export class StatelessLifecycleExecutor { target.status = writtenStatus }) this.lastWrittenLifecycleStatus.delete(host.name) - console.log( + log.info( `[StatelessLifecycle] host=${host.name} cancel-drain: activity evidence while draining — reverting status to active` ) } @@ -1223,7 +1251,7 @@ export class StatelessLifecycleExecutor { if (Number.isNaN(parsed)) { if (this.malformedWakeAnnotationLogged.get(host.name) !== raw) { this.malformedWakeAnnotationLogged.set(host.name, raw) - console.error( + log.error( `[HostReconciler] Malformed ${WAKE_REQUESTED_ANNOTATION} annotation on Host "${host.name}": ${JSON.stringify( raw )} is not an integer; treating it as no wake intent` @@ -1281,9 +1309,9 @@ export class StatelessLifecycleExecutor { try { initialFresh = await this.readFreshHost(host) } catch (err) { - console.error( + log.error( `[HostReconciler] Wake fast-path fresh Host read failed for "${host.name}" (${getErrorCode(err) ?? 'no code'}):`, - err + { err } ) return false } @@ -1304,12 +1332,12 @@ export class StatelessLifecycleExecutor { } const currentState = initialFresh.status?.lifecycle?.state ?? 'active' - console.log( + log.info( `[HostReconciler] Wake fast-path for "${host.name}": generation ${freshWakeRequested} > handled ${freshWakeHandled} while ${currentState}` ) // Stage 6 (W2): machine-parseable wake-phase timestamps for the // wake-budget script. One line per phase, correlated by generation. - console.log( + log.info( `[StatelessWake] host=${host.name} generation=${freshWakeRequested} phase=wake_observed ts=${this.now().getTime()}` ) @@ -1370,9 +1398,9 @@ export class StatelessLifecycleExecutor { // lost — the annotation is durable and the drained-pre-scale guard in // reconcile() re-checks it before any replicas:0 apply, so a pending // wake can never be raced away by the suspension. - console.error( + log.error( `[HostReconciler] Wake fast-path status write failed for "${host.name}" (${getErrorCode(err) ?? 'no code'}):`, - err + { err } ) return false } @@ -1385,7 +1413,7 @@ export class StatelessLifecycleExecutor { target.annotations = latestFresh.annotations target.status = reflectedStatus }) - console.log( + log.info( `[StatelessWake] host=${host.name} generation=${writtenLifecycle.wakeHandledGeneration} phase=status_flipped ts=${this.now().getTime()}` ) @@ -1395,6 +1423,10 @@ export class StatelessLifecycleExecutor { return false } + // Cache recovery uses the full reconcile's owned, resource-version-fenced + // replica-only update. Do not send the unfenced wake scale patch here. + if (!this.isCommunicationChannelCacheSynced()) return true + // Minimal scale patch — NOT the full buildDeployment replace. It also // starts the pod even when the heavy body aborts early (e.g. on a // transient Secret read failure). @@ -1407,8 +1439,8 @@ export class StatelessLifecycleExecutor { ], } ) - console.log(`[HostReconciler] Wake fast-path scaled Deployment "${host.name}" to replicas=1`) - console.log( + log.info(`[HostReconciler] Wake fast-path scaled Deployment "${host.name}" to replicas=1`) + log.info( `[StatelessWake] host=${host.name} generation=${writtenLifecycle.wakeHandledGeneration} phase=replicas_patched ts=${this.now().getTime()}` ) this.markHostNotSuspended(host.name) @@ -1418,7 +1450,7 @@ export class StatelessLifecycleExecutor { if (getErrorCode(err) === 404) { // Deployment missing (e.g. deleted while suspended): fall through — // the full reconcile below builds it with replicas=1. - console.log( + log.info( `[HostReconciler] Wake fast-path: Deployment "${host.name}" not found; the full reconcile will create it` ) return true @@ -1441,29 +1473,27 @@ export class StatelessLifecycleExecutor { namespace: host.namespace, body: deployment, }) - console.warn( + log.warn( `[HostReconciler] Wake fast-path scale patch forbidden for "${host.name}"; used deployment update fallback` ) - console.log( - `[HostReconciler] Wake fast-path scaled Deployment "${host.name}" to replicas=1` - ) - console.log( + log.info(`[HostReconciler] Wake fast-path scaled Deployment "${host.name}" to replicas=1`) + log.info( `[StatelessWake] host=${host.name} generation=${writtenLifecycle.wakeHandledGeneration} phase=replicas_patched ts=${this.now().getTime()}` ) this.markHostNotSuspended(host.name) this.recordScaleTransition(host.name, 'up') return true } catch (fallbackErr) { - console.error( + log.error( `[HostReconciler] Wake fast-path deployment update fallback failed for "${host.name}":`, - fallbackErr + { err: fallbackErr } ) return true } } // Surfaced loudly; ensureDeployment in this same reconcile pass // replaces the Deployment with replicas=1, so nothing is masked. - console.error(`[HostReconciler] Wake fast-path scale patch failed for "${host.name}":`, err) + log.error(`[HostReconciler] Wake fast-path scale patch failed for "${host.name}":`, { err }) return true } } @@ -1477,7 +1507,7 @@ export class StatelessLifecycleExecutor { private recordScaleTransition(hostName: string, direction: 'up' | 'down'): void { const total = (this.scaleTransitionsByHost.get(hostName) ?? 0) + 1 this.scaleTransitionsByHost.set(hostName, total) - console.log( + log.info( `[StatelessMetric] scale_transition host=${hostName} direction=${direction} total=${total}` ) } @@ -1496,7 +1526,7 @@ export class StatelessLifecycleExecutor { return } this.suspendedAppliedLoggedByHost.add(hostName) - console.log( + log.info( `[StatelessSuspend] host=${hostName} phase=suspended_applied ts=${this.now().getTime()}` ) } @@ -1595,7 +1625,7 @@ export class StatelessLifecycleExecutor { // The CR changed under us (a concurrent lifecycle writer or an // unrelated metadata write). Re-read fresh and re-evaluate the // guard so a decision the racing writer just made is respected. - console.warn( + log.warn( `[HostReconciler] Status write for "${host.name}" hit 409 (stale resourceVersion); re-reading fresh and retrying (attempt ${attempt}/${maxAttempts})` ) fresh = undefined @@ -1763,6 +1793,11 @@ export class StatelessLifecycleExecutor { host: HostCRD, assessment: HostLifecycleAssessment ): Promise { + if (assessment.effective.suspensionBlocked) { + // Cache authority loss blocks lifecycle-driven scale-down; preserve the + // durable state for accurate reporting without mutating the Deployment. + return assessment + } if (!assessment.effective.stateless) { // Kill-switch/rejection assessments INTEND active+replicas:1 regardless // of the durable state — never second-guess them from fresh. @@ -1777,16 +1812,16 @@ export class StatelessLifecycleExecutor { // Fail loud and skip the scale-down this pass: a suspend derived from // a possibly-stale cache must NOT strand a racing wake at replicas:0. // The periodic resync re-runs this reconcile with a fresh cache. - console.error( + log.error( `[HostReconciler] Drained-pre-scale guard for "${host.name}": fresh Host read failed; skipping replicas=0 this pass (resync will retry):`, - err + { err } ) } else { // Same posture in the mirror direction: never commit replicas from a // possibly-stale cached state when the fresh read is unavailable. - console.error( + log.error( `[HostReconciler] Stateless replicas guard for "${host.name}": fresh Host read failed; skipping the Deployment scale this pass (resync will retry):`, - err + { err } ) } return null @@ -1796,7 +1831,7 @@ export class StatelessLifecycleExecutor { const freshState = fresh.status?.lifecycle?.state ?? 'active' if (freshWakeRequested > freshWakeHandled) { if (cachedState === 'suspended') { - console.log( + log.info( `[HostReconciler] Drained-pre-scale guard for "${host.name}": aborting replicas=0 — fresh wake generation ${freshWakeRequested} > handled ${freshWakeHandled}` ) // A wake is pending in the fresh read — run the wake transition @@ -1825,7 +1860,7 @@ export class StatelessLifecycleExecutor { if ((cachedState === 'suspended') === freshSuspended) { return assessment } - console.log( + log.info( `[HostReconciler] Stateless replicas guard for "${host.name}": cached lifecycle state "${cachedState}" disagrees with fresh "${freshState}" — deriving replicas from FRESH state` ) // Reflect fresh onto the in-memory host (mirroring the wake fast-path's diff --git a/host-context-controller/src/statelessLifecycleTracker.ts b/host-context-controller/src/statelessLifecycleTracker.ts index d3c74c3450..087741c5a4 100644 --- a/host-context-controller/src/statelessLifecycleTracker.ts +++ b/host-context-controller/src/statelessLifecycleTracker.ts @@ -296,6 +296,16 @@ export class StatelessLifecycleTracker implements HeartbeatLifecycleTracker { * because its persisted lastActivityTs predates the wake. */ private readonly wakeStartupGraceUntilByHost = new Map() + /** + * Hosts whose durable state was already checked with a fresh read during the + * current suspension-blocked (CommunicationChannel authority loss) epoch. The + * cached Host can be stale about a `draining` write that landed just before + * the loss, so the first blocked beat bypasses the cached state once; later + * beats in the same epoch revert only on cached `draining` evidence instead of + * issuing a fresh GET and a skip log on every heartbeat of the outage. The + * entry is dropped on the first beat that is no longer blocked. + */ + private readonly suspensionBlockedFreshCheckDone = new Set() /** * Last FULLY successful heartbeat poll reported by the poller (AP-3 feed * evidence). Undefined until the first report — the silent-emitter expiry @@ -343,6 +353,7 @@ export class StatelessLifecycleTracker implements HeartbeatLifecycleTracker { this.lastBeatByHost.clear() this.lastWakeHandledGenerationByHost.clear() this.wakeStartupGraceUntilByHost.clear() + this.suspensionBlockedFreshCheckDone.clear() } /** @@ -432,12 +443,25 @@ export class StatelessLifecycleTracker implements HeartbeatLifecycleTracker { return { drain: false } } const effective = this.reconciler.getEffectiveLifecycle(host) + if (!effective.suspensionBlocked) { + this.suspensionBlockedFreshCheckDone.delete(hostRef) + } if (!effective.stateless) { // Rejected/disabled stateless lifecycle: never drain. The reconciler's // StatelessEnableRejected condition already names the reason. this.clearDrainGrace(hostRef) return { drain: false } } + if (effective.suspensionBlocked) { + this.clearDrainGrace(hostRef) + if (this.suspensionBlockedFreshCheckDone.has(hostRef)) { + await this.cancelDrainOnEvidence(hostRef, host) + } else if (await this.cancelDrainOnEvidence(hostRef, host, true)) { + // A failed fresh check is retried on the next blocked beat. + this.suspensionBlockedFreshCheckDone.add(hostRef) + } + return { drain: false } + } this.noteWakeHandledGeneration(hostRef, host) let wakePending = @@ -610,15 +634,26 @@ export class StatelessLifecycleTracker implements HeartbeatLifecycleTracker { * a suspended Host is exclusively the wake fast-path's job). Loud, not * fatal on failure — the emitter keeps beating while fenced, so the * revert retries on the next polled heartbeat. + * + * When CommunicationChannel authority is unknown, the cached Host may still + * say `active` while the durable CR already says `draining`. The caller then + * bypasses the cache precheck so this fresh-read writer can safely revert the + * durable `draining` state (and no-op for active/suspended). */ - private async cancelDrainOnEvidence(hostRef: string, host: HostCRD): Promise { - if ((host.status?.lifecycle?.state ?? 'active') !== 'draining') { - return + private async cancelDrainOnEvidence( + hostRef: string, + host: HostCRD, + bypassCachedState = false + ): Promise { + if (!bypassCachedState && (host.status?.lifecycle?.state ?? 'active') !== 'draining') { + return true } try { await this.reconciler.markHostActiveFromHeartbeat(host) + return true } catch (err) { console.error(`[StatelessLifecycleTracker] Cancel-drain revert failed for "${hostRef}":`, err) + return false } } @@ -793,7 +828,8 @@ export class StatelessLifecycleTracker implements HeartbeatLifecycleTracker { ) return } - if (!this.reconciler.getEffectiveLifecycle(host).stateless) { + const effective = this.reconciler.getEffectiveLifecycle(host) + if (!effective.stateless || effective.suspensionBlocked) { return } // KZ-R1: the informer-cached Host can be STALE here exactly as it is on the diff --git a/host-context-controller/src/types.ts b/host-context-controller/src/types.ts index 6d3ed5940c..1886a2c153 100644 --- a/host-context-controller/src/types.ts +++ b/host-context-controller/src/types.ts @@ -559,7 +559,7 @@ export interface HostLifecycleStatus { * Written by the mcp-host runtime; HCC only preserves it across writes. */ wakeHandledGeneration: number - /** Human-readable explanation when the stateless request was rejected. */ + /** Human-readable rejection or temporary suspension-block reason. */ reason?: string } diff --git a/host-context-controller/test/hostReconciler.channelReaderIntegration.test.ts b/host-context-controller/test/hostReconciler.channelReaderIntegration.test.ts index b0e5605c3d..daf609c934 100644 --- a/host-context-controller/test/hostReconciler.channelReaderIntegration.test.ts +++ b/host-context-controller/test/hostReconciler.channelReaderIntegration.test.ts @@ -205,48 +205,28 @@ describe('reconcile() — B1 integration: channel-reader Deployment shape (synce }) }) -// ── B2: unsynced cache + existing Deployment has replicas=1 → must preserve ── -describe('reconcile() — B2 integration: replica preservation when CC cache is unsynced (409 path)', () => { - it('preserves replicas=1 on replaceNamespacedDeployment body when cache is unsynced and existing Deployment is live', async () => { - // Scenario: HCC restarts mid-flight. CC cache is still loading (unsynced). - // The existing live Deployment has replicas=1 (channel-reader is running). - // reconcile() MUST NOT scale it to 0 — that would drop inbound messages. +// ── B2: unsynced scope with no retained observation must defer Deployment ── +describe('reconcile() — B2 integration: Deployment is deferred without an authoritative scope observation', () => { + it('does not mutate Deployment when the CC cache is unsynced and scope was not retained', async () => { + // Scenario: HCC restarts mid-flight. CC cache is still loading (unsynced), + // and the runtime Secret has no trusted retained scope observation. The + // reconciler must wait before changing the runtime Deployment. const { reconciler, appsApi } = createReconciler() - // Force the 409 → read-existing → replace path ONLY for the channels namespace - // channel-reader Deployment. The mcp-host Deployment (mcp-host ns) must succeed - // so that reconcile() progresses to the channel-reader step. + // A Deployment create would expose the runtime to credentials minted from + // an incomplete channel inventory; make any attempted create fail loudly. appsApi.createNamespacedDeployment.mockImplementation( ({ body }: { namespace: string; body: k8s.V1Deployment }) => { const dep = body as k8s.V1Deployment - if (dep.metadata?.namespace === 'channels') { - return Promise.reject({ code: 409 }) - } - // mcp-host Deployment (mcp-host ns) and everything else succeeds normally - return Promise.resolve({}) + return Promise.reject( + new Error(`Unexpected Deployment create in ${dep.metadata?.namespace}`) + ) } ) - // readNamespacedDeployment is called for both the host (name=alpha-host, ns=mcp-host) - // and the channel-reader (name=channel-reader-alpha-host, ns=channels). - // Return a "live replicas=1" response for the channel-reader read; for the host - // deployment read, return HCC-owned labels so reconcile() continues normally. + // Return an existing live, HCC-owned runtime Deployment. It must remain + // untouched while the reconciler waits for authoritative channel scope. appsApi.readNamespacedDeployment.mockImplementation( ({ name, namespace }: { name: string; namespace: string }) => { - if (name === 'channel-reader-alpha-host') { - return Promise.resolve({ - metadata: { - name: 'channel-reader-alpha-host', - namespace: 'channels', - resourceVersion: '77', - labels: { - 'clerum.io/host': 'alpha-host', - 'clerum.io/managed-by': 'host-context-controller', - }, - }, - spec: { replicas: 1 }, - }) - } - // Default: HCC-owned host Deployment (for readiness check) return Promise.resolve({ status: { readyReplicas: 1 }, metadata: { @@ -270,15 +250,13 @@ describe('reconcile() — B2 integration: replica preservation when CC cache is await reconciler.reconcile(makeHost()) - // The replace call body must carry replicas=1, not 0 - expect( - appsApi.replaceNamespacedDeployment.mock.calls.filter( - ([request]) => request.namespace === 'channels' - ) - ).toHaveLength(1) + expect(appsApi.replaceNamespacedDeployment).not.toHaveBeenCalled() expect(appsApi.createNamespacedDeployment).not.toHaveBeenCalled() - const replaceBody = getChannelReaderReplaceBody(appsApi) - expect(replaceBody.spec?.replicas).toBe(1) + expect(reconciler.getStatus('alpha-host')).toMatchObject({ + deployed: true, + ready: false, + message: 'Waiting for authoritative scope observation', + }) }) }) diff --git a/host-context-controller/test/hostReconciler.heartbeat.test.ts b/host-context-controller/test/hostReconciler.heartbeat.test.ts index 98e47c6607..a71694304c 100644 --- a/host-context-controller/test/hostReconciler.heartbeat.test.ts +++ b/host-context-controller/test/hostReconciler.heartbeat.test.ts @@ -113,7 +113,7 @@ function makeStatelessHost( } } -function createReconciler() { +function createReconciler(options: { isCommunicationChannelCacheSynced?: () => boolean } = {}) { const appsApi = createMockAppsApi() const coreApi = createMockCoreApi() const networkingApi = createMockNetworkingApi() @@ -140,7 +140,7 @@ function createReconciler() { // Heartbeat cases model the steady state after the watcher completed its // CommunicationChannel initial list. Cache-startup fail-closed behavior // is covered in hostReconciler.lifecycle.test.ts. - isCommunicationChannelCacheSynced: () => true, + isCommunicationChannelCacheSynced: options.isCommunicationChannelCacheSynced ?? (() => true), }) return { reconciler, appsApi, coreApi, networkingApi, rbacApi, customApi } @@ -1076,7 +1076,7 @@ describe('HostReconciler.suspendHostFromHeartbeat', () => { expect(appsApi.replaceNamespacedDeployment).not.toHaveBeenCalled() expect(host.status?.lifecycle?.state).toBe('draining') const staleLines = logSpy.mock.calls - .map(args => String(args[0])) + .map(args => JSON.parse(String(args[0])).msg as string) .filter(line => line.includes('phase=drained_report_stale')) expect(staleLines).toEqual([ expect.stringMatching( @@ -1117,7 +1117,7 @@ describe('HostReconciler.suspendHostFromHeartbeat', () => { expect(appsApi.createNamespacedDeployment).not.toHaveBeenCalled() expect(appsApi.replaceNamespacedDeployment).not.toHaveBeenCalled() const staleLines = logSpy.mock.calls - .map(args => String(args[0])) + .map(args => JSON.parse(String(args[0])).msg as string) .filter(line => line.includes('phase=drained_report_stale')) expect(staleLines).toEqual([ expect.stringMatching( @@ -1528,6 +1528,65 @@ describe('HostReconciler.markHostActiveFromHeartbeat', () => { await expect(reconciler.markHostActiveFromHeartbeat(host)).rejects.toThrow('api conflict') expect(host.status?.lifecycle?.state).toBe('draining') }) + + it('reverts a draining Host while channel authority loss blocks suspension', async () => { + // A suspension-blocked Host can still be draining from a decision made + // before the loss. Cancel-drain must stay available, otherwise the emitter + // remains fenced behind {drain:true} for the whole outage. + const { reconciler, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + const host = makeStatelessHost({ + status: { lifecycle: { state: 'draining', wakeHandledGeneration: 4 } }, + }) + customApi.getNamespacedCustomObject.mockResolvedValue( + freshHostRead({ state: 'draining', wakeHandledGeneration: 4 }) + ) + expect(reconciler.getEffectiveLifecycle(host)).toEqual({ + stateless: true, + state: 'draining', + suspensionBlocked: true, + }) + + await reconciler.markHostActiveFromHeartbeat(host) + + expect(customApi.getNamespacedCustomObject).toHaveBeenCalled() + const writes = lifecycleStatusWrites(customApi) + expect(writes).toHaveLength(1) + expect(writes[0].lifecycle).toEqual({ state: 'active', wakeHandledGeneration: 4 }) + expect(host.status?.lifecycle?.state).toBe('active') + }) + + it.each([ + { channels: 1, expectedWrites: 0, finalState: 'draining' }, + { channels: 0, expectedWrites: 1, finalState: 'active' }, + ] as const)( + 'reverts a same-revision draining Host only while its fresh read is stateless (channels=$channels)', + async ({ channels, expectedWrites, finalState }) => { + // Channel ingress makes a Host effectively not stateless, so the lifecycle + // it would revert is not the one the stateless executor owns. + const { reconciler, customApi } = createReconciler() + reconciler.setCountCommunicationChannels(() => channels) + const host = { + ...makeStatelessHost({ + status: { lifecycle: { state: 'draining', wakeHandledGeneration: 3 } }, + }), + generation: 5, + } + const fresh = freshHostRead({ state: 'draining', wakeHandledGeneration: 3 }) + customApi.getNamespacedCustomObject.mockResolvedValue({ + ...fresh, + metadata: { ...fresh.metadata, generation: 5, resourceVersion: 'rv-draining' }, + }) + + await reconciler.markHostActiveFromHeartbeat(host) + + // Liveness witness: the decision was taken on exactly one fresh read. + expect(customApi.getNamespacedCustomObject).toHaveBeenCalledOnce() + expect(customApi.patchNamespacedCustomObjectStatus).toHaveBeenCalledTimes(expectedWrites) + expect(host.status?.lifecycle?.state).toBe(finalState) + } + ) }) describe('HostReconciler heartbeat cores — fresh-read guard (cross-instance staleness)', () => { // The Host watch callback builds a brand-new HostCRD per ADDED/MODIFIED @@ -1876,7 +1935,7 @@ describe('HostReconciler.markHostDrainingFromHeartbeat — AP-1 entry-epoch guar expect(customApi.patchNamespacedCustomObjectStatus).not.toHaveBeenCalled() expect(host.status?.lifecycle?.state).toBe('active') const staleLines = logSpy.mock.calls - .map(args => String(args[0])) + .map(args => JSON.parse(String(args[0])).msg as string) .filter(line => line.includes('phase=draining_write_stale')) expect(staleLines).toEqual([ expect.stringMatching( @@ -1914,7 +1973,7 @@ describe('HostReconciler.markHostDrainingFromHeartbeat — AP-1 entry-epoch guar expect(customApi.patchNamespacedCustomObjectStatus).not.toHaveBeenCalled() const staleLines = logSpy.mock.calls - .map(args => String(args[0])) + .map(args => JSON.parse(String(args[0])).msg as string) .filter(line => line.includes('phase=draining_write_stale')) expect(staleLines).toEqual([ expect.stringMatching( @@ -2018,7 +2077,7 @@ describe('HostReconciler reconcile — stateless replicas derive from FRESH stat // pod is never scaled to 0. expect(hostDeploymentBody(appsApi, 'stateless-host').spec?.replicas).toBe(1) const guardLines = logSpy.mock.calls - .map(args => String(args[0])) + .map(args => JSON.parse(String(args[0])).msg as string) .filter(line => line.includes('disagrees with fresh')) expect(guardLines).toEqual([ '[HostReconciler] Stateless replicas guard for "stateless-host": cached lifecycle state "suspended" disagrees with fresh "active" — deriving replicas from FRESH state', diff --git a/host-context-controller/test/hostReconciler.lifecycle.test.ts b/host-context-controller/test/hostReconciler.lifecycle.test.ts index d947e96012..fc3abc1622 100644 --- a/host-context-controller/test/hostReconciler.lifecycle.test.ts +++ b/host-context-controller/test/hostReconciler.lifecycle.test.ts @@ -1,15 +1,23 @@ import { afterEach, describe, expect, it, vi } from 'vitest' import * as k8s from '@kubernetes/client-node' +import type { AdministrativeOutcomeReporter } from '../src/administrativeOutcomeReporter' import { config } from '../src/config' import { mintHostGfsToken } from '../src/gfsHostBinding' import { type EffectiveHostLifecycle, HostReconciler, + OAUTH_USER_TOKEN_SCOPE, type ResolvedSfsMount, } from '../src/hostReconciler' import type { InfrastructureTelemetryReporter } from '../src/infrastructureTelemetryReporter' import { HostContextLogger } from '../src/logger' import { issueMcpHostRuntimeTokens } from '../src/mcpHostRuntimeTokenIssuerClient' +import { + MCP_HOST_GFS_TOKEN_SECRET_KEY, + MCP_HOST_RUNTIME_TOKEN_SECRET_ACCESS_KEY, + MCP_HOST_RUNTIME_TOKEN_SECRET_CONTROL_KEY, + MCP_HOST_RUNTIME_TOKEN_SECRET_REFRESH_KEY, +} from '../src/secretFactory' import { HostCRD, HostCrdStatus } from '../src/types' import { type MockAppsApi, @@ -162,6 +170,7 @@ function createReconciler(deps?: { isCommunicationChannelCacheSynced?: () => boolean resolveContextMounts?: (host: HostCRD) => Promise infrastructureTelemetryReporter?: InfrastructureTelemetryReporter + administrativeOutcomeReporter?: AdministrativeOutcomeReporter }) { const appsApi = createMockAppsApi() const coreApi = createMockCoreApi() @@ -195,6 +204,253 @@ function runtimeTokenProvision(host: HostCRD, hasChannelIngress = false) { } } +/** + * A hand-built credential record with the identity, revision and refresh-window + * annotations only. It deliberately omits the bootstrap-state, rollout marker + * and scope hash the producer always writes, so it models a legacy record. It + * is kept for the held-runtime identity guards and the GFS renewal window, + * which read none of those. A test whose decision depends on the bootstrap or + * scope annotations uses `mintedRuntimeCredentialRecord` instead. + */ +function runtimeCredentialRecord( + host: HostCRD, + options: { + gfsRefreshBefore?: string + frontsOAuthServer?: string + hasChannelIngress?: string + hostUid?: string + managedByHost?: boolean + } = {} +) { + const data = { + [MCP_HOST_RUNTIME_TOKEN_SECRET_ACCESS_KEY]: Buffer.from('access-value').toString('base64'), + [MCP_HOST_RUNTIME_TOKEN_SECRET_REFRESH_KEY]: Buffer.from('refresh-value').toString('base64'), + [MCP_HOST_RUNTIME_TOKEN_SECRET_CONTROL_KEY]: Buffer.from('control-value').toString('base64'), + [MCP_HOST_GFS_TOKEN_SECRET_KEY]: Buffer.from('gfs-value').toString('base64'), + } + const revision = ( + HostReconciler as unknown as { + runtimeTokenSecretRevision(data: Record): string + } + ).runtimeTokenSecretRevision(data) + return { + metadata: { + name: `host-${host.name}-mcp-host-runtime-tokens`, + namespace: host.namespace, + resourceVersion: '88', + labels: + options.managedByHost === false + ? { 'clerum.io/managed-by': 'someone-else' } + : { + 'clerum.io/managed-by': 'host-context-controller', + 'clerum.io/host': host.name, + }, + annotations: { + 'clerum.io/gfs-token-host-uid': options.hostUid ?? host.uid ?? '', + 'clerum.io/runtime-token-secret-revision': revision, + 'clerum.io/runtime-token-has-channel-ingress': options.hasChannelIngress ?? 'false', + 'clerum.io/runtime-token-fronts-oauth-server': options.frontsOAuthServer ?? 'false', + 'clerum.io/gfs-token-refresh-before': + options.gfsRefreshBefore ?? '2999-01-01T00:00:00.000Z', + }, + }, + data, + } as unknown as k8s.V1Secret +} + +function trustedRuntimeDeployment( + reconciler: HostReconciler, + host: HostCRD, + options: { readyReplicas?: number; replicas?: number } = {} +) { + const deployment = reconciler.buildDeployment(host) + deployment.metadata = { + ...deployment.metadata, + uid: `deployment-${host.name}`, + resourceVersion: '91', + labels: { + ...(deployment.metadata?.labels ?? {}), + 'clerum.io/managed-by': 'host-context-controller', + 'clerum.io/host': host.name, + }, + annotations: { + ...(deployment.metadata?.annotations ?? {}), + 'clerum.io/host-uid': host.uid ?? '', + }, + } + if (deployment.spec) { + deployment.spec.replicas = options.replicas ?? 1 + } + deployment.status = { readyReplicas: options.readyReplicas ?? 1 } + return deployment +} + +function markPersistedRuntimeTrusted( + deployment: k8s.V1Deployment, + host: HostCRD, + readyReplicas = 1 +): k8s.V1Deployment { + deployment.metadata = { + ...deployment.metadata, + labels: { + ...(deployment.metadata?.labels ?? {}), + 'clerum.io/managed-by': 'host-context-controller', + 'clerum.io/host': host.name, + }, + annotations: { + ...(deployment.metadata?.annotations ?? {}), + 'clerum.io/host-uid': host.uid ?? '', + }, + } + deployment.status = { readyReplicas } + return deployment +} + +function encodedRuntimeRefreshMaterial(expiresAtMs: number): string { + const encode = (value: unknown) => Buffer.from(JSON.stringify(value)).toString('base64url') + return [ + encode({ alg: 'none', typ: 'JWT' }), + encode({ exp: Math.floor(expiresAtMs / 1000) }), + 'test-signature', + ].join('.') +} + +const PRODUCER_RUNTIME_ANNOTATIONS = [ + 'clerum.io/runtime-token-bootstrap-state', + 'clerum.io/runtime-token-rollout-required', + 'clerum.io/runtime-token-scope-hash', +] as const + +/** Fails loud when a credential fixture lacks an annotation the producer always writes. */ +function assertProducerRuntimeAnnotations(record: k8s.V1Secret): k8s.V1Secret { + const annotations = record.metadata?.annotations ?? {} + const missing = PRODUCER_RUNTIME_ANNOTATIONS.filter(key => !(key in annotations)) + if (missing.length > 0) { + throw new Error(`runtime credential fixture lacks producer annotations: ${missing.join(', ')}`) + } + return record +} + +/** + * The runtime credential record as HCC persists it, produced by one real + * `ensureMcpHostRuntimeTokenSecret` mint pass on a dedicated producer + * reconciler, so the caller's API mocks record none of the producer's calls. + * The written `stringData` is stored as base64 `data`, the way the apiserver + * returns it. With `bootstrap: 'consumed'` (the default) a second real pass + * against a Ready Deployment running the minted revision consumes the + * bootstrap, which is what a running pod's record looks like. `annotations` + * overrides only what a test drives explicitly. + */ +async function mintedRuntimeCredentialRecord( + host: HostCRD, + options: { + frontsOAuthServer?: boolean + hasChannelIngress?: boolean + bootstrap?: 'fresh' | 'consumed' + annotations?: Record + } = {} +): Promise { + const { + reconciler: producer, + appsApi, + coreApi, + } = createReconciler({ + countCommunicationChannels: () => (options.hasChannelIngress ? 1 : 0), + }) + producer.setHostFrontsOAuthServer(async () => options.frontsOAuthServer ?? false) + const issue = vi.mocked(issueMcpHostRuntimeTokens) + const issueDefaults = issue.getMockImplementation() + if (!issueDefaults) throw new Error('issueMcpHostRuntimeTokens mock has no implementation') + // A decodable refresh token, so a later reuse decision can reach `current`. + issue.mockImplementationOnce(async (...args) => ({ + ...(await issueDefaults(...args)), + refreshToken: encodedRuntimeRefreshMaterial(Date.now() + 3_600_000), + })) + coreApi.readNamespacedSecret.mockRejectedValue({ code: 404 }) + await (producer as any).ensureMcpHostRuntimeTokenSecret(host) + const created = coreApi.createNamespacedSecret.mock.calls.at(-1)?.[0].body as + | k8s.V1Secret + | undefined + if (!created?.stringData) throw new Error('producer pass wrote no runtime credential Secret') + let record: k8s.V1Secret = { + ...created, + metadata: { ...created.metadata, resourceVersion: '88' }, + data: Object.fromEntries( + Object.entries(created.stringData).map(([key, value]) => [ + key, + Buffer.from(value).toString('base64'), + ]) + ), + } + delete record.stringData + if (options.bootstrap !== 'fresh') { + const revision = record.metadata?.annotations?.['clerum.io/runtime-token-secret-revision'] + if (!revision) throw new Error('producer pass wrote no runtime credential revision') + const running = trustedRuntimeDeployment(producer, host) + running.spec!.template!.metadata!.annotations = { + ...running.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': revision, + } + appsApi.readNamespacedDeployment.mockResolvedValue(running) + coreApi.readNamespacedSecret.mockResolvedValue(record) + await (producer as any).ensureMcpHostRuntimeTokenSecret(host) + const consumed = coreApi.replaceNamespacedSecret.mock.calls.at(-1)?.[0].body as + | k8s.V1Secret + | undefined + if ( + consumed?.metadata?.annotations?.['clerum.io/runtime-token-bootstrap-state'] !== 'consumed' + ) { + throw new Error('producer pass did not consume the runtime credential bootstrap') + } + record = consumed + } + record.metadata = { + ...record.metadata, + annotations: { ...(record.metadata?.annotations ?? {}), ...options.annotations }, + } + return assertProducerRuntimeAnnotations(record) +} + +function withReadableRuntimeRefreshMaterial(record: k8s.V1Secret): k8s.V1Secret { + const material = + ( + HostReconciler as unknown as { + runtimeTokenSecretData(record: k8s.V1Secret): Record | null + } + ).runtimeTokenSecretData(record) ?? {} + return { + ...record, + data: { + ...material, + [MCP_HOST_RUNTIME_TOKEN_SECRET_REFRESH_KEY]: Buffer.from( + encodedRuntimeRefreshMaterial(Date.now() + 3_600_000) + ).toString('base64'), + }, + } +} + +/** Model persisted API state so a loss-of-authority test cannot invent a template. */ +function persistHostDeployment(appsApi: MockAppsApi, host: HostCRD, initial: k8s.V1Deployment) { + let live = structuredClone(initial) + live.metadata = { + ...live.metadata, + uid: live.metadata?.uid ?? 'deployment-uid', + resourceVersion: live.metadata?.resourceVersion ?? '73', + } + const read = appsApi.readNamespacedDeployment.getMockImplementation()! + appsApi.readNamespacedDeployment.mockImplementation(request => + request.name === host.name ? Promise.resolve(structuredClone(live)) : read(request) + ) + const replace = appsApi.replaceNamespacedDeployment.getMockImplementation()! + appsApi.replaceNamespacedDeployment.mockImplementation(async request => { + if (request.name !== host.name) return replace(request) + expect(request.body.metadata.resourceVersion).toBe(live.metadata!.resourceVersion) + live = structuredClone(request.body) + return live + }) + return () => live +} + /** The mcp-host Deployment body sent to the K8s API (excludes channel-reader). */ function hostDeploymentBody(appsApi: MockAppsApi, name: string): k8s.V1Deployment { const calls = [ @@ -291,13 +547,13 @@ describe('HostReconciler stateless lifecycle — buildDeployment replicas', () = expect(dep.spec?.replicas).toBe(1) }) - it('forces stateless+suspended to replicas=1 while the channel cache is unsynced', () => { + it('preserves stateless+suspended replicas while the channel cache is unsynced', () => { const { reconciler } = createReconciler({ isCommunicationChannelCacheSynced: () => false, }) const dep = reconciler.buildDeployment(makeStatelessHost({ status: suspendedStatus() })) - expect(dep.spec?.replicas).toBe(1) - expect(dep.spec?.template.spec?.priorityClassName).toBeUndefined() + expect(dep.spec?.replicas).toBe(0) + expect(dep.spec?.template.spec?.priorityClassName).toBe('clerum-interactive-host') }) it('derives replicas=1 for stateless+active and stateless+draining', () => { @@ -358,7 +614,10 @@ describe('HostReconciler ensureDeployment — idempotent replacement', () => { uid: 'deployment-uid', generation: 7, creationTimestamp: new Date('2026-07-10T00:00:00Z'), - annotations: { 'deployment.kubernetes.io/revision': '7' }, + annotations: { + ...deployment.metadata?.annotations, + 'deployment.kubernetes.io/revision': '7', + }, } deployment.status = { readyReplicas: 1, availableReplicas: 1 } deployment.spec = { @@ -570,194 +829,2299 @@ describe('HostReconciler ensureDeployment — idempotent replacement', () => { .mockResolvedValueOnce(existingDeployment(reconciler, host, 'revision-a')) appsApi.replaceNamespacedDeployment.mockRejectedValueOnce({ code: 409 }) - await (reconciler as any).ensureDeployment(host, [], 'revision-a') + await (reconciler as any).ensureDeployment(host, [], 'revision-a') + + expect(appsApi.readNamespacedDeployment).toHaveBeenCalledTimes(2) + expect(appsApi.replaceNamespacedDeployment).toHaveBeenCalledOnce() + }) + + it('replaces an existing Deployment when the runtime token revision changes', async () => { + const { reconciler, appsApi } = createReconciler() + const host = makeHost({ name: 'chatllm' }) + appsApi.createNamespacedDeployment.mockRejectedValue({ code: 409 }) + appsApi.readNamespacedDeployment.mockResolvedValue( + existingDeployment(reconciler, host, 'revision-a') + ) + + await (reconciler as any).ensureDeployment(host, [], 'revision-b') + + expect(appsApi.replaceNamespacedDeployment).toHaveBeenCalledOnce() + const replacement = appsApi.replaceNamespacedDeployment.mock.calls[0][0] + .body as k8s.V1Deployment + expect(replacement.metadata?.resourceVersion).toBe('42') + expect( + replacement.spec?.template.metadata?.annotations?.['clerum.io/runtime-token-revision'] + ).toBe('revision-b') + }) + + it('replaces an existing Deployment when the stateless lifecycle target changes', async () => { + const { reconciler, appsApi } = createReconciler() + const host = makeStatelessHost() + const active: EffectiveHostLifecycle = { stateless: true, state: 'active' } + const suspended: EffectiveHostLifecycle = { stateless: true, state: 'suspended' } + appsApi.createNamespacedDeployment.mockRejectedValue({ code: 409 }) + appsApi.readNamespacedDeployment.mockResolvedValue( + existingDeployment(reconciler, host, 'revision-a', active) + ) + + await (reconciler as any).ensureDeployment(host, [], 'revision-a', suspended) + + expect(appsApi.replaceNamespacedDeployment).toHaveBeenCalledOnce() + const replacement = appsApi.replaceNamespacedDeployment.mock.calls[0][0] + .body as k8s.V1Deployment + expect(replacement.metadata?.resourceVersion).toBe('42') + expect(replacement.spec?.replicas).toBe(0) + expect( + replacement.spec?.template.metadata?.annotations?.['clerum.io/runtime-token-revision'] + ).toBe('revision-a') + }) + + it('replaces a running stateful Host with the stateless template when lifecycle is enabled', async () => { + const { reconciler, appsApi, customApi } = createReconciler() + const stateful = makeHost({ name: 'transition-host' }) + + await reconciler.reconcile(stateful) + const runningStateful = structuredClone(hostDeploymentBody(appsApi, 'transition-host')) + runningStateful.metadata = { ...runningStateful.metadata, resourceVersion: '42' } + runningStateful.status = { readyReplicas: 1, availableReplicas: 1 } + expect(runningStateful.spec?.replicas).toBe(1) + expect(containerEnv(runningStateful).map(entry => entry.name)).not.toContain( + 'CLERUM_STATELESS_LIFECYCLE' + ) + + appsApi.replaceNamespacedDeployment.mockClear() + const readDeployment = appsApi.readNamespacedDeployment.getMockImplementation()! + appsApi.readNamespacedDeployment.mockImplementation(request => + request.name === stateful.name ? Promise.resolve(runningStateful) : readDeployment(request) + ) + + const stateless = makeStatelessHost({ name: 'transition-host' }) + customApi.getNamespacedCustomObject.mockResolvedValue({ + metadata: { name: stateless.name, namespace: stateless.namespace, uid: stateless.uid }, + spec: stateless.spec, + status: { lifecycle: { state: 'active', wakeHandledGeneration: 0 } }, + }) + + await reconciler.reconcile(stateless) + + const hostReplacements = appsApi.replaceNamespacedDeployment.mock.calls.filter( + ([request]) => request.name === stateful.name + ) + expect(hostReplacements).toHaveLength(1) + const replacement = hostReplacements[0][0].body as k8s.V1Deployment + expect(replacement.spec?.replicas).toBe(1) + expect(replacement.spec?.template.spec?.priorityClassName).toBe('clerum-interactive-host') + expect( + replacement.spec?.template.spec?.initContainers?.map(container => container.name) + ).toContain('workspace-layout-init') + expect(envValue(replacement, 'CLERUM_STATELESS_LIFECYCLE')).toBe('true') + expect(replacement.spec?.template.spec?.containers?.[0]?.volumeMounts).toContainEqual({ + name: 'workspace', + mountPath: '/workspace', + subPath: 'workspace', + }) + expect(replacement.spec?.template.spec?.containers?.[0]?.volumeMounts).toContainEqual({ + name: 'workspace', + mountPath: '/var/lib/clerum/state', + subPath: 'state', + }) + }) +}) + +describe('HostReconciler stateless lifecycle — env injection', () => { + it('creates the stateless template directly for a new Host', async () => { + const { reconciler, appsApi, customApi } = createReconciler() + const stateless = makeStatelessHost({ name: 'new-stateless-host' }) + customApi.getNamespacedCustomObject.mockResolvedValue({ + metadata: { name: stateless.name, namespace: stateless.namespace, uid: stateless.uid }, + spec: stateless.spec, + status: { lifecycle: { state: 'active', wakeHandledGeneration: 0 } }, + }) + + // Model a genuinely new principal Deployment, then expose its created state. + let principalCreated = false + const readDeployment = appsApi.readNamespacedDeployment.getMockImplementation()! + appsApi.readNamespacedDeployment.mockImplementation(request => { + if (request.name === stateless.name && !principalCreated) return Promise.reject({ code: 404 }) + return readDeployment(request) + }) + appsApi.createNamespacedDeployment.mockImplementation(async request => { + if (request.body.metadata?.name === stateless.name) principalCreated = true + return request.body + }) + await reconciler.reconcile(stateless) + + const creates = appsApi.createNamespacedDeployment.mock.calls.filter( + ([arg]) => (arg as { body?: k8s.V1Deployment }).body?.metadata?.name === 'new-stateless-host' + ) + const replacements = appsApi.replaceNamespacedDeployment.mock.calls.filter( + ([arg]) => (arg as { body?: k8s.V1Deployment }).body?.metadata?.name === 'new-stateless-host' + ) + expect(creates).toHaveLength(1) + expect(replacements).toHaveLength(0) + const created = hostDeploymentBody(appsApi, 'new-stateless-host') + expect(created.spec?.replicas).toBe(1) + expect(created.spec?.template.spec?.priorityClassName).toBe('clerum-interactive-host') + expect(created.spec?.template.spec?.initContainers?.map(container => container.name)).toContain( + 'workspace-layout-init' + ) + expect(envValue(created, 'CLERUM_STATELESS_LIFECYCLE')).toBe('true') + expect(created.spec?.template.spec?.containers?.[0]?.volumeMounts).toContainEqual({ + name: 'workspace', + mountPath: '/workspace', + subPath: 'workspace', + }) + expect(created.spec?.template.spec?.containers?.[0]?.volumeMounts).toContainEqual({ + name: 'workspace', + mountPath: '/var/lib/clerum/state', + subPath: 'state', + }) + }) + + it('injects the three stateless env vars when stateless is enabled', () => { + const { reconciler } = createReconciler() + const dep = reconciler.buildDeployment(makeStatelessHost()) + expect(envValue(dep, 'CLERUM_STATELESS_LIFECYCLE')).toBe('true') + expect(envValue(dep, 'CLERUM_SESSION_STORE')).toBe('sqlite') + expect(envValue(dep, 'CLERUM_SESSION_DB_DIR')).toBe('/var/lib/clerum/state') + }) + + it('does not inject the stateless env vars when stateless is off', () => { + const { reconciler } = createReconciler() + const names = containerEnv(reconciler.buildDeployment(makeHost())).map(e => e.name) + expect(names).not.toContain('CLERUM_STATELESS_LIFECYCLE') + expect(names).not.toContain('CLERUM_SESSION_STORE') + expect(names).not.toContain('CLERUM_SESSION_DB_DIR') + }) +}) + +describe('HostReconciler stateless lifecycle — rejection matrix', () => { + it('preserves a rejected stateful runtime across cold channel LIST failure and recovery', async () => { + const host = makeStatelessHost() + const previous = createReconciler({ countCommunicationChannels: () => 1 }) + previous.customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + vi.spyOn(previous.reconciler as any, 'provisionRuntimeTokenRevision').mockResolvedValue( + runtimeTokenProvision(host, true) + ) + await previous.reconciler.reconcile(host) + host.status = lifecycleStatusWrites(previous.customApi).at(-1) + expect(rejectedCondition(host.status!).reason).toBe('ActiveCommunicationChannels') + const applied = hostDeploymentBody(previous.appsApi, host.name) + applied.metadata = { ...applied.metadata, uid: 'deployment-uid', resourceVersion: '73' } + const baseline = JSON.stringify(applied.spec!.template) + expect(containerEnv(applied).some(entry => entry.name === 'CLERUM_STATELESS_LIFECYCLE')).toBe( + false + ) + let synced = false + let channels = 0 + const { reconciler, appsApi, customApi } = createReconciler({ + countCommunicationChannels: () => channels, + isCommunicationChannelCacheSynced: () => synced, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const live = persistHostDeployment(appsApi, host, applied) + vi.spyOn(reconciler as any, 'provisionRuntimeTokenRevision').mockResolvedValue( + runtimeTokenProvision(host, true) + ) + await reconciler.reconcile(host) + expect(JSON.stringify(live().spec!.template)).toBe(baseline) + expect(live().spec!.replicas).toBe(1) + expect( + appsApi.replaceNamespacedDeployment.mock.calls.filter(([r]) => r.name === host.name) + ).toHaveLength(0) + host.status = lifecycleStatusWrites(customApi).at(-1) + synced = true + channels = 1 + await reconciler.reconcile(host) + expect(JSON.stringify(live().spec!.template)).toBe(baseline) + expect(rejectedCondition(lifecycleStatusWrites(customApi).at(-1)!).reason).toBe( + 'ActiveCommunicationChannels' + ) + }) + + it('does not use a stale positive channel cache to change an applied stateless template', async () => { + const host = makeStatelessHost() + const { reconciler, appsApi, customApi } = createReconciler({ + countCommunicationChannels: () => 1, + isCommunicationChannelCacheSynced: () => false, + }) + const read = appsApi.readNamespacedDeployment.getMockImplementation()! + appsApi.readNamespacedDeployment.mockImplementation(request => + request.name === host.name ? Promise.reject({ code: 404 }) : read(request) + ) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = createReconciler().reconciler.buildDeployment(host) + applied.metadata = { ...applied.metadata, uid: 'deployment-uid', resourceVersion: '74' } + const live = persistHostDeployment(appsApi, host, applied) + await reconciler.reconcile(host) + expect(live().spec!.template).toEqual(applied.spec!.template) + expect(rejectedCondition(lifecycleStatusWrites(customApi).at(-1)!).reason).toBe( + 'CommunicationChannelCacheUnsynced' + ) + }) + + it('defers creating a missing Deployment until the channel inventory is authoritative', async () => { + let synced = false + const host = makeStatelessHost() + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => synced, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const read = appsApi.readNamespacedDeployment.getMockImplementation()! + appsApi.readNamespacedDeployment.mockImplementation(request => + request.name === host.name ? Promise.reject({ code: 404 }) : read(request) + ) + vi.mocked(issueMcpHostRuntimeTokens).mockClear() + await reconciler.reconcile(host) + expect( + appsApi.createNamespacedDeployment.mock.calls.filter( + ([r]) => r.body.metadata.name === host.name + ) + ).toHaveLength(0) + expect(vi.mocked(issueMcpHostRuntimeTokens)).not.toHaveBeenCalled() + expect(reconciler.getStatus(host.name)).toMatchObject({ deployed: false, ready: false }) + synced = true + await reconciler.reconcile(host) + expect( + appsApi.createNamespacedDeployment.mock.calls.filter( + ([r]) => r.body.metadata.name === host.name + ) + ).toHaveLength(1) + }) + + it('preserves the applied suspended replica count during cache loss', async () => { + const host = makeStatelessHost({ status: suspendedStatus() }) + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = createReconciler().reconciler.buildDeployment(host) + const current = structuredClone(applied) + current.metadata = { ...current.metadata, uid: 'deployment-uid', resourceVersion: '41' } + const baseline = JSON.stringify(current.spec!.template) + appsApi.readNamespacedDeployment.mockImplementation(async () => structuredClone(current)) + await reconciler.reconcile(host) + expect(appsApi.replaceNamespacedDeployment).not.toHaveBeenCalled() + expect(current.spec!.replicas).toBe(0) + expect(JSON.stringify(applied.spec!.template)).toBe(baseline) + expect(current.spec!.template).toEqual(applied.spec!.template) + }) + + it.each([ + 'foreign owner', + 'missing UID', + 'missing resourceVersion', + 'prior Host UID', + 'unannotated legacy', + ])('does not scale an unverified Deployment during cache loss: %s', async invalid => { + const host = makeStatelessHost({ status: suspendedStatus() }) + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = createReconciler().reconciler.buildDeployment(host) + applied.metadata = { ...applied.metadata, uid: 'deployment-uid', resourceVersion: '73' } + if (invalid === 'foreign owner') applied.metadata.labels!['clerum.io/host'] = 'another-host' + if (invalid === 'missing UID') delete applied.metadata.uid + if (invalid === 'missing resourceVersion') delete applied.metadata.resourceVersion + if (invalid === 'prior Host UID') { + applied.metadata.annotations = { + ...applied.metadata.annotations, + 'clerum.io/host-uid': 'prior-host-uid', + } + } + if (invalid === 'unannotated legacy') + delete applied.metadata.annotations?.['clerum.io/host-uid'] + appsApi.readNamespacedDeployment.mockResolvedValue(applied) + await expect(reconciler.reconcile(host)).rejects.toThrow('unverified Deployment') + expect(appsApi.replaceNamespacedDeployment).not.toHaveBeenCalled() + expect(appsApi.patchNamespacedDeployment).not.toHaveBeenCalled() + }) + + it('never scales an unannotated legacy Deployment with an existing UID and resourceVersion', async () => { + const host = makeStatelessHost({ status: suspendedStatus() }) + const { reconciler, appsApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + const legacy = reconciler.buildDeployment(host) + delete legacy.metadata?.annotations?.['clerum.io/host-uid'] + legacy.metadata = { + ...legacy.metadata, + uid: 'deployment-uid', + resourceVersion: '73', + } + appsApi.readNamespacedDeployment.mockResolvedValue(legacy) + // Derive the held lifecycle from the Host itself so the fixture cannot + // contradict the durable suspended status. + const heldLifecycle = reconciler.getEffectiveLifecycle(host) + expect(heldLifecycle).toEqual({ + stateless: true, + state: 'suspended', + suspensionBlocked: true, + }) + + await expect( + (reconciler as any).ensureDeployment(host, [], undefined, { + ...heldLifecycle, + allowScaleUpDuringHold: true, + }) + ).rejects.toThrow('unverified Deployment') + expect(appsApi.readNamespacedDeployment).toHaveBeenCalledWith({ + namespace: host.namespace, + name: host.name, + }) + expect(appsApi.replaceNamespacedDeployment).not.toHaveBeenCalled() + expect(appsApi.createNamespacedDeployment).not.toHaveBeenCalled() + }) + + it('throws held-path PVC and Service errors before any credential or Deployment mutation', async () => { + const host = makeStatelessHost({ status: suspendedStatus() }) + host.annotations = { 'clerum.io/wake-requested': '1' } + const { reconciler, appsApi, coreApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = reconciler.buildDeployment(host) + applied.spec!.replicas = 0 + applied.spec!.template!.metadata!.annotations = { + ...applied.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': 'applied-runtime-revision', + } + markPersistedRuntimeTrusted(applied, host, 0) + const live = persistHostDeployment(appsApi, host, applied) + const templateBefore = structuredClone(live().spec!.template) + const provision = vi + .spyOn(reconciler as any, 'provisionRuntimeTokenRevision') + .mockResolvedValue(runtimeTokenProvision(host)) + const ingress = vi + .spyOn(reconciler as any, 'ensureMcpHostIngressNetworkPolicy') + .mockResolvedValue('up_to_date') + + coreApi.readNamespacedPersistentVolumeClaim.mockRejectedValueOnce(new Error('PVC read failed')) + coreApi.readNamespacedService.mockRejectedValueOnce(new Error('Service read failed')) + + const rejection = await reconciler.reconcile(host).then( + () => undefined, + (error: unknown) => error + ) + + expect(rejection).toBeInstanceOf(AggregateError) + expect((rejection as AggregateError).errors.map(error => (error as Error).message)).toEqual([ + 'PVC read failed', + 'Service read failed', + ]) + // Liveness witness: the pass reached the resource reads and the policy + // phase before rejecting. + expect(coreApi.readNamespacedPersistentVolumeClaim).toHaveBeenCalledWith({ + namespace: host.namespace, + name: `${host.name}-workspace`, + }) + expect(coreApi.readNamespacedService).toHaveBeenCalledWith({ + namespace: host.namespace, + name: host.name, + }) + expect(ingress).toHaveBeenCalledWith(host) + expect(provision).not.toHaveBeenCalled() + expect(live().spec!.replicas).toBe(0) + expect(live().spec!.template).toEqual(templateBefore) + }) + + it('rethrows a single held-path PVC error unwrapped and never enters the hold', async () => { + const host = makeStatelessHost({ status: suspendedStatus() }) + host.annotations = { 'clerum.io/wake-requested': '1' } + const { reconciler, appsApi, coreApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = reconciler.buildDeployment(host) + applied.spec!.replicas = 0 + applied.spec!.template!.metadata!.annotations = { + ...applied.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': 'applied-runtime-revision', + } + markPersistedRuntimeTrusted(applied, host, 0) + const live = persistHostDeployment(appsApi, host, applied) + const templateBefore = structuredClone(live().spec!.template) + const provision = vi + .spyOn(reconciler as any, 'provisionRuntimeTokenRevision') + .mockResolvedValue(runtimeTokenProvision(host)) + const ingress = vi + .spyOn(reconciler as any, 'ensureMcpHostIngressNetworkPolicy') + .mockResolvedValue('up_to_date') + const ready = vi.spyOn(reconciler as any, 'checkDeploymentReady') + + coreApi.readNamespacedPersistentVolumeClaim.mockRejectedValueOnce(new Error('PVC read failed')) + + const rejection = await reconciler.reconcile(host).then( + () => undefined, + (error: unknown) => error + ) + + expect(rejection).toBeInstanceOf(Error) + expect(rejection).not.toBeInstanceOf(AggregateError) + expect((rejection as Error).message).toBe('PVC read failed') + // Liveness witness: the pass read the PVC and Service and ran the policy + // phase, the last step before the throw. The hold path (which would mint + // credentials for this held wake and poll readiness) never ran. + expect(coreApi.readNamespacedPersistentVolumeClaim).toHaveBeenCalledWith({ + namespace: host.namespace, + name: `${host.name}-workspace`, + }) + expect(coreApi.readNamespacedService).toHaveBeenCalledWith({ + namespace: host.namespace, + name: host.name, + }) + expect(ingress).toHaveBeenCalledWith(host) + expect(provision).not.toHaveBeenCalled() + expect(ready).not.toHaveBeenCalled() + expect(appsApi.replaceNamespacedDeployment).not.toHaveBeenCalled() + expect(live().spec!.replicas).toBe(0) + expect(live().spec!.template).toEqual(templateBefore) + }) + + it('waits for a verified applied runtime when a wake is requested and the live Deployment has no applied-runtime annotation', async () => { + const host = makeStatelessHost({ status: suspendedStatus() }) + host.annotations = { 'clerum.io/wake-requested': '1' } + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + // A legacy Deployment: owned by the Host and at zero replicas, but its pod + // template carries no runtime-token-revision annotation. + const legacy = reconciler.buildDeployment(host) + legacy.spec!.replicas = 0 + delete legacy.spec!.template!.metadata!.annotations?.['clerum.io/runtime-token-revision'] + markPersistedRuntimeTrusted(legacy, host, 0) + const live = persistHostDeployment(appsApi, host, legacy) + const templateBefore = structuredClone(live().spec!.template) + const provision = vi + .spyOn(reconciler as any, 'provisionRuntimeTokenRevision') + .mockResolvedValue(runtimeTokenProvision(host)) + const poll = vi.spyOn(reconciler as any, 'pollReadiness').mockImplementation(() => undefined) + + await reconciler.reconcile(host) + + expect(reconciler.getStatus(host.name)).toEqual({ + deployed: true, + ready: false, + message: + 'Waiting for a verified applied runtime before waking during CommunicationChannel inventory loss', + }) + // Liveness witness: the held pass read the live Deployment before deciding. + expect(appsApi.readNamespacedDeployment).toHaveBeenCalledWith({ + namespace: host.namespace, + name: host.name, + }) + expect(provision).not.toHaveBeenCalled() + expect(poll).not.toHaveBeenCalled() + expect(appsApi.replaceNamespacedDeployment).not.toHaveBeenCalled() + expect(live().spec!.replicas).toBe(0) + expect(live().spec!.template).toEqual(templateBefore) + }) + + it('converges the held runtime boundary and reports policy failures without polling', async () => { + // An active, not-Ready held runtime would be polled if the boundary were + // complete, so only the policy failure keeps the poll from running. + const host = makeStatelessHost({ + status: { lifecycle: { state: 'active', wakeHandledGeneration: 1 } }, + }) + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = reconciler.buildDeployment(host) + markPersistedRuntimeTrusted(applied, host, 0) + const live = persistHostDeployment(appsApi, host, applied) + const readySpy = vi.spyOn(reconciler as any, 'checkDeploymentReady') + const templateBefore = structuredClone(live().spec!.template) + const replicasBefore = live().spec!.replicas + const poll = vi.spyOn(reconciler as any, 'pollReadiness') + + const ingress = vi + .spyOn(reconciler as any, 'ensureMcpHostIngressNetworkPolicy') + .mockRejectedValueOnce(new Error('ingress policy failed')) + const gfsEgress = vi + .spyOn(reconciler as any, 'ensureMcpHostGfsEgressNetworkPolicy') + .mockResolvedValue('up_to_date') + const codexEgress = vi + .spyOn(reconciler as any, 'reconcileMcpHostCodexProxyEgressNetworkPolicy') + .mockResolvedValue('up_to_date') + + await reconciler.reconcile(host) + + expect(ingress).toHaveBeenCalledOnce() + expect(ingress).toHaveBeenCalledWith(host) + expect(gfsEgress).toHaveBeenCalledOnce() + expect(codexEgress).toHaveBeenCalledOnce() + expect(live().spec!.replicas).toBe(replicasBefore) + expect(live().spec!.template).toEqual(templateBefore) + expect(reconciler.getStatus(host.name)).toMatchObject({ + deployed: true, + ready: false, + message: expect.stringContaining('mcp-host ingress: ingress policy failed'), + }) + expect(reconciler.getStatus(host.name)?.message).not.toContain('Host PVC') + expect(reconciler.getStatus(host.name)?.message).not.toContain('Host Service') + expect(reconciler.getStatus(host.name)?.message).not.toContain('GFS egress') + expect(reconciler.getStatus(host.name)?.message).not.toContain('Codex proxy egress') + // Liveness witness: the hold reached its readiness check and found the + // runtime not Ready, the state in which a complete boundary is polled. + expect(readySpy).toHaveBeenCalledWith(host.name, host.namespace) + await expect(readySpy.mock.results[0].value).resolves.toBe(false) + expect(poll).not.toHaveBeenCalled() + }) + + it('polls readiness after a held wake scales the Host up from zero', async () => { + const host = makeStatelessHost({ status: suspendedStatus() }) + host.annotations = { 'clerum.io/wake-requested': '1' } + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = reconciler.buildDeployment(host) + applied.spec!.replicas = 0 + applied.spec!.template!.metadata!.annotations = { + ...applied.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': 'applied-runtime-revision', + } + markPersistedRuntimeTrusted(applied, host, 0) + const live = persistHostDeployment(appsApi, host, applied) + vi.spyOn(reconciler as any, 'provisionRuntimeTokenRevision').mockResolvedValue( + runtimeTokenProvision(host) + ) + const poll = vi.spyOn(reconciler as any, 'pollReadiness').mockImplementation(() => undefined) + + await reconciler.reconcile(host) + + expect(live().spec!.replicas).toBe(1) + expect(reconciler.getStatus(host.name)).toEqual({ + deployed: true, + ready: false, + message: 'CommunicationChannel inventory unavailable; preserved applied runtime is not Ready', + }) + expect(poll).toHaveBeenCalledOnce() + expect(poll).toHaveBeenCalledWith(host.name, host.namespace) + }) + + it.each([ + { + runtime: 'Ready', + readyReplicas: 1, + ready: true, + message: 'CommunicationChannel inventory unavailable; preserving applied runtime', + }, + { + runtime: 'not Ready', + readyReplicas: 0, + ready: false, + message: 'CommunicationChannel inventory unavailable; preserved applied runtime is not Ready', + }, + ])( + 'holds a held wake on a suspended Host whose applied runtime still runs ($runtime)', + async ({ readyReplicas, ready, message }) => { + // The suspension reached the Host status but its scale-down never reached + // the Deployment, so the applied runtime still runs at one replica. + const host = makeStatelessHost({ status: suspendedStatus() }) + host.annotations = { 'clerum.io/wake-requested': '1' } + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = reconciler.buildDeployment(host) + applied.spec!.replicas = 1 + applied.spec!.template!.metadata!.annotations = { + ...applied.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': 'applied-runtime-revision', + } + markPersistedRuntimeTrusted(applied, host, readyReplicas) + const live = persistHostDeployment(appsApi, host, applied) + const templateBefore = structuredClone(live().spec!.template) + const provision = vi + .spyOn(reconciler as any, 'provisionRuntimeTokenRevision') + .mockResolvedValue(runtimeTokenProvision(host)) + const readySpy = vi.spyOn(reconciler as any, 'checkDeploymentReady') + const poll = vi.spyOn(reconciler as any, 'pollReadiness').mockImplementation(() => undefined) + + await reconciler.reconcile(host) + + // Liveness witness: the held pass reached its readiness check. + expect(readySpy).toHaveBeenCalledWith(host.name, host.namespace) + await expect(readySpy.mock.results[0].value).resolves.toBe(readyReplicas > 0) + expect(reconciler.getStatus(host.name)).toEqual({ deployed: true, ready, message }) + // A running runtime renews only its GFS credential; it never mints for + // the wake, and a not-Ready one is polled instead of renewed. + expect(provision.mock.calls).toEqual( + readyReplicas > 0 ? [[host, { targetSuspended: false, refreshGfsOnly: true }]] : [] + ) + expect(poll.mock.calls).toEqual(readyReplicas > 0 ? [] : [[host.name, host.namespace]]) + expect(live().spec!.replicas).toBe(1) + expect(live().spec!.template).toEqual(templateBefore) + } + ) + + it('does not report a Deployment applied when create conflict is followed by a missing read', async () => { + const host = makeStatelessHost() + const { reconciler, appsApi } = createReconciler() + appsApi.readNamespacedDeployment + .mockRejectedValueOnce({ code: 404 }) + .mockRejectedValueOnce({ code: 404 }) + appsApi.createNamespacedDeployment.mockRejectedValueOnce({ code: 409 }) + + await expect((reconciler as any).ensureDeployment(host, [], 'runtime-revision')).resolves.toBe( + false + ) + expect(appsApi.replaceNamespacedDeployment).not.toHaveBeenCalled() + }) + + it('refreshes only the GFS token for a running held Host during cache loss', async () => { + const host = makeStatelessHost({ + status: { lifecycle: { state: 'active', wakeHandledGeneration: 1 } }, + }) + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = reconciler.buildDeployment(host) + applied.spec!.template.metadata!.annotations = { 'example.org/applied': 'keep-exactly' } + markPersistedRuntimeTrusted(applied, host) + const live = persistHostDeployment(appsApi, host, applied) + const provision = vi + .spyOn(reconciler as any, 'provisionRuntimeTokenRevision') + .mockResolvedValue(runtimeTokenProvision(host)) + await reconciler.reconcile(host) + expect(provision).toHaveBeenCalledOnce() + expect(provision).toHaveBeenCalledWith(host, { + refreshGfsOnly: true, + targetSuspended: false, + }) + expect(live().spec!.replicas).toBe(1) + expect(live().spec!.template).toEqual(applied.spec!.template) + expect(rejectedCondition(lifecycleStatusWrites(customApi).at(-1)!).reason).toBe( + 'CommunicationChannelCacheUnsynced' + ) + }) + + it('fails closed when held credential renewal cannot verify the runtime identity', async () => { + const host = makeStatelessHost({ + status: { lifecycle: { state: 'active', wakeHandledGeneration: 1 } }, + }) + const { reconciler, appsApi, coreApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = reconciler.buildDeployment(host) + markPersistedRuntimeTrusted(applied, host) + const live = persistHostDeployment(appsApi, host, applied) + const templateBefore = structuredClone(live().spec!.template) + // The record was minted for a previous Host incarnation, so the real + // refreshGfsOnly provision refuses to renew it and returns null. + coreApi.readNamespacedSecret.mockResolvedValue( + runtimeCredentialRecord(host, { hostUid: 'prior-host-uid' }) as any + ) + const runtimeSecretName = `host-${host.name}-mcp-host-runtime-tokens` + vi.mocked(issueMcpHostRuntimeTokens).mockClear() + + await reconciler.reconcile(host) + + // Liveness witness: the held renewal read the runtime credential record. + expect(coreApi.readNamespacedSecret).toHaveBeenCalledWith({ + name: runtimeSecretName, + namespace: host.namespace, + }) + expect(reconciler.getStatus(host.name)).toEqual({ + deployed: true, + ready: false, + message: 'Held runtime credential renewal is not safely available', + }) + expect(vi.mocked(issueMcpHostRuntimeTokens)).not.toHaveBeenCalled() + expect( + coreApi.replaceNamespacedSecret.mock.calls.filter( + ([request]) => (request as { name?: string }).name === runtimeSecretName + ) + ).toHaveLength(0) + expect(appsApi.replaceNamespacedDeployment).not.toHaveBeenCalled() + expect(live().spec!.replicas).toBe(1) + expect(live().spec!.template).toEqual(templateBefore) + }) + + it('keeps a suspended Host at zero replicas during cache loss without a pending wake', async () => { + const host = makeStatelessHost({ status: suspendedStatus(3) }) + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = reconciler.buildDeployment(host) + applied.spec!.template.metadata!.annotations = { 'example.org/applied': 'keep-exactly' } + applied.spec!.replicas = 0 + const live = persistHostDeployment(appsApi, host, applied) + const provision = vi + .spyOn(reconciler as any, 'provisionRuntimeTokenRevision') + .mockResolvedValue(runtimeTokenProvision(host)) + + await reconciler.reconcile(host) + + expect(provision).not.toHaveBeenCalled() + expect(live().spec!.replicas).toBe(0) + expect(live().spec!.template).toEqual(applied.spec!.template) + expect(lifecycleStatusWrites(customApi).at(-1)?.lifecycle).toMatchObject({ + state: 'suspended', + wakeHandledGeneration: 3, + }) + }) + + it('keeps a preserved suspended wake target at zero replicas during cache loss without a pending wake', async () => { + // This Deployment IS a preserved wake target (applied runtime revision at + // zero replicas), so only the missing wake keeps the hold from minting + // bootstrap credentials and scaling it up. + const host = makeStatelessHost({ status: suspendedStatus() }) + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = reconciler.buildDeployment(host) + applied.spec!.replicas = 0 + applied.spec!.template!.metadata!.annotations = { + ...applied.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': 'applied-runtime-revision', + } + markPersistedRuntimeTrusted(applied, host, 0) + const live = persistHostDeployment(appsApi, host, applied) + const templateBefore = structuredClone(live().spec!.template) + const provision = vi + .spyOn(reconciler as any, 'provisionRuntimeTokenRevision') + .mockResolvedValue(runtimeTokenProvision(host)) + const ensureDeployment = vi.spyOn(reconciler as any, 'ensureDeployment') + const poll = vi.spyOn(reconciler as any, 'pollReadiness').mockImplementation(() => undefined) + + await reconciler.reconcile(host) + + // Liveness witness: the pass read the live Deployment and applied the hold + // with the suspended, cache-loss lifecycle. + expect( + appsApi.readNamespacedDeployment.mock.calls.filter(([request]) => request.name === host.name) + .length + ).toBeGreaterThan(0) + expect(ensureDeployment).toHaveBeenCalledOnce() + expect(ensureDeployment.mock.calls[0][3]).toMatchObject({ + state: 'suspended', + suspensionBlocked: true, + allowScaleUpDuringHold: false, + }) + await expect(ensureDeployment.mock.results[0].value).resolves.toBe(true) + expect(reconciler.getStatus(host.name)).toEqual({ + deployed: true, + ready: false, + message: 'CommunicationChannel inventory unavailable; preserving suspended Host replicas', + }) + expect(provision).not.toHaveBeenCalled() + expect(poll).not.toHaveBeenCalled() + expect(live().spec!.replicas).toBe(0) + expect(live().spec!.template).toEqual(templateBefore) + }) + + it('does not poll readiness when the held Deployment disappears before it is applied', async () => { + // An active, not-Ready held runtime with a complete boundary is polled, so + // only the unapplied Deployment keeps the poll from running. + const host = makeStatelessHost({ + status: { lifecycle: { state: 'active', wakeHandledGeneration: 1 } }, + }) + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = reconciler.buildDeployment(host) + markPersistedRuntimeTrusted(applied, host, 0) + persistHostDeployment(appsApi, host, applied) + const persistedRead = appsApi.readNamespacedDeployment.getMockImplementation()! + let deleted = false + appsApi.readNamespacedDeployment.mockImplementation(request => + deleted && request.name === host.name ? Promise.reject({ code: 404 }) : persistedRead(request) + ) + const realEnsureDeployment = (reconciler as any).ensureDeployment.bind(reconciler) + const ensureDeployment = vi + .spyOn(reconciler as any, 'ensureDeployment') + .mockImplementation(async (...args: unknown[]) => { + // The Deployment is deleted after the hold read it and before it is applied. + deleted = true + return realEnsureDeployment(...args) + }) + const readySpy = vi.spyOn(reconciler as any, 'checkDeploymentReady') + const poll = vi.spyOn(reconciler as any, 'pollReadiness').mockImplementation(() => undefined) + + await reconciler.reconcile(host) + + // Liveness witness: the hold reached ensureDeployment, whose re-read found + // the Deployment gone and left it unapplied. + expect(ensureDeployment).toHaveBeenCalledOnce() + await expect(ensureDeployment.mock.results[0].value).resolves.toBe(false) + expect(reconciler.getStatus(host.name)).toEqual({ + deployed: false, + ready: false, + message: 'Waiting for CommunicationChannel inventory before creating runtime', + }) + expect( + appsApi.createNamespacedDeployment.mock.calls.filter( + ([request]) => request.body.metadata.name === host.name + ) + ).toHaveLength(0) + expect(readySpy).not.toHaveBeenCalled() + expect(poll).not.toHaveBeenCalled() + }) + + it('uses the fenced replica-only update for a pending wake during cache loss', async () => { + const host = makeStatelessHost({ status: suspendedStatus() }) + host.annotations = { 'clerum.io/wake-requested': '1' } + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = createReconciler().reconciler.buildDeployment(host) + applied.spec!.replicas = 0 + applied.spec!.template!.metadata!.annotations = { + ...applied.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': 'applied-runtime-revision', + } + markPersistedRuntimeTrusted(applied, host, 0) + const live = persistHostDeployment(appsApi, host, applied) + const provision = vi + .spyOn(reconciler as any, 'provisionRuntimeTokenRevision') + .mockResolvedValue(runtimeTokenProvision(host)) + await reconciler.reconcile(host) + expect(provision).toHaveBeenCalledOnce() + expect(provision).toHaveBeenCalledWith(host, { + forceFreshForWake: true, + targetSuspended: false, + preserveDeploymentTemplateOnWake: true, + }) + expect(provision.mock.invocationCallOrder[0]).toBeLessThan( + appsApi.replaceNamespacedDeployment.mock.invocationCallOrder.at(-1)! + ) + expect(appsApi.patchNamespacedDeployment).not.toHaveBeenCalled() + expect(live().spec!.replicas).toBe(1) + expect(live().spec!.template).toEqual(applied.spec!.template) + }) + + it('does not apply a channel rejection after that inventory loses authority mid-reconcile', async () => { + let synced = true + const host = makeStatelessHost() + const { reconciler, appsApi, customApi, networkingApi } = createReconciler({ + countCommunicationChannels: () => 1, + isCommunicationChannelCacheSynced: () => synced, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = createReconciler().reconciler.buildDeployment(host) + const live = persistHostDeployment(appsApi, host, applied) + const readPolicy = networkingApi.readNamespacedNetworkPolicy.getMockImplementation()! + networkingApi.readNamespacedNetworkPolicy.mockImplementation(async request => { + synced = false + return readPolicy(request) + }) + await reconciler.reconcile(host) + expect(live().spec!.template).toEqual(applied.spec!.template) + expect(rejectedCondition(lifecycleStatusWrites(customApi).at(-1)!).reason).toBe( + 'CommunicationChannelCacheUnsynced' + ) + }) + + it('preserves the template when authority is lost during the final asynchronous scope lookup', async () => { + let synced = true + let lookup = 0 + const host = makeStatelessHost() + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => synced, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = reconciler.buildDeployment(host) + applied.spec!.template.metadata!.annotations = { 'example.org/applied': 'keep-exactly' } + const live = persistHostDeployment(appsApi, host, applied) + vi.spyOn(reconciler as any, 'provisionRuntimeTokenRevision').mockResolvedValue( + runtimeTokenProvision(host) + ) + // The final lookup loses channel authority and its McpServer read fails too: + // neither an unobservable OAuth answer nor the lost authority may commit + // the template. + reconciler.setHostFrontsOAuthServer(async () => { + if (++lookup === 1) return false + synced = false + throw new Error('mcp-server watch retired') + }) + await reconciler.reconcile(host) + // One observed lookup, then the final lookup's three attempts. + expect(lookup).toBe(4) + expect(live().spec!.template).toEqual(applied.spec!.template) + expect(rejectedCondition(lifecycleStatusWrites(customApi).at(-1)!).reason).toBe( + 'CommunicationChannelCacheUnsynced' + ) + }) + + it('preserves an independently confirmed desktop rejection while the channel cache is unknown', async () => { + const host = makeStatelessHost({ spec: { desktop: { browser: true } } }) + const { reconciler, appsApi, coreApi, customApi } = createReconciler({ + countCommunicationChannels: () => 1, + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + coreApi.readNamespacedSecret.mockResolvedValue({ + metadata: { + resourceVersion: '1', + labels: { + 'clerum.io/managed-by': 'host-context-controller', + 'clerum.io/host': host.name, + }, + annotations: { + 'clerum.io/gfs-token-host-uid': host.uid, + 'clerum.io/runtime-token-has-channel-ingress': 'true', + 'clerum.io/runtime-token-fronts-oauth-server': 'false', + }, + }, + data: {}, + } as any) + await reconciler.reconcile(host) + const condition = rejectedCondition(lifecycleStatusWrites(customApi).at(-1)!) + expect(condition.reason).toBe('DesktopEnabled') + expect(condition.status).toBe('True') + expect( + containerEnv(hostDeploymentBody(appsApi, host.name)).some( + entry => entry.name === 'CLERUM_STATELESS_LIFECYCLE' + ) + ).toBe(false) + }) + + it('defers a confirmed desktop rejection when no trusted scope observation exists', async () => { + const host = makeStatelessHost({ spec: { desktop: { browser: true } } }) + const { reconciler, appsApi, customApi } = createReconciler({ + countCommunicationChannels: () => 1, + isCommunicationChannelCacheSynced: () => false, + }) + const readDeployment = appsApi.readNamespacedDeployment.getMockImplementation()! + appsApi.readNamespacedDeployment.mockImplementation(request => + request.name === host.name ? Promise.reject({ code: 404 }) : readDeployment(request) + ) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + + await reconciler.reconcile(host) + + expect(rejectedCondition(lifecycleStatusWrites(customApi).at(-1)!).reason).toBe( + 'DesktopEnabled' + ) + expect( + [ + ...appsApi.createNamespacedDeployment.mock.calls, + ...appsApi.replaceNamespacedDeployment.mock.calls, + ].filter( + ([request]) => (request as { body?: k8s.V1Deployment }).body?.metadata?.name === host.name + ) + ).toHaveLength(0) + expect(reconciler.getStatus(host.name)).toMatchObject({ deployed: false, ready: false }) + }) + + it('does not let a failed live probe narrow a retained OAuth grant before its renewal window', async () => { + const host = makeStatelessHost() + const { reconciler, coreApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => true, + }) + coreApi.readNamespacedSecret.mockResolvedValue( + await mintedRuntimeCredentialRecord(host, { frontsOAuthServer: true }) + ) + const oauthResolver = vi.fn(async (): Promise => { + throw new Error('oauth lookup unavailable') + }) + reconciler.setHostFrontsOAuthServer(oauthResolver) + vi.mocked(issueMcpHostRuntimeTokens).mockClear() + + const provision = await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host) + + // Liveness witness: the record was read and the live probe ran all its attempts. + expect(coreApi.readNamespacedSecret).toHaveBeenCalled() + expect(oauthResolver).toHaveBeenCalledTimes(3) + expect(provision).toBeNull() + expect(vi.mocked(issueMcpHostRuntimeTokens)).not.toHaveBeenCalled() + expect(coreApi.replaceNamespacedSecret).not.toHaveBeenCalled() + expect(coreApi.createNamespacedSecret).not.toHaveBeenCalled() + }) + + it('uses the retained channel observation but a live OAuth observation while channel authority is unavailable', async () => { + const host = makeStatelessHost() + const countChannels = vi.fn(() => 0) + const { reconciler, coreApi } = createReconciler({ + countCommunicationChannels: countChannels, + isCommunicationChannelCacheSynced: () => false, + }) + coreApi.readNamespacedSecret.mockResolvedValue( + await mintedRuntimeCredentialRecord(host, { + frontsOAuthServer: false, + hasChannelIngress: true, + }) + ) + const oauthResolver = vi.fn(async () => true) + reconciler.setHostFrontsOAuthServer(oauthResolver) + vi.mocked(issueMcpHostRuntimeTokens).mockClear() + + const provision = await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host) + + expect(provision).not.toBeNull() + expect(oauthResolver).toHaveBeenCalledWith(host) + expect(countChannels).not.toHaveBeenCalled() + expect(vi.mocked(issueMcpHostRuntimeTokens).mock.calls.at(-1)?.[2]).toContain( + OAUTH_USER_TOKEN_SCOPE + ) + const write = coreApi.replaceNamespacedSecret.mock.calls.at(-1)?.[0].body as k8s.V1Secret + expect(write.metadata?.annotations).toMatchObject({ + 'clerum.io/runtime-token-fronts-oauth-server': 'true', + 'clerum.io/runtime-token-has-channel-ingress': 'true', + }) + }) + + describe('McpServer OAuth observation unavailable while channel authority is synced', () => { + /** + * A Host whose Ready pod runs a consumed credential that grants + * oauth:user-token, over stateful Secret and Deployment mocks. + */ + async function oauthGrantedRuntime(annotations: Record = {}) { + const host = makeHost() + const { reconciler, appsApi, coreApi } = createReconciler() + let record = await mintedRuntimeCredentialRecord(host, { + frontsOAuthServer: true, + annotations, + }) + const secretName = record.metadata!.name! + const readSecret = coreApi.readNamespacedSecret.getMockImplementation()! + coreApi.readNamespacedSecret.mockImplementation(request => + request.name === secretName ? Promise.resolve(structuredClone(record)) : readSecret(request) + ) + coreApi.replaceNamespacedSecret.mockImplementation(async request => { + if (request.name === secretName) record = structuredClone(request.body as k8s.V1Secret) + return request.body + }) + let live = trustedRuntimeDeployment(reconciler, host) + live.spec!.template!.metadata!.annotations = { + ...live.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': + record.metadata!.annotations!['clerum.io/runtime-token-secret-revision'], + } + const readDeployment = appsApi.readNamespacedDeployment.getMockImplementation()! + appsApi.readNamespacedDeployment.mockImplementation(request => + request.name === host.name + ? Promise.resolve(structuredClone(live)) + : readDeployment(request) + ) + const replaceDeployment = appsApi.replaceNamespacedDeployment.getMockImplementation()! + appsApi.replaceNamespacedDeployment.mockImplementation(async request => { + if (request.name !== host.name) return replaceDeployment(request) + // The replaced pod stays Ready, as a running runtime does. + live = { ...structuredClone(request.body), status: { readyReplicas: 1 } } + return live + }) + const hostDeploymentWrites = () => + [ + ...appsApi.createNamespacedDeployment.mock.calls, + ...appsApi.replaceNamespacedDeployment.mock.calls, + ...appsApi.patchNamespacedDeployment.mock.calls, + ].filter(([request]) => (request as { name?: string }).name === host.name) + return { + host, + reconciler, + appsApi, + coreApi, + record: () => record, + hostDeploymentWrites, + } + } + + it('keeps oauth:user-token and does not roll the Deployment when the McpServer probe fails between relists', async () => { + const { host, reconciler, appsApi, record, hostDeploymentWrites } = + await oauthGrantedRuntime() + const oauthResolver = vi.fn(async (): Promise => true) + reconciler.setHostFrontsOAuthServer(oauthResolver) + const issue = vi.mocked(issueMcpHostRuntimeTokens) + issue.mockClear() + for (let pass = 1; pass <= 3; pass++) await reconciler.reconcile(host) + // The steady state is stable: no runtime mint while the grant is observed. + expect(issue).not.toHaveBeenCalled() + + oauthResolver.mockImplementation(async () => { + throw new Error('mcp-server watch retired') + }) + oauthResolver.mockClear() + appsApi.createNamespacedDeployment.mockClear() + appsApi.replaceNamespacedDeployment.mockClear() + appsApi.patchNamespacedDeployment.mockClear() + const warn = vi.spyOn(HostContextLogger.prototype, 'warn').mockImplementation(() => undefined) + try { + await reconciler.reconcile(host) + + expect(issue).not.toHaveBeenCalled() + expect(hostDeploymentWrites()).toHaveLength(0) + expect(record().metadata!.annotations).toMatchObject({ + 'clerum.io/runtime-token-fronts-oauth-server': 'true', + 'clerum.io/runtime-token-rollout-required': 'false', + }) + // Liveness witness: the live probe ran all its attempts and the reconcile + // deferred with the reason it could not decide. + expect(oauthResolver).toHaveBeenCalledTimes(3) + expect(warn).toHaveBeenCalledWith( + 'deferring runtime token decision: OAuth observation unavailable and retained scope grants oauth:user-token', + { + host: host.name, + namespace: host.namespace, + observation: 'frontsOAuthServer', + attempts: 3, + err: 'mcp-server watch retired', + } + ) + expect(reconciler.getStatus(host.name)).toMatchObject({ + deployed: true, + ready: false, + message: 'Waiting for authoritative scope observation', + }) + } finally { + warn.mockRestore() + } + }) + + it('lets the Deployment drift guard keep an observed grant when its own McpServer read fails', async () => { + const { host, reconciler, record } = await oauthGrantedRuntime() + const oauthResolver = vi.fn(async (): Promise => true) + reconciler.setHostFrontsOAuthServer(oauthResolver) + const issue = vi.mocked(issueMcpHostRuntimeTokens) + issue.mockClear() + for (let pass = 1; pass <= 3; pass++) await reconciler.reconcile(host) + expect(issue).not.toHaveBeenCalled() + const steadyStatus = structuredClone(reconciler.getStatus(host.name)) + + // Issuance observes the grant; every later read (the drift guard's) fails. + oauthResolver.mockClear() + oauthResolver.mockImplementation(async () => { + if (oauthResolver.mock.calls.length === 1) return true + throw new Error('mcp-server watch retired') + }) + await reconciler.reconcile(host) + + // Liveness witness: the drift guard read after issuance and exhausted its retries. + expect(oauthResolver.mock.calls.length).toBeGreaterThanOrEqual(4) + expect(issue).not.toHaveBeenCalled() + expect(reconciler.getStatus(host.name)).toEqual(steadyStatus) + expect(record().metadata!.annotations).toMatchObject({ + 'clerum.io/runtime-token-fronts-oauth-server': 'true', + 'clerum.io/runtime-token-rollout-required': 'false', + }) + }) + + it('falls back to a fail-closed mint when the OAuth observation is unavailable inside the renewal window', async () => { + const { host, reconciler, record } = await oauthGrantedRuntime({ + 'clerum.io/runtime-token-refresh-before': '2000-01-01T00:00:00.000Z', + }) + const oauthResolver = vi.fn(async (): Promise => { + throw new Error('mcp-server watch retired') + }) + reconciler.setHostFrontsOAuthServer(oauthResolver) + const issue = vi.mocked(issueMcpHostRuntimeTokens) + issue.mockClear() + + const provision = await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host) + + // Liveness witness: the live probe ran all its attempts before the mint. + expect(oauthResolver).toHaveBeenCalledTimes(3) + expect(provision).not.toBeNull() + expect(issue).toHaveBeenCalledOnce() + expect(issue.mock.calls[0]?.[2]).not.toContain(OAUTH_USER_TOKEN_SCOPE) + expect(record().metadata!.annotations).toMatchObject({ + 'clerum.io/runtime-token-fronts-oauth-server': 'false', + 'clerum.io/runtime-token-rollout-required': 'true', + }) + }) + }) + + describe('held wake OAuth scope during channel cache loss', () => { + function heldZeroReplicaWake(options: { + cacheSynced: boolean + retainedFrontsOAuth: 'true' | 'false' + resolveFrontsOAuth: () => Promise + }) { + const host = makeStatelessHost({ + status: { lifecycle: { state: 'active', wakeHandledGeneration: 1 } }, + }) + const { reconciler, appsApi, coreApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => options.cacheSynced, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const runtimeRecord = runtimeCredentialRecord(host, { + frontsOAuthServer: options.retainedFrontsOAuth, + }) + const runtimeRecordReads: string[] = [] + coreApi.readNamespacedSecret.mockImplementation(({ name }) => { + if (name?.includes('runtime-tokens')) { + runtimeRecordReads.push(name) + return Promise.resolve(runtimeRecord) + } + return Promise.resolve({ metadata: { resourceVersion: '1' }, data: {} } as any) + }) + const applied = reconciler.buildDeployment(host) + applied.spec!.replicas = 0 + applied.spec!.template!.metadata!.annotations = { + ...applied.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': 'applied-runtime-revision', + } + markPersistedRuntimeTrusted(applied, host, 0) + const live = persistHostDeployment(appsApi, host, applied) + const oauthResolver = vi.fn(options.resolveFrontsOAuth) + reconciler.setHostFrontsOAuthServer(oauthResolver) + vi.mocked(issueMcpHostRuntimeTokens).mockClear() + return { host, reconciler, coreApi, live, oauthResolver, runtimeRecordReads } + } + + function runtimeTokenWrite(coreApi: ReturnType): k8s.V1Secret { + const write = coreApi.replaceNamespacedSecret.mock.calls.find(([request]) => + request.name?.includes('runtime-tokens') + )?.[0].body as k8s.V1Secret | undefined + if (!write) throw new Error('No runtime token Secret write recorded') + return write + } + + it.each([false, true])( + 'mints without oauth:user-token when the OAuth mcp-server was removed (cache synced: %s)', + async cacheSynced => { + const { host, reconciler, coreApi, live, oauthResolver } = heldZeroReplicaWake({ + cacheSynced, + retainedFrontsOAuth: 'true', + resolveFrontsOAuth: async () => false, + }) + + await reconciler.reconcile(host) + + expect(oauthResolver).toHaveBeenCalledWith(host) + expect(vi.mocked(issueMcpHostRuntimeTokens)).toHaveBeenCalledOnce() + expect(vi.mocked(issueMcpHostRuntimeTokens).mock.calls[0]?.[2]).not.toContain( + OAUTH_USER_TOKEN_SCOPE + ) + expect( + runtimeTokenWrite(coreApi).metadata?.annotations?.[ + 'clerum.io/runtime-token-fronts-oauth-server' + ] + ).toBe('false') + expect(live().spec!.replicas).toBe(1) + } + ) + + it('skips a held wake mint when the OAuth read fails and the retained scope would grant it', async () => { + const { host, reconciler, coreApi, live, oauthResolver, runtimeRecordReads } = + heldZeroReplicaWake({ + cacheSynced: false, + retainedFrontsOAuth: 'true', + resolveFrontsOAuth: async () => { + throw new Error('McpServer read unavailable') + }, + }) + const warn = vi.spyOn(HostContextLogger.prototype, 'warn').mockImplementation(() => undefined) + try { + await reconciler.reconcile(host) + expect(warn).toHaveBeenCalledWith( + 'skipping runtime token mint during channel cache loss without an authoritative OAuth observation', + { + host: host.name, + namespace: host.namespace, + observation: 'frontsOAuthServer', + attempts: 3, + err: 'McpServer read unavailable', + } + ) + } finally { + warn.mockRestore() + } + + expect(runtimeRecordReads.length).toBeGreaterThan(0) + expect(oauthResolver).toHaveBeenCalledTimes(3) + expect(vi.mocked(issueMcpHostRuntimeTokens)).not.toHaveBeenCalled() + expect( + coreApi.replaceNamespacedSecret.mock.calls.filter(([request]) => + request.name?.includes('runtime-tokens') + ) + ).toHaveLength(0) + expect(live().spec!.replicas).toBe(0) + expect(reconciler.getStatus(host.name)).toMatchObject({ + deployed: true, + ready: false, + message: 'Waiting for authoritative scope observation', + }) + }) + + it('uses a retained non-granting OAuth observation when the OAuth read fails', async () => { + const { host, reconciler, coreApi, live, oauthResolver } = heldZeroReplicaWake({ + cacheSynced: false, + retainedFrontsOAuth: 'false', + resolveFrontsOAuth: async () => { + throw new Error('McpServer read unavailable') + }, + }) + + await reconciler.reconcile(host) + + expect(oauthResolver).toHaveBeenCalledTimes(3) + expect(vi.mocked(issueMcpHostRuntimeTokens)).toHaveBeenCalledOnce() + expect(vi.mocked(issueMcpHostRuntimeTokens).mock.calls[0]?.[2]).not.toContain( + OAUTH_USER_TOKEN_SCOPE + ) + expect( + runtimeTokenWrite(coreApi).metadata?.annotations?.[ + 'clerum.io/runtime-token-fronts-oauth-server' + ] + ).toBe('false') + expect(live().spec!.replicas).toBe(1) + }) + }) + + it('does not renew held-runtime GFS before its refresh window', async () => { + const host = makeStatelessHost() + const { reconciler, appsApi, coreApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + coreApi.readNamespacedSecret.mockResolvedValue(runtimeCredentialRecord(host)) + appsApi.readNamespacedDeployment.mockResolvedValue(trustedRuntimeDeployment(reconciler, host)) + vi.mocked(mintHostGfsToken).mockClear() + + const provision = await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host, { + refreshGfsOnly: true, + }) + + expect(provision).not.toBeNull() + expect(vi.mocked(mintHostGfsToken)).not.toHaveBeenCalled() + expect(coreApi.replaceNamespacedSecret).not.toHaveBeenCalled() + }) + + it('renews only held-runtime GFS at the refresh boundary', async () => { + const host = makeStatelessHost() + const { reconciler, appsApi, coreApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + const record = runtimeCredentialRecord(host, { + gfsRefreshBefore: '2000-01-01T00:00:00.000Z', + }) + coreApi.readNamespacedSecret.mockResolvedValue(record) + const deployment = trustedRuntimeDeployment(reconciler, host) + deployment.spec!.template!.metadata!.annotations = { + ...deployment.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': 'applied-runtime-revision', + } + appsApi.readNamespacedDeployment.mockResolvedValue(deployment) + vi.mocked(mintHostGfsToken).mockClear() + + const provision = await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host, { + refreshGfsOnly: true, + }) + + expect(vi.mocked(mintHostGfsToken)).toHaveBeenCalledOnce() + expect(provision.revision).toBe('applied-runtime-revision') + const write = coreApi.replaceNamespacedSecret.mock.calls.at(-1)?.[0].body as k8s.V1Secret + expect(write.data?.[MCP_HOST_RUNTIME_TOKEN_SECRET_ACCESS_KEY]).toBe( + record.data?.[MCP_HOST_RUNTIME_TOKEN_SECRET_ACCESS_KEY] + ) + expect(write.data?.[MCP_HOST_RUNTIME_TOKEN_SECRET_REFRESH_KEY]).toBe( + record.data?.[MCP_HOST_RUNTIME_TOKEN_SECRET_REFRESH_KEY] + ) + expect(write.data?.[MCP_HOST_RUNTIME_TOKEN_SECRET_CONTROL_KEY]).toBe( + record.data?.[MCP_HOST_RUNTIME_TOKEN_SECRET_CONTROL_KEY] + ) + expect(write.data?.[MCP_HOST_GFS_TOKEN_SECRET_KEY]).not.toBe( + record.data?.[MCP_HOST_GFS_TOKEN_SECRET_KEY] + ) + }) + + it('never falls back to full issuance from a closed held-runtime GFS renewal', async () => { + const host = makeStatelessHost() + const { reconciler, appsApi, coreApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + coreApi.readNamespacedSecret.mockRejectedValue({ code: 404 }) + appsApi.readNamespacedDeployment.mockResolvedValue(trustedRuntimeDeployment(reconciler, host)) + vi.mocked(issueMcpHostRuntimeTokens).mockClear() + vi.mocked(mintHostGfsToken).mockClear() + + const provision = await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host, { + refreshGfsOnly: true, + }) + + expect(provision).toBeNull() + expect(vi.mocked(issueMcpHostRuntimeTokens)).not.toHaveBeenCalled() + expect(vi.mocked(mintHostGfsToken)).not.toHaveBeenCalled() + }) + + it('does not treat a requested but unready held runtime as ready for GFS renewal', async () => { + const host = makeStatelessHost() + const { reconciler, appsApi, coreApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + coreApi.readNamespacedSecret.mockResolvedValue( + runtimeCredentialRecord(host, { gfsRefreshBefore: '2000-01-01T00:00:00.000Z' }) + ) + appsApi.readNamespacedDeployment.mockResolvedValue( + trustedRuntimeDeployment(reconciler, host, { readyReplicas: 0 }) + ) + vi.mocked(mintHostGfsToken).mockClear() + + const provision = await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host, { + refreshGfsOnly: true, + }) + + expect(provision).toBeNull() + expect(vi.mocked(mintHostGfsToken)).not.toHaveBeenCalled() + }) + + it('does not reassign a prior Host UID during held-runtime GFS renewal', async () => { + const host = makeStatelessHost() + const { reconciler, appsApi, coreApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + coreApi.readNamespacedSecret.mockResolvedValue( + runtimeCredentialRecord(host, { hostUid: 'prior-host-uid' }) + ) + appsApi.readNamespacedDeployment.mockResolvedValue(trustedRuntimeDeployment(reconciler, host)) + vi.mocked(mintHostGfsToken).mockClear() + + const provision = await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host, { + refreshGfsOnly: true, + }) + + expect(provision).toBeNull() + expect(vi.mocked(mintHostGfsToken)).not.toHaveBeenCalled() + expect(coreApi.replaceNamespacedSecret).not.toHaveBeenCalled() + }) + + describe('held-runtime GFS renewal identity guards', () => { + const dueForRenewal = { gfsRefreshBefore: '2000-01-01T00:00:00.000Z' } + + /** Runs one refreshGfsOnly provision against a due record and a Ready Deployment. */ + async function renewHeldGfs( + scenario: { + host?: HostCRD + record?: (host: HostCRD) => k8s.V1Secret + deployment?: (reconciler: HostReconciler, host: HostCRD) => k8s.V1Deployment | null + } = {} + ) { + const host = scenario.host ?? makeStatelessHost() + const { reconciler, appsApi, coreApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + coreApi.readNamespacedSecret.mockResolvedValue( + scenario.record ? scenario.record(host) : runtimeCredentialRecord(host, dueForRenewal) + ) + const deployment = scenario.deployment + ? scenario.deployment(reconciler, host) + : trustedRuntimeDeployment(reconciler, host) + if (deployment) { + appsApi.readNamespacedDeployment.mockResolvedValue(deployment) + } else { + appsApi.readNamespacedDeployment.mockRejectedValue({ code: 404 }) + } + vi.mocked(mintHostGfsToken).mockClear() + + const provision = await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host, { + refreshGfsOnly: true, + }) + + return { + provision, + mints: vi.mocked(mintHostGfsToken).mock.calls.length, + recordReads: coreApi.readNamespacedSecret.mock.calls.length, + deploymentReads: appsApi.readNamespacedDeployment.mock.calls.length, + recordWrites: coreApi.replaceNamespacedSecret.mock.calls.length, + } + } + + /** The unmodified scenario mints, so every refusal below is a real refusal. */ + async function expectControlRenews() { + const control = await renewHeldGfs() + expect(control.provision).not.toBeNull() + expect(control.mints).toBe(1) + expect(control.recordWrites).toBe(1) + } + + async function expectRefusedAfterReads(scenario: Parameters[0]) { + await expectControlRenews() + const refused = await renewHeldGfs(scenario) + // Liveness witness: the credential record and the Deployment were both read + // before the guard refused, so the guarded path was entered. + expect(refused.recordReads).toBeGreaterThan(0) + expect(refused.deploymentReads).toBeGreaterThan(0) + expect(refused.provision).toBeNull() + expect(refused.mints).toBe(0) + expect(refused.recordWrites).toBe(0) + } + + it('refuses to renew an incomplete credential record', async () => { + await expectRefusedAfterReads({ + record: host => { + const record = runtimeCredentialRecord(host, dueForRenewal) + delete record.data![MCP_HOST_GFS_TOKEN_SECRET_KEY] + return record + }, + }) + }) + + it.each([ + [ + 'managed by another controller', + (host: HostCRD) => + runtimeCredentialRecord(host, { ...dueForRenewal, managedByHost: false }), + ], + [ + 'labelled for another Host', + (host: HostCRD) => { + const record = runtimeCredentialRecord(host, dueForRenewal) + record.metadata!.labels!['clerum.io/host'] = 'other-host' + return record + }, + ], + ])('refuses to renew a credential record %s', async (_label, record) => { + await expectRefusedAfterReads({ record }) + }) + + it.each<[string, (deployment: k8s.V1Deployment) => void]>([ + [ + 'labelled for another Host', + deployment => { + deployment.metadata!.labels!['clerum.io/host'] = 'other-host' + }, + ], + [ + 'managed by another controller', + deployment => { + deployment.metadata!.labels!['clerum.io/managed-by'] = 'someone-else' + }, + ], + [ + 'named after another Deployment', + deployment => { + deployment.metadata!.name = 'other-host' + }, + ], + [ + 'created for a prior Host incarnation', + deployment => { + deployment.metadata!.annotations!['clerum.io/host-uid'] = 'prior-host-uid' + }, + ], + [ + 'being deleted', + deployment => { + deployment.metadata!.deletionTimestamp = new Date('2026-07-02T00:00:00.000Z') + }, + ], + [ + 'without a uid', + deployment => { + delete deployment.metadata!.uid + }, + ], + [ + 'without a resourceVersion', + deployment => { + delete deployment.metadata!.resourceVersion + }, + ], + [ + 'without a pod template spec', + deployment => { + delete deployment.spec!.template!.spec + }, + ], + ])('refuses to renew when the Deployment is %s', async (_label, corrupt) => { + await expectRefusedAfterReads({ + deployment: (reconciler, host) => { + const deployment = trustedRuntimeDeployment(reconciler, host) + corrupt(deployment) + return deployment + }, + }) + }) + + it('refuses to renew when the Deployment is absent', async () => { + await expectRefusedAfterReads({ deployment: () => null }) + }) + + it.each<[string, (deployment: k8s.V1Deployment) => void]>([ + [ + 'scaled to zero replicas', + deployment => { + deployment.spec!.replicas = 0 + }, + ], + [ + 'without a replica count', + deployment => { + delete deployment.spec!.replicas + }, + ], + ])('refuses to renew when the Ready-status Deployment is %s', async (_label, corrupt) => { + await expectRefusedAfterReads({ + deployment: (reconciler, host) => { + // readyReplicas stays 1 so only the replica-count guard can refuse. + const deployment = trustedRuntimeDeployment(reconciler, host, { readyReplicas: 1 }) + corrupt(deployment) + return deployment + }, + }) + }) + + describe('with an unconsumed wake bootstrap', () => { + const APPLIED_REVISION = 'applied-runtime-revision' + + /** A due record whose fresh bootstrap is bound to the given Deployment UID. */ + const freshBootstrapBoundTo = (boundUid: string) => (host: HostCRD) => { + const record = runtimeCredentialRecord(host, dueForRenewal) + Object.assign(record.metadata!.annotations!, { + 'clerum.io/runtime-token-bootstrap-state': 'fresh', + 'clerum.io/runtime-token-bootstrap-deployment-uid': boundUid, + 'clerum.io/runtime-token-bootstrap-applied-revision': APPLIED_REVISION, + }) + return record + } + const appliedDeployment = (reconciler: HostReconciler, host: HostCRD) => { + const deployment = trustedRuntimeDeployment(reconciler, host) + deployment.spec!.template!.metadata!.annotations = { + ...deployment.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': APPLIED_REVISION, + } + return deployment + } + + it('marks a bootstrap bound to the applied Deployment consumed without minting', async () => { + const host = makeStatelessHost() + + const bound = await renewHeldGfs({ + host, + record: freshBootstrapBoundTo(`deployment-${host.name}`), + deployment: appliedDeployment, + }) + + // Liveness witness: the record was read and the consumed marker was written. + expect(bound.recordReads).toBeGreaterThan(0) + expect(bound.recordWrites).toBe(1) + expect(bound.provision.revision).toBe(APPLIED_REVISION) + expect(bound.mints).toBe(0) + }) + + it('renews once the bootstrap is bound to a different Deployment UID', async () => { + const host = makeStatelessHost() + const bound = await renewHeldGfs({ + host, + record: freshBootstrapBoundTo(`deployment-${host.name}`), + deployment: appliedDeployment, + }) + expect(bound.mints).toBe(0) + expect(bound.recordWrites).toBe(1) + + const replaced = await renewHeldGfs({ + host, + record: freshBootstrapBoundTo('deployment-replaced-uid'), + deployment: appliedDeployment, + }) + + expect(replaced.mints).toBe(1) + expect(replaced.recordWrites).toBe(1) + }) + }) + }) + + it('consumes a fresh bootstrap on GFS-only refresh when the Ready Deployment runs its revision', async () => { + vi.useFakeTimers({ toFake: ['Date'] }) + try { + const t0 = Date.parse('2026-07-03T00:00:00.000Z') + vi.setSystemTime(t0) + let cacheSynced = true + const host = makeStatelessHost() + const { reconciler, appsApi, coreApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => cacheSynced, + }) + reconciler.setHostFrontsOAuthServer(async () => false) + // The real producer mints a fresh bootstrap and the pod boots Ready on it, + // with channel authority lost before the resync that would consume it. + let record = await mintedRuntimeCredentialRecord(host, { bootstrap: 'fresh' }) + const deployedRevision = + record.metadata!.annotations!['clerum.io/runtime-token-secret-revision'] + expect(record.metadata!.annotations!['clerum.io/runtime-token-bootstrap-state']).toBe('fresh') + const deployment = trustedRuntimeDeployment(reconciler, host) + deployment.spec!.template!.metadata!.annotations = { + ...deployment.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': deployedRevision, + } + appsApi.readNamespacedDeployment.mockResolvedValue(deployment) + coreApi.readNamespacedSecret.mockImplementation(async () => structuredClone(record)) + coreApi.replaceNamespacedSecret.mockImplementation(async request => { + const body = request.body as k8s.V1Secret + // Two writes on one resourceVersion would be a 409 from the apiserver. + expect(body.metadata?.resourceVersion).toBe(record.metadata!.resourceVersion) + record = structuredClone({ + ...body, + metadata: { + ...body.metadata, + resourceVersion: String(Number(body.metadata!.resourceVersion) + 1), + }, + }) + return record + }) + vi.mocked(mintHostGfsToken).mockClear() + // A renewed GFS credential differs from the one the pod booted with, so + // the record's revision moves away from the deployed one. + vi.mocked(mintHostGfsToken).mockImplementationOnce(async ({ name, namespace }) => ({ + // The literal key type keeps the mock assignable to GfsHostToken. + [('to' + 'ken') as 'token']: 'gfs-renewed-value', + expiresInSeconds: 600, + subject: `host:1st:${namespace}/${name}`, + })) + vi.mocked(issueMcpHostRuntimeTokens).mockClear() + + cacheSynced = false + vi.setSystemTime(t0 + 601_000) + await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host, { refreshGfsOnly: true }) + vi.setSystemTime(t0 + 601_500) + await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host, { refreshGfsOnly: true }) + + expect(record.metadata!.annotations!['clerum.io/runtime-token-bootstrap-state']).toBe( + 'consumed' + ) + + cacheSynced = true + vi.setSystemTime(t0 + 602_000) + const provision = await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host) + + // Liveness witness: the hold renewed GFS once and wrote the record twice + // (the consumed marker, then the renewed GFS credential). + expect(vi.mocked(mintHostGfsToken)).toHaveBeenCalledOnce() + expect(coreApi.replaceNamespacedSecret.mock.calls.length).toBeGreaterThanOrEqual(2) + expect(record.metadata!.annotations!['clerum.io/runtime-token-rollout-required']).not.toBe( + 'true' + ) + // The Ready pod already rotated the refresh token it booted with, so moving + // the Deployment to another revision without a runtime mint would hand the + // new pod a revoked token. + if (provision.revision !== deployedRevision) { + expect(vi.mocked(issueMcpHostRuntimeTokens).mock.calls.length).toBeGreaterThan(0) + } + expect(provision.revision).toBe(deployedRevision) + } finally { + vi.useRealTimers() + } + }) + + it('keeps a Ready held runtime deployed when its GFS renewal fails, without retry backoff', async () => { + // An administrative intent on the Host: the pass that keeps the runtime + // Ready must report exactly one outcome for it. + const host: HostCRD = { + ...makeStatelessHost({ + status: { lifecycle: { state: 'active', wakeHandledGeneration: 1 } }, + }), + generation: 4, + annotations: { + 'clerum.io/administrative-intent-id': '22222222-2222-4222-8222-222222222222', + }, + } + const infrastructureTelemetryReporter = createTelemetryReporterMock() + const administrativeOutcomeReporter = { + enqueueHostOutcome: vi.fn(), + stop: vi.fn(async () => undefined), + } + const { reconciler, appsApi, coreApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + infrastructureTelemetryReporter, + administrativeOutcomeReporter, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + reconciler.setHostFrontsOAuthServer(async () => false) + // A running pod's consumed record whose GFS credential is due for renewal. + const record = await mintedRuntimeCredentialRecord(host, { + annotations: { 'clerum.io/gfs-token-refresh-before': '2000-01-01T00:00:00.000Z' }, + }) + const runtimeSecretName = record.metadata!.name! + const readSecret = coreApi.readNamespacedSecret.getMockImplementation()! + coreApi.readNamespacedSecret.mockImplementation(request => + request.name === runtimeSecretName + ? Promise.resolve(structuredClone(record)) + : readSecret(request) + ) + const applied = trustedRuntimeDeployment(reconciler, host) + applied.spec!.template!.metadata!.annotations = { + ...applied.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': + record.metadata!.annotations!['clerum.io/runtime-token-secret-revision'], + } + const live = persistHostDeployment(appsApi, host, applied) + const gfsMint = vi.mocked(mintHostGfsToken) + const defaultGfsMint = gfsMint.getMockImplementation()! + gfsMint.mockClear() + gfsMint.mockImplementation(async () => { + throw new Error('gfs token endpoint unavailable') + }) + vi.mocked(issueMcpHostRuntimeTokens).mockClear() + try { + const started = performance.now() + await reconciler.reconcile(host) + const elapsedMs = performance.now() - started + + // Liveness witness: the hold attempted the GFS renewal exactly once. + expect(gfsMint).toHaveBeenCalledOnce() + expect(elapsedMs).toBeLessThan(1000) + expect(reconciler.getStatus(host.name)).toEqual({ + deployed: true, + ready: true, + message: 'Held runtime credential renewal failed; runtime kept', + }) + const controllerErrors = vi + .mocked(infrastructureTelemetryReporter.enqueue) + .mock.calls.filter(([event]) => event.telemetryType === 'controller_error') + expect(controllerErrors).toHaveLength(1) + expect(controllerErrors[0]![0].payload).toMatchObject({ + reason_code: 'RuntimeCredentialRenewalFailed', + status: 'failed', + }) + // The renewal failure is telemetry; the intent's one outcome is the Ready + // runtime, never a `failed` next to a `succeeded` for the same generation. + expect(administrativeOutcomeReporter.enqueueHostOutcome).toHaveBeenCalledOnce() + expect(administrativeOutcomeReporter.enqueueHostOutcome).toHaveBeenCalledWith( + expect.objectContaining({ + sourceEventId: + 'hcc-admin-outcome-v2:22222222-2222-4222-8222-222222222222:4:stateless-host-uid:succeeded', + outcome: 'succeeded', + }) + ) + expect(live().spec!.replicas).toBe(1) + expect(live().spec!.template).toEqual(applied.spec!.template) + expect(vi.mocked(issueMcpHostRuntimeTokens)).not.toHaveBeenCalled() + } finally { + gfsMint.mockImplementation(defaultGfsMint) + } + }) + + it('recovers an active held Deployment from zero after minting credentials on each pass', async () => { + const host = makeStatelessHost({ + status: { lifecycle: { state: 'active', wakeHandledGeneration: 1 } }, + }) + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = reconciler.buildDeployment(host) + applied.spec!.replicas = 0 + applied.spec!.template.metadata!.annotations = { 'example.org/applied': 'keep-exactly' } + applied.spec!.template!.metadata!.annotations = { + ...applied.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': 'applied-runtime-revision', + } + markPersistedRuntimeTrusted(applied, host, 0) + const live = persistHostDeployment(appsApi, host, applied) + const provision = vi + .spyOn(reconciler as any, 'provisionRuntimeTokenRevision') + .mockResolvedValue(runtimeTokenProvision(host)) + + for (let pass = 1; pass <= 3; pass++) { + await reconciler.reconcile(host) + + // One credential mint per pass, made before that pass's replica-only update. + expect(provision).toHaveBeenCalledTimes(pass) + expect(provision).toHaveBeenLastCalledWith(host, { + forceFreshForWake: true, + targetSuspended: false, + preserveDeploymentTemplateOnWake: true, + }) + const passReplace = appsApi.replaceNamespacedDeployment.mock.invocationCallOrder.at(-1)! + expect(provision.mock.invocationCallOrder.at(-1)!).toBeLessThan(passReplace) + expect(live().spec!.replicas).toBe(1) + expect(live().spec!.template).toEqual(applied.spec!.template) + + // Scale the held Deployment back to zero so the next pass recovers it again. + live().spec!.replicas = 0 + live().status = { readyReplicas: 0 } + } + }) + + it('does not roll a held wake bootstrap when the watch recovers before Ready', async () => { + let cacheSynced = false + const host = makeStatelessHost({ + status: { lifecycle: { state: 'active', wakeHandledGeneration: 1 } }, + }) + const { reconciler, appsApi, coreApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => cacheSynced, + }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + let runtimeRecord = runtimeCredentialRecord(host) + coreApi.readNamespacedSecret.mockImplementation(({ name }) => + name?.includes('runtime-tokens') + ? Promise.resolve(runtimeRecord) + : Promise.resolve({ metadata: { resourceVersion: '1' }, data: {} } as any) + ) + const applied = reconciler.buildDeployment(host) + applied.spec!.replicas = 0 + applied.spec!.template!.metadata!.annotations = { + ...applied.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': 'applied-runtime-revision', + } + markPersistedRuntimeTrusted(applied, host, 0) + const live = persistHostDeployment(appsApi, host, applied) + const issueRuntimeMaterial = vi.mocked(issueMcpHostRuntimeTokens) + issueRuntimeMaterial.mockClear() - expect(appsApi.readNamespacedDeployment).toHaveBeenCalledTimes(2) - expect(appsApi.replaceNamespacedDeployment).toHaveBeenCalledOnce() - }) + await reconciler.reconcile(host) - it('replaces an existing Deployment when the runtime token revision changes', async () => { - const { reconciler, appsApi } = createReconciler() - const host = makeHost({ name: 'chatllm' }) - appsApi.createNamespacedDeployment.mockRejectedValue({ code: 409 }) - appsApi.readNamespacedDeployment.mockResolvedValue( - existingDeployment(reconciler, host, 'revision-a') + expect(issueRuntimeMaterial).toHaveBeenCalledOnce() + expect(live().spec!.replicas).toBe(1) + expect(live().spec!.template!.metadata!.annotations?.['clerum.io/runtime-token-revision']).toBe( + 'applied-runtime-revision' ) + const wakeWrite = coreApi.replaceNamespacedSecret.mock.calls.find(([request]) => + request.name?.includes('runtime-tokens') + )?.[0].body as k8s.V1Secret + expect(wakeWrite.metadata?.annotations).toMatchObject({ + 'clerum.io/runtime-token-bootstrap-state': 'fresh', + 'clerum.io/runtime-token-rollout-required': 'false', + 'clerum.io/runtime-token-bootstrap-deployment-uid': 'deployment-uid', + 'clerum.io/runtime-token-bootstrap-applied-revision': 'applied-runtime-revision', + }) + runtimeRecord = withReadableRuntimeRefreshMaterial(wakeWrite) + issueRuntimeMaterial.mockClear() + appsApi.createNamespacedDeployment.mockClear() + appsApi.replaceNamespacedDeployment.mockClear() + cacheSynced = true + await reconciler.reconcile(host) - await (reconciler as any).ensureDeployment(host, [], 'revision-b') + expect(issueRuntimeMaterial).not.toHaveBeenCalled() + expect( + appsApi.replaceNamespacedDeployment.mock.calls.filter( + ([request]) => request.body?.metadata?.name === host.name + ) + ).toHaveLength(0) - expect(appsApi.replaceNamespacedDeployment).toHaveBeenCalledOnce() - const replacement = appsApi.replaceNamespacedDeployment.mock.calls[0][0] - .body as k8s.V1Deployment - expect(replacement.metadata?.resourceVersion).toBe('42') + live().status = { readyReplicas: 1 } + coreApi.replaceNamespacedSecret.mockClear() + await reconciler.reconcile(host) + + expect(issueRuntimeMaterial).not.toHaveBeenCalled() expect( - replacement.spec?.template.metadata?.annotations?.['clerum.io/runtime-token-revision'] - ).toBe('revision-b') + appsApi.replaceNamespacedDeployment.mock.calls.filter( + ([request]) => request.body?.metadata?.name === host.name + ) + ).toHaveLength(0) + const consumedWrite = coreApi.replaceNamespacedSecret.mock.calls.find(([request]) => + request.name?.includes('runtime-tokens') + )?.[0].body as k8s.V1Secret + expect(consumedWrite.metadata?.annotations?.['clerum.io/runtime-token-bootstrap-state']).toBe( + 'consumed' + ) + runtimeRecord = consumedWrite + + live().spec!.replicas = 0 + live().status = { readyReplicas: 0 } + issueRuntimeMaterial.mockClear() + await reconciler.reconcile(host) + + expect(issueRuntimeMaterial).toHaveBeenCalledOnce() + expect(live().spec!.replicas).toBe(1) }) - it('replaces an existing Deployment when the stateless lifecycle target changes', async () => { - const { reconciler, appsApi } = createReconciler() + it('marks a newly created preserved wake bootstrap for scale-only delivery', async () => { const host = makeStatelessHost() - const active: EffectiveHostLifecycle = { stateless: true, state: 'active' } - const suspended: EffectiveHostLifecycle = { stateless: true, state: 'suspended' } - appsApi.createNamespacedDeployment.mockRejectedValue({ code: 409 }) - appsApi.readNamespacedDeployment.mockResolvedValue( - existingDeployment(reconciler, host, 'revision-a', active) - ) + const { reconciler, appsApi, coreApi } = createReconciler() + const deployment = trustedRuntimeDeployment(reconciler, host, { + replicas: 0, + readyReplicas: 0, + }) + deployment.spec!.template!.metadata!.annotations = { + ...deployment.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': 'applied-runtime-revision', + } + appsApi.readNamespacedDeployment.mockResolvedValue(deployment) + coreApi.readNamespacedSecret.mockRejectedValue({ code: 404 }) + const issueRuntimeMaterial = vi.mocked(issueMcpHostRuntimeTokens) + issueRuntimeMaterial.mockClear() + + const first = await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host, { + forceFreshForWake: true, + targetSuspended: false, + preserveDeploymentTemplateOnWake: true, + }) - await (reconciler as any).ensureDeployment(host, [], 'revision-a', suspended) + expect(first.revision).toBe('applied-runtime-revision') + const created = coreApi.createNamespacedSecret.mock.calls.find(([request]) => + request.body?.metadata?.name?.includes('runtime-tokens') + )?.[0].body as k8s.V1Secret + expect(created.metadata?.annotations).toMatchObject({ + 'clerum.io/runtime-token-bootstrap-state': 'fresh', + 'clerum.io/runtime-token-rollout-required': 'false', + 'clerum.io/runtime-token-bootstrap-applied-revision': 'applied-runtime-revision', + }) - expect(appsApi.replaceNamespacedDeployment).toHaveBeenCalledOnce() - const replacement = appsApi.replaceNamespacedDeployment.mock.calls[0][0] - .body as k8s.V1Deployment - expect(replacement.metadata?.resourceVersion).toBe('42') - expect(replacement.spec?.replicas).toBe(0) - expect( - replacement.spec?.template.metadata?.annotations?.['clerum.io/runtime-token-revision'] - ).toBe('revision-a') + const readableCreated = withReadableRuntimeRefreshMaterial(created) + coreApi.readNamespacedSecret.mockResolvedValue(readableCreated) + deployment.spec!.replicas = 1 + issueRuntimeMaterial.mockClear() + + const second = await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host, { + targetSuspended: false, + preserveDeploymentTemplateOnWake: true, + }) + + expect(issueRuntimeMaterial).not.toHaveBeenCalled() + expect(second.revision).toBe('applied-runtime-revision') }) - it('replaces a running stateful Host with the stateless template when lifecycle is enabled', async () => { - const { reconciler, appsApi, customApi } = createReconciler() - const stateful = makeHost({ name: 'transition-host' }) + describe('wake bootstrap binding to the applied Deployment', () => { + const BOOTSTRAP_UID = 'clerum.io/runtime-token-bootstrap-deployment-uid' + const BOOTSTRAP_REVISION = 'clerum.io/runtime-token-bootstrap-applied-revision' + const APPLIED_REVISION = 'applied-runtime-revision' + + /** + * Mints a wake bootstrap for a zero-replica Deployment through the real + * producer, then models the wake: the same Deployment at one replica that is + * not Ready yet, with the persisted record bound to it. + */ + async function boundWakeBootstrap() { + const host = makeStatelessHost() + const { reconciler, appsApi, coreApi } = createReconciler() + const deployment = trustedRuntimeDeployment(reconciler, host, { + replicas: 0, + readyReplicas: 0, + }) + deployment.spec!.template!.metadata!.annotations = { + ...deployment.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': APPLIED_REVISION, + } + appsApi.readNamespacedDeployment.mockResolvedValue(deployment) + coreApi.readNamespacedSecret.mockRejectedValue({ code: 404 }) + const issueRuntimeMaterial = vi.mocked(issueMcpHostRuntimeTokens) + issueRuntimeMaterial.mockClear() + + await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host, { + forceFreshForWake: true, + targetSuspended: false, + preserveDeploymentTemplateOnWake: true, + }) - await reconciler.reconcile(stateful) - const runningStateful = structuredClone(hostDeploymentBody(appsApi, 'transition-host')) - runningStateful.metadata = { ...runningStateful.metadata, resourceVersion: '42' } - runningStateful.status = { readyReplicas: 1, availableReplicas: 1 } - expect(runningStateful.spec?.replicas).toBe(1) - expect(containerEnv(runningStateful).map(entry => entry.name)).not.toContain( - 'CLERUM_STATELESS_LIFECYCLE' - ) + const created = coreApi.createNamespacedSecret.mock.calls.find(([request]) => + request.body?.metadata?.name?.includes('runtime-tokens') + )?.[0].body as k8s.V1Secret + expect(created.metadata?.annotations).toMatchObject({ + 'clerum.io/runtime-token-bootstrap-state': 'fresh', + [BOOTSTRAP_UID]: deployment.metadata!.uid, + [BOOTSTRAP_REVISION]: APPLIED_REVISION, + }) + const record = withReadableRuntimeRefreshMaterial(created) + coreApi.readNamespacedSecret.mockResolvedValue(record) + deployment.spec!.replicas = 1 + + /** One follow-up provision on the woken Deployment, with the counters it moved. */ + const provisionOnWokenDeployment = async () => { + issueRuntimeMaterial.mockClear() + coreApi.replaceNamespacedSecret.mockClear() + const recordReadsBefore = coreApi.readNamespacedSecret.mock.calls.length + const deploymentReadsBefore = appsApi.readNamespacedDeployment.mock.calls.length + const provision = await (reconciler as any).ensureMcpHostRuntimeTokenSecret(host, { + targetSuspended: false, + preserveDeploymentTemplateOnWake: true, + }) + return { + provision, + mints: issueRuntimeMaterial.mock.calls.length, + recordReads: coreApi.readNamespacedSecret.mock.calls.length - recordReadsBefore, + deploymentReads: + appsApi.readNamespacedDeployment.mock.calls.length - deploymentReadsBefore, + writes: coreApi.replaceNamespacedSecret.mock.calls.map(([request]) => request.body), + } + } + return { deployment, record, provisionOnWokenDeployment } + } - appsApi.replaceNamespacedDeployment.mockClear() - const readDeployment = appsApi.readNamespacedDeployment.getMockImplementation()! - appsApi.readNamespacedDeployment.mockImplementation(request => - request.name === stateful.name ? Promise.resolve(runningStateful) : readDeployment(request) - ) + it('reuses a wake bootstrap bound to the applied Deployment UID and revision', async () => { + const { provisionOnWokenDeployment } = await boundWakeBootstrap() - const stateless = makeStatelessHost({ name: 'transition-host' }) - customApi.getNamespacedCustomObject.mockResolvedValue({ - metadata: { name: stateless.name, namespace: stateless.namespace, uid: stateless.uid }, - spec: stateless.spec, - status: { lifecycle: { state: 'active', wakeHandledGeneration: 0 } }, - }) + const reused = await provisionOnWokenDeployment() - await reconciler.reconcile(stateless) + // Liveness witness: the record and the Deployment were read and the reuse + // path returned the applied revision it binds. + expect(reused.recordReads).toBeGreaterThan(0) + expect(reused.deploymentReads).toBeGreaterThan(0) + expect(reused.provision.revision).toBe(APPLIED_REVISION) + expect(reused.mints).toBe(0) + expect(reused.writes).toHaveLength(0) + }) - const hostReplacements = appsApi.replaceNamespacedDeployment.mock.calls.filter( - ([request]) => request.name === stateful.name - ) - expect(hostReplacements).toHaveLength(1) - const replacement = hostReplacements[0][0].body as k8s.V1Deployment - expect(replacement.spec?.replicas).toBe(1) - expect(replacement.spec?.template.spec?.priorityClassName).toBe('clerum-interactive-host') - expect( - replacement.spec?.template.spec?.initContainers?.map(container => container.name) - ).toContain('workspace-layout-init') - expect(envValue(replacement, 'CLERUM_STATELESS_LIFECYCLE')).toBe('true') - expect(replacement.spec?.template.spec?.containers?.[0]?.volumeMounts).toContainEqual({ - name: 'workspace', - mountPath: '/workspace', - subPath: 'workspace', + it.each<[string, (deployment: k8s.V1Deployment) => void]>([ + [ + 'Deployment UID', + deployment => { + deployment.metadata!.uid = 'deployment-replaced-uid' + }, + ], + [ + 'applied runtime revision', + deployment => { + deployment.spec!.template!.metadata!.annotations = { + ...deployment.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': 'other-applied-revision', + } + }, + ], + ])('mints again once the bound %s changes', async (_label, changeDeployment) => { + const { deployment, provisionOnWokenDeployment } = await boundWakeBootstrap() + const reused = await provisionOnWokenDeployment() + expect(reused.recordReads).toBeGreaterThan(0) + expect(reused.mints).toBe(0) + + changeDeployment(deployment) + const reissued = await provisionOnWokenDeployment() + + expect(reissued.mints).toBe(1) + expect(reissued.writes).toHaveLength(1) + expect((reissued.writes[0] as k8s.V1Secret).metadata?.annotations).toMatchObject({ + 'clerum.io/runtime-token-bootstrap-state': 'fresh', + }) }) - expect(replacement.spec?.template.spec?.containers?.[0]?.volumeMounts).toContainEqual({ - name: 'workspace', - mountPath: '/var/lib/clerum/state', - subPath: 'state', + + it.each<[string, (deployment: k8s.V1Deployment, record: k8s.V1Secret) => void]>([ + [ + 'empty Deployment UID', + (deployment, record) => { + deployment.metadata!.uid = '' + record.metadata!.annotations![BOOTSTRAP_UID] = '' + }, + ], + [ + 'empty applied runtime revision', + (deployment, record) => { + deployment.spec!.template!.metadata!.annotations = { + ...deployment.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': '', + } + record.metadata!.annotations![BOOTSTRAP_REVISION] = '' + }, + ], + ])('does not treat an %s on both sides as a binding', async (_label, empty) => { + const { deployment, record, provisionOnWokenDeployment } = await boundWakeBootstrap() + const reused = await provisionOnWokenDeployment() + expect(reused.recordReads).toBeGreaterThan(0) + expect(reused.mints).toBe(0) + + empty(deployment, record) + const reissued = await provisionOnWokenDeployment() + + expect(reissued.mints).toBe(1) + expect(reissued.writes).toHaveLength(1) }) }) -}) -describe('HostReconciler stateless lifecycle — env injection', () => { - it('creates the stateless template directly for a new Host', async () => { - const { reconciler, appsApi, customApi } = createReconciler() - const stateless = makeStatelessHost({ name: 'new-stateless-host' }) - customApi.getNamespacedCustomObject.mockResolvedValue({ - metadata: { name: stateless.name, namespace: stateless.namespace, uid: stateless.uid }, - spec: stateless.spec, - status: { lifecycle: { state: 'active', wakeHandledGeneration: 0 } }, + it('keeps an active held Deployment at zero when credential minting is unavailable', async () => { + const host = makeStatelessHost({ + status: { lifecycle: { state: 'active', wakeHandledGeneration: 1 } }, + }) + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => false, }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) + const applied = reconciler.buildDeployment(host) + applied.spec!.replicas = 0 + applied.spec!.template!.metadata!.annotations = { + ...applied.spec!.template!.metadata!.annotations, + 'clerum.io/runtime-token-revision': 'applied-runtime-revision', + } + markPersistedRuntimeTrusted(applied, host, 0) + const live = persistHostDeployment(appsApi, host, applied) + appsApi.replaceNamespacedDeployment.mockClear() + const provision = vi + .spyOn(reconciler as any, 'provisionRuntimeTokenRevision') + .mockResolvedValue(null) - // Model a genuinely new principal Deployment, then expose its created state. - let principalCreated = false - const readDeployment = appsApi.readNamespacedDeployment.getMockImplementation()! - appsApi.readNamespacedDeployment.mockImplementation(request => { - if (request.name === stateless.name && !principalCreated) return Promise.reject({ code: 404 }) - return readDeployment(request) + await reconciler.reconcile(host) + + expect(provision).toHaveBeenCalledOnce() + expect(provision).toHaveBeenCalledWith(host, { + forceFreshForWake: true, + targetSuspended: false, + preserveDeploymentTemplateOnWake: true, }) - appsApi.createNamespacedDeployment.mockImplementation(async request => { - if (request.body.metadata?.name === stateless.name) principalCreated = true - return request.body + expect(appsApi.replaceNamespacedDeployment).not.toHaveBeenCalled() + expect(live().spec!.replicas).toBe(0) + expect(reconciler.getStatus(host.name)).toMatchObject({ + deployed: true, + ready: false, + message: 'Waiting for authoritative scope observation', }) - await reconciler.reconcile(stateless) + }) - const creates = appsApi.createNamespacedDeployment.mock.calls.filter( - ([arg]) => (arg as { body?: k8s.V1Deployment }).body?.metadata?.name === 'new-stateless-host' - ) - const replacements = appsApi.replaceNamespacedDeployment.mock.calls.filter( - ([arg]) => (arg as { body?: k8s.V1Deployment }).body?.metadata?.name === 'new-stateless-host' - ) - expect(creates).toHaveLength(1) - expect(replacements).toHaveLength(0) - const created = hostDeploymentBody(appsApi, 'new-stateless-host') - expect(created.spec?.replicas).toBe(1) - expect(created.spec?.template.spec?.priorityClassName).toBe('clerum-interactive-host') - expect(created.spec?.template.spec?.initContainers?.map(container => container.name)).toContain( - 'workspace-layout-init' - ) - expect(envValue(created, 'CLERUM_STATELESS_LIFECYCLE')).toBe('true') - expect(created.spec?.template.spec?.containers?.[0]?.volumeMounts).toContainEqual({ - name: 'workspace', - mountPath: '/workspace', - subPath: 'workspace', + it('reconciles an active Host twice through cache loss without changing its session path or template', async () => { + let cacheSynced = true + const host = makeStatelessHost({ + status: { lifecycle: { state: 'active', wakeHandledGeneration: 2 } }, }) - expect(created.spec?.template.spec?.containers?.[0]?.volumeMounts).toContainEqual({ - name: 'workspace', - mountPath: '/var/lib/clerum/state', - subPath: 'state', + let serverHost = host + const { reconciler, appsApi, customApi } = createReconciler({ + isCommunicationChannelCacheSynced: () => cacheSynced, }) - }) + customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(serverHost)) - it('injects the three stateless env vars when stateless is enabled', () => { - const { reconciler } = createReconciler() - const dep = reconciler.buildDeployment(makeStatelessHost()) - expect(envValue(dep, 'CLERUM_STATELESS_LIFECYCLE')).toBe('true') - expect(envValue(dep, 'CLERUM_SESSION_STORE')).toBe('sqlite') - expect(envValue(dep, 'CLERUM_SESSION_DB_DIR')).toBe('/var/lib/clerum/state') - }) + await reconciler.reconcile(host) + const live = persistHostDeployment(appsApi, host, hostDeploymentBody(appsApi, host.name)) + const baseline = structuredClone(live().spec?.template) + expect(envValue(live(), 'CLERUM_SESSION_DB_DIR')).toBe('/var/lib/clerum/state') - it('does not inject the stateless env vars when stateless is off', () => { - const { reconciler } = createReconciler() - const names = containerEnv(reconciler.buildDeployment(makeHost())).map(e => e.name) - expect(names).not.toContain('CLERUM_STATELESS_LIFECYCLE') - expect(names).not.toContain('CLERUM_SESSION_STORE') - expect(names).not.toContain('CLERUM_SESSION_DB_DIR') + for (let cycle = 1; cycle <= 2; cycle++) { + cacheSynced = false + // Exercise the fresh-read race as well as the cached active state: a + // suspend may have landed after the initial Host watch event. + serverHost = makeStatelessHost({ status: suspendedStatus(2 + cycle) }) + appsApi.createNamespacedDeployment.mockClear() + appsApi.replaceNamespacedDeployment.mockClear() + await reconciler.reconcile(host) + + const deployment = live() + expect(deployment.spec?.replicas).toBe(1) + expect(deployment.spec?.template).toEqual(baseline) + expect(envValue(deployment, 'CLERUM_SESSION_DB_DIR')).toBe('/var/lib/clerum/state') + expect(reconciler.getEffectiveLifecycle(host)).toMatchObject({ + stateless: true, + suspensionBlocked: true, + }) + expect(rejectedCondition(lifecycleStatusWrites(customApi).at(-1)!).reason).toBe( + 'CommunicationChannelCacheUnsynced' + ) + expect(lifecycleStatusWrites(customApi).at(-1)?.lifecycle?.state).toBe('suspended') + expect( + appsApi.replaceNamespacedDeployment.mock.calls.filter(([r]) => r.name === host.name) + ).toHaveLength(0) + cacheSynced = true + serverHost = host + await reconciler.reconcile(host) + expect(live().spec?.template).toEqual(baseline) + } }) -}) -describe('HostReconciler stateless lifecycle — rejection matrix', () => { - it('fails closed while the CommunicationChannel cache is unsynced', async () => { + it('preserves a suspended Host while the CommunicationChannel cache is unsynced', async () => { const { reconciler, appsApi, customApi } = createReconciler({ isCommunicationChannelCacheSynced: () => false, }) - await reconciler.reconcile(makeStatelessHost({ status: suspendedStatus() })) + const host = makeStatelessHost({ status: suspendedStatus() }) + const live = persistHostDeployment( + appsApi, + host, + createReconciler().reconciler.buildDeployment(host) + ) + await reconciler.reconcile(host) - expect(hostDeploymentBody(appsApi, 'stateless-host').spec?.replicas).toBe(1) + const deployment = live() + expect(deployment.spec?.replicas).toBe(0) const writes = lifecycleStatusWrites(customApi) expect(writes).toHaveLength(1) - expect(writes[0].lifecycle?.state).toBe('active') + expect(writes[0].lifecycle?.state).toBe('suspended') const condition = rejectedCondition(writes[0]) - expect(condition.status).toBe('True') + expect(condition.status).toBe('False') expect(condition.reason).toContain('CommunicationChannelCacheUnsynced') }) - it('fails closed when the channel cache becomes unsynced during reconciliation', async () => { + it('preserves the stateless template when the channel cache becomes unsynced during reconciliation', async () => { let cacheSynced = true const { reconciler, appsApi, customApi, networkingApi } = createReconciler({ isCommunicationChannelCacheSynced: () => cacheSynced, }) const host = makeStatelessHost({ status: suspendedStatus(4) }) + const applied = reconciler.buildDeployment(host) + markPersistedRuntimeTrusted(applied, host) + const live = persistHostDeployment(appsApi, host, applied) customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) const readPolicy = networkingApi.readNamespacedNetworkPolicy.getMockImplementation()! networkingApi.readNamespacedNetworkPolicy.mockImplementation(async request => { @@ -771,25 +3135,21 @@ describe('HostReconciler stateless lifecycle — rejection matrix', () => { await reconciler.reconcile(host) expect(networkingApi.readNamespacedNetworkPolicy).toHaveBeenCalled() - expect(provision).toHaveBeenCalledOnce() - expect(provision).toHaveBeenCalledWith( - host, - expect.objectContaining({ targetSuspended: false }) - ) - const deployment = hostDeploymentBody(appsApi, host.name) - expect(deployment.spec?.replicas).toBe(1) - expect(containerEnv(deployment).map(entry => entry.name)).not.toContain( - 'CLERUM_STATELESS_LIFECYCLE' - ) + expect(provision).not.toHaveBeenCalled() + const deployment = live() + expect(deployment.spec?.replicas).toBe(0) + expect(envValue(deployment, 'CLERUM_STATELESS_LIFECYCLE')).toBe('true') + expect(envValue(deployment, 'CLERUM_SESSION_STORE')).toBe('sqlite') + expect(envValue(deployment, 'CLERUM_SESSION_DB_DIR')).toBe('/var/lib/clerum/state') const writes = lifecycleStatusWrites(customApi) expect(writes).toHaveLength(2) expect(writes[0].lifecycle?.state).toBe('suspended') - expect(writes[1].lifecycle).toEqual({ - state: 'active', + expect(writes[1].lifecycle).toMatchObject({ + state: 'suspended', wakeHandledGeneration: 4, - reason: 'CommunicationChannel cache is not synchronized; stateless lifecycle is held active', }) expect(rejectedCondition(writes[1]).reason).toBe('CommunicationChannelCacheUnsynced') + expect(rejectedCondition(writes[1]).status).toBe('False') }) it('fails closed when a channel starts referencing the Host during reconciliation', async () => { @@ -1052,7 +3412,7 @@ describe('HostReconciler stateless lifecycle — rejection matrix', () => { expect(provision).toHaveBeenCalledOnce() }) - it('does not repeat bootstrap provisioning for active cache loss without channels', async () => { + it('refreshes only the GFS token during active cache loss without changing replicas', async () => { let cacheSynced = true const { reconciler, appsApi, customApi, networkingApi } = createReconciler({ isCommunicationChannelCacheSynced: () => cacheSynced, @@ -1060,6 +3420,9 @@ describe('HostReconciler stateless lifecycle — rejection matrix', () => { const host = makeStatelessHost({ status: { lifecycle: { state: 'active', wakeHandledGeneration: 1 } }, }) + const applied = reconciler.buildDeployment(host) + markPersistedRuntimeTrusted(applied, host) + const live = persistHostDeployment(appsApi, host, applied) customApi.getNamespacedCustomObject.mockImplementation(async () => hostApiObject(host)) const readPolicy = networkingApi.readNamespacedNetworkPolicy.getMockImplementation()! networkingApi.readNamespacedNetworkPolicy.mockImplementation(async request => { @@ -1074,11 +3437,15 @@ describe('HostReconciler stateless lifecycle — rejection matrix', () => { expect(networkingApi.readNamespacedNetworkPolicy).toHaveBeenCalled() expect(provision).toHaveBeenCalledOnce() - const deployment = hostDeploymentBody(appsApi, host.name) + expect(provision).toHaveBeenCalledWith(host, { + refreshGfsOnly: true, + targetSuspended: false, + }) + const deployment = live() expect(deployment.spec?.replicas).toBe(1) - expect(containerEnv(deployment).map(entry => entry.name)).not.toContain( - 'CLERUM_STATELESS_LIFECYCLE' - ) + expect(envValue(deployment, 'CLERUM_STATELESS_LIFECYCLE')).toBe('true') + expect(envValue(deployment, 'CLERUM_SESSION_STORE')).toBe('sqlite') + expect(envValue(deployment, 'CLERUM_SESSION_DB_DIR')).toBe('/var/lib/clerum/state') }) it('rejects stateless by default when CommunicationChannels reference the host', async () => { @@ -1742,15 +4109,12 @@ describe('HostReconciler stateless lifecycle — AP-1 fresh-read status writer', }) }) - it('reject-while-active keeps state from FRESH (8th costume preserved): a suspend that landed is not resurrected', async () => { + it('confirmed incompatibility restores active state after a concurrent suspend', async () => { const { reconciler, customApi } = createReconciler({ countCommunicationChannels: () => 1, }) - // Cached active, but a heartbeat suspend landed between the snapshot and - // the writer's fresh read (fresh = suspended, gen 5). The rejection reason - // is an intended override, but STATE must still come from fresh — the - // reject branch must NOT reintroduce the 8th costume by echoing the cached - // active state over a just-suspended Host. + // A confirmed channel conflict has the existing always-on policy. If a + // heartbeat suspended the Host during the fresh read, reconcile wakes it. const host = makeStatelessHost({ status: { lifecycle: { state: 'active', wakeHandledGeneration: 3 } }, }) @@ -1762,7 +4126,7 @@ describe('HostReconciler stateless lifecycle — AP-1 fresh-read status writer', const writes = lifecycleStatusWrites(customApi) expect(writes.at(-1)?.lifecycle).toEqual({ - state: 'suspended', + state: 'active', wakeHandledGeneration: 5, reason: '1 CommunicationChannel(s) reference this Host; disassociate them to enable the requested stateless lifecycle', @@ -1875,6 +4239,71 @@ describe('HostReconciler stateless lifecycle — suspension durability', () => { }) }) +describe('HostReconciler readiness poll generation fence', () => { + afterEach(() => { + vi.useRealTimers() + }) + + async function pollOnce(options: { supersede: boolean }) { + vi.useFakeTimers() + const { reconciler } = createReconciler() + const internals = reconciler as unknown as { + advanceReconcileGeneration(name: string): number + setStatus(name: string, status: { deployed: boolean; ready: boolean; message: string }): void + pollReadiness( + name: string, + namespace: string, + intervalMs?: number, + maxAttempts?: number + ): void + } + const readySpy = vi.spyOn(reconciler as any, 'checkDeploymentReady').mockResolvedValue(true) + internals.advanceReconcileGeneration('stateless-host') + internals.setStatus('stateless-host', { + deployed: true, + ready: false, + message: 'Deployed, waiting for readiness', + }) + internals.pollReadiness('stateless-host', 'tenant-ns', 10, 2) + if (options.supersede) { + // A later reconcile pass owns the status now; the older poll must not + // overwrite its verdict with a stale 'Running'. + internals.advanceReconcileGeneration('stateless-host') + internals.setStatus('stateless-host', { + deployed: true, + ready: false, + message: 'Degraded: newer reconcile verdict', + }) + } + await vi.advanceTimersByTimeAsync(10) + return { reconciler, readySpy } + } + + it('reports Running when no newer reconcile superseded the poll', async () => { + const { reconciler, readySpy } = await pollOnce({ supersede: false }) + + expect(readySpy).toHaveBeenCalledTimes(1) + expect(reconciler.getStatus('stateless-host')).toEqual({ + deployed: true, + ready: true, + message: 'Running', + }) + }) + + it('keeps the newer reconcile verdict when the generation advanced during the poll', async () => { + const { reconciler, readySpy } = await pollOnce({ supersede: true }) + + // Liveness witness: the poll ran and observed a ready Deployment. + expect(readySpy).toHaveBeenCalledTimes(1) + expect(readySpy).toHaveBeenCalledWith('stateless-host', 'tenant-ns') + expect(reconciler.getStatus('stateless-host')).toEqual({ + deployed: true, + ready: false, + message: 'Degraded: newer reconcile verdict', + }) + }) +}) + describe('HostReconciler stateless lifecycle — probe tuning (Stage 6, W7)', () => { it('gives the stateless pod an aggressive startup probe preserving the 310s allowance', () => { const { reconciler } = createReconciler() diff --git a/host-context-controller/test/hostReconciler.networkpolicy.test.ts b/host-context-controller/test/hostReconciler.networkpolicy.test.ts index 3f7964cf19..87f342f287 100644 --- a/host-context-controller/test/hostReconciler.networkpolicy.test.ts +++ b/host-context-controller/test/hostReconciler.networkpolicy.test.ts @@ -215,10 +215,10 @@ describe('HostReconciler.reconcile — NP wiring', () => { revision: 'runtime-revision', scopeHash: (HostReconciler as any).runtimeTokenScopeHash(HOST, false), }) - vi.spyOn(reconciler as any, 'ensurePvc').mockResolvedValue(undefined) - vi.spyOn(reconciler as any, 'ensureService').mockResolvedValue(undefined) - vi.spyOn(reconciler as any, 'ensureDeployment').mockResolvedValue(undefined) - vi.spyOn(reconciler as any, 'ensureDesktopNetworkPolicy').mockResolvedValue(undefined) + vi.spyOn(reconciler as any, 'ensurePvc').mockResolvedValue(true) + vi.spyOn(reconciler as any, 'ensureService').mockResolvedValue(true) + vi.spyOn(reconciler as any, 'ensureDeployment').mockResolvedValue(true) + vi.spyOn(reconciler as any, 'ensureDesktopNetworkPolicy').mockResolvedValue('up_to_date') vi.spyOn(reconciler as any, 'reconcileChannelReaderDeployment').mockResolvedValue(undefined) vi.spyOn(reconciler as any, 'checkDeploymentReady').mockResolvedValue(true) }) @@ -229,8 +229,8 @@ describe('HostReconciler.reconcile — NP wiring', () => { // than racing the deny-all default. const ingressSpy = vi .spyOn(reconciler as any, 'ensureMcpHostIngressNetworkPolicy') - .mockResolvedValue(undefined) - const deploySpy = vi.spyOn(reconciler as any, 'ensureDeployment').mockResolvedValue(undefined) + .mockResolvedValue('up_to_date') + const deploySpy = vi.spyOn(reconciler as any, 'ensureDeployment').mockResolvedValue(true) const readySpy = vi.spyOn(reconciler as any, 'checkDeploymentReady').mockResolvedValue(true) await reconciler.reconcile(HOST) @@ -246,7 +246,7 @@ describe('HostReconciler.reconcile — NP wiring', () => { it('calls ensureChannelReaderEgressNetworkPolicy before reconcileChannelReaderDeployment', async () => { const egressSpy = vi .spyOn(reconciler as any, 'ensureChannelReaderEgressNetworkPolicy') - .mockResolvedValue(undefined) + .mockResolvedValue('up_to_date') const cReaderSpy = vi .spyOn(reconciler as any, 'reconcileChannelReaderDeployment') .mockResolvedValue(undefined) @@ -266,7 +266,7 @@ describe('HostReconciler.reconcile — NP wiring', () => { new Error('rbac: networkpolicies.networking.k8s.io is forbidden') ) vi.spyOn(reconciler as any, 'ensureChannelReaderEgressNetworkPolicy').mockResolvedValue( - undefined + 'up_to_date' ) await reconciler.reconcile(HOST) @@ -278,11 +278,13 @@ describe('HostReconciler.reconcile — NP wiring', () => { expect(status!.deployed).toBe(true) expect(status!.ready).toBe(false) expect(status!.message).toMatch(/degraded/i) - expect(status!.message).toMatch(/mcp-host NP/) + expect(status!.message).toMatch( + /mcp-host ingress: rbac: networkpolicies\.networking\.k8s\.io is forbidden/ + ) }) it('surfaces egress NP apply failure in Host status as degraded', async () => { - vi.spyOn(reconciler as any, 'ensureMcpHostIngressNetworkPolicy').mockResolvedValue(undefined) + vi.spyOn(reconciler as any, 'ensureMcpHostIngressNetworkPolicy').mockResolvedValue('up_to_date') vi.spyOn(reconciler as any, 'ensureChannelReaderEgressNetworkPolicy').mockRejectedValue( new Error('cluster connectivity failure') ) @@ -296,7 +298,25 @@ describe('HostReconciler.reconcile — NP wiring', () => { expect(status!.deployed).toBe(true) expect(status!.ready).toBe(false) expect(status!.message).toMatch(/degraded/i) - expect(status!.message).toMatch(/egress NP/) + expect(status!.message).toMatch(/channel-reader egress: cluster connectivity failure/) + }) + + it('surfaces a NetworkPolicy apply result of missing as degraded', async () => { + vi.spyOn(reconciler as any, 'ensureMcpHostIngressNetworkPolicy').mockResolvedValue('missing') + vi.spyOn(reconciler as any, 'ensureChannelReaderEgressNetworkPolicy').mockResolvedValue( + 'up_to_date' + ) + + await reconciler.reconcile(HOST) + + const status = (reconciler as any).statusMap.get('chatllm') as + | { deployed?: boolean; ready?: boolean; message?: string } + | undefined + expect(status).toMatchObject({ + deployed: true, + ready: false, + message: expect.stringContaining('mcp-host ingress: apply did not converge (missing)'), + }) }) }) diff --git a/host-context-controller/test/hostReconciler.test.ts b/host-context-controller/test/hostReconciler.test.ts index af0e94d7f5..d7eb92e7d6 100644 --- a/host-context-controller/test/hostReconciler.test.ts +++ b/host-context-controller/test/hostReconciler.test.ts @@ -171,6 +171,7 @@ function createReconciler( coreApi: asCoreApi(coreApi), networkingApi: asNetworkingApi(networkingApi), rbacApi: asRbacApi(rbacApi), + isCommunicationChannelCacheSynced: () => true, infrastructureTelemetryReporter: options.infrastructureTelemetryReporter, administrativeOutcomeReporter: options.administrativeOutcomeReporter, newTelemetryOccurrenceId: options.newTelemetryOccurrenceId, @@ -2630,15 +2631,24 @@ describe('HostReconciler oauth:user-token runtime scope provisioning', () => { it('fails closed (no oauth:user-token) when the oauth-server probe throws', async () => { vi.mocked(issueMcpHostRuntimeTokens).mockClear() - const { reconciler } = createReconciler() - reconciler.setHostFrontsOAuthServer(async () => { - throw new Error('context read failed') - }) + const warnSpy = vi.spyOn(HostContextLogger.prototype, 'warn') + try { + const { reconciler } = createReconciler() + reconciler.setHostFrontsOAuthServer(async () => { + throw new Error('context read failed') + }) - await reconciler.reconcile(makeHost()) + await reconciler.reconcile(makeHost()) - expect(lastIssuedScopes()).toEqual([...DEFAULT_FIRST_PARTY_WORKFLOW_CONTROL_SCOPES].sort()) - expect(lastIssuedScopes()).not.toContain('oauth:user-token') + expect(lastIssuedScopes()).toEqual([...DEFAULT_FIRST_PARTY_WORKFLOW_CONTROL_SCOPES].sort()) + expect(lastIssuedScopes()).not.toContain('oauth:user-token') + // A new Host has no retained grant to preserve, so the unknown answer + // fails closed at issuance, and says so. + const retryWarn = warnSpy.mock.calls.find(([msg]) => String(msg).includes('after retries')) + expect(retryWarn?.[1]).toMatchObject({ attempts: 3, err: 'context read failed' }) + } finally { + warnSpy.mockRestore() + } }) it('toggling oauth-server presence changes the runtime-token scope hash (drift re-issues)', () => { @@ -2670,27 +2680,38 @@ describe('HostReconciler oauth:user-token runtime scope provisioning', () => { }) }) -describe('HostReconciler.frontsOAuthServer transient-blip retry (R4-M9)', () => { +describe('HostReconciler.observeFrontsOAuthServer transient-blip retry (R4-M9)', () => { // The oauth-server probe is exercised through the REAL reconciler built by // createReconciler() and the REAL setHostFrontsOAuthServer setter — no - // hand-minted value. We assert on the private method directly (rather than the - // full reconcile) on purpose: the deployment drift guard - // (ensureCurrentRuntimeTokenScope) re-mints the token when the scope hash - // changes mid-reconcile, so a probe that throws once and then returns `true` - // would be MASKED at the public issuance observable — the guard would re-mint - // with the recovered `true` value, hiding the collapse. The boolean returned by - // frontsOAuthServer IS the observable whose flip drives the churn this fixes. - const callFronts = (reconciler: HostReconciler, host: HostCRD): Promise => + // hand-minted value. We assert on the private observation directly (rather + // than the full reconcile) on purpose: the deployment drift guard + // (ensureCurrentRuntimeTokenScope) re-reads the observation mid-reconcile, so + // a probe that throws once and then returns `true` would be MASKED at the + // public issuance observable. The observation IS the value whose flip drives + // the churn this fixes; what an unobserved answer means is decided by its + // callers and covered at the reconcile level. + const observeFronts = ( + reconciler: HostReconciler, + host: HostCRD + ): Promise< + | { observed: true; frontsOAuthServer: boolean } + | { observed: false; attempts: number; error: unknown } + > => ( reconciler as unknown as { - frontsOAuthServer(host: HostCRD): Promise + observeFrontsOAuthServer( + host: HostCRD + ): Promise< + | { observed: true; frontsOAuthServer: boolean } + | { observed: false; attempts: number; error: unknown } + > } - ).frontsOAuthServer(host) + ).observeFrontsOAuthServer(host) it('absorbs a transient throw: probe throws once then returns true → true (blip smoothed)', async () => { - // Regression case (T3): against the pre-fix head the throw collapses to - // `false` and this assertion fails. With the bounded retry the second - // attempt returns the authoritative `true`. + // Regression case (T3): without the bounded retry the throw is reported as + // unobserved and this assertion fails. With the retry the second attempt + // returns the authoritative `true`. const { reconciler } = createReconciler() let calls = 0 reconciler.setHostFrontsOAuthServer(async () => { @@ -2699,7 +2720,10 @@ describe('HostReconciler.frontsOAuthServer transient-blip retry (R4-M9)', () => return true }) - await expect(callFronts(reconciler, makeHost())).resolves.toBe(true) + await expect(observeFronts(reconciler, makeHost())).resolves.toEqual({ + observed: true, + frontsOAuthServer: true, + }) expect(calls).toBe(2) }) @@ -2713,29 +2737,28 @@ describe('HostReconciler.frontsOAuthServer transient-blip retry (R4-M9)', () => return false }) - await expect(callFronts(reconciler, makeHost())).resolves.toBe(false) + await expect(observeFronts(reconciler, makeHost())).resolves.toEqual({ + observed: true, + frontsOAuthServer: false, + }) expect(calls).toBe(1) }) - it('fails closed after exhausting retries: probe always throws → false, N attempts, warn logged', async () => { - const warnSpy = vi.spyOn(HostContextLogger.prototype, 'warn') - try { - const { reconciler } = createReconciler() - let calls = 0 - reconciler.setHostFrontsOAuthServer(async () => { - calls += 1 - throw new Error('sustained apiserver outage') - }) + it('reports an unobserved answer after exhausting retries instead of collapsing it to false', async () => { + const { reconciler } = createReconciler() + let calls = 0 + const outage = new Error('sustained apiserver outage') + reconciler.setHostFrontsOAuthServer(async () => { + calls += 1 + throw outage + }) - await expect(callFronts(reconciler, makeHost())).resolves.toBe(false) - // Retried up to the bounded ceiling (fix uses 3 total attempts); pre-fix - // head would call the probe exactly once. - expect(calls).toBe(3) - const retryWarn = warnSpy.mock.calls.find(([msg]) => String(msg).includes('after retries')) - expect(retryWarn).toBeDefined() - expect(retryWarn?.[1]).toMatchObject({ attempts: 3 }) - } finally { - warnSpy.mockRestore() - } + await expect(observeFronts(reconciler, makeHost())).resolves.toEqual({ + observed: false, + attempts: 3, + error: outage, + }) + // Retried up to the bounded ceiling (3 total attempts). + expect(calls).toBe(3) }) }) diff --git a/host-context-controller/test/hostReconciler.wake.test.ts b/host-context-controller/test/hostReconciler.wake.test.ts index 4c8fcb062a..93b8175b83 100644 --- a/host-context-controller/test/hostReconciler.wake.test.ts +++ b/host-context-controller/test/hostReconciler.wake.test.ts @@ -358,7 +358,7 @@ describe('HostReconciler wake fast-path — wake × state transition matrix', () ) expect( logSpy.mock.calls - .map(args => String(args[0])) + .map(args => JSON.parse(String(args[0])).msg as string) .some(line => line.includes('phase=replicas_patched')) ).toBe(true) } finally { @@ -581,7 +581,7 @@ describe('HostReconciler wake fast-path — durability and restart behavior', () String(args[0]).includes('Malformed clerum.io/wake-requested') ) expect(malformedLogs).toHaveLength(1) - expect(String(malformedLogs[0][0])).toContain('"not-a-number"') + expect(JSON.parse(String(malformedLogs[0][0])).msg).toContain('"not-a-number"') } finally { errorSpy.mockRestore() } @@ -599,7 +599,7 @@ describe('HostReconciler wake fast-path — structured wake timestamps (Stage 6, await reconciler.reconcile(makeWakeHost({ wakeRequested: '2', state: 'suspended' })) const wakeLines = logSpy.mock.calls - .map(args => String(args[0])) + .map(args => JSON.parse(String(args[0])).msg as string) .filter(line => line.startsWith('[StatelessWake]')) expect(wakeLines).toEqual([ expect.stringMatching( @@ -628,7 +628,7 @@ describe('HostReconciler wake fast-path — structured wake timestamps (Stage 6, makeWakeHost({ wakeRequested: '4', state: 'draining', wakeHandledGeneration: 1 }) ) - const lines = logSpy.mock.calls.map(args => String(args[0])) + const lines = logSpy.mock.calls.map(args => JSON.parse(String(args[0])).msg as string) const phases = lines .filter(line => line.startsWith('[StatelessWake]')) .map(line => /phase=(\w+)/.exec(line)?.[1]) @@ -683,7 +683,7 @@ describe('HostReconciler — scale-transition counter (Stage 6 metric)', () => { await reconciler.suspendHostFromHeartbeat(host, 'idle', 2) const metricLines = logSpy.mock.calls - .map(args => String(args[0])) + .map(args => JSON.parse(String(args[0])).msg as string) .filter(line => line.startsWith('[StatelessMetric]')) expect(metricLines).toEqual([ '[StatelessMetric] scale_transition host=stateless-host direction=up total=1', @@ -691,7 +691,7 @@ describe('HostReconciler — scale-transition counter (Stage 6 metric)', () => { ]) const suspendLines = logSpy.mock.calls - .map(args => String(args[0])) + .map(args => JSON.parse(String(args[0])).msg as string) .filter(line => line.startsWith('[StatelessSuspend]')) expect(suspendLines).toEqual([ expect.stringMatching( @@ -720,7 +720,7 @@ describe('HostReconciler wake fast-path — AP-1 fresh re-decide at the commit p expect(customApi.patchNamespacedCustomObjectStatus).not.toHaveBeenCalled() expect(appsApi.patchNamespacedDeployment).not.toHaveBeenCalled() - const lines = logSpy.mock.calls.map(args => String(args[0])) + const lines = logSpy.mock.calls.map(args => JSON.parse(String(args[0])).msg as string) expect(lines.filter(line => line.startsWith('[StatelessWake]'))).toEqual([]) expect(lines.filter(line => line.includes('Wake fast-path for'))).toEqual([]) } finally { @@ -786,7 +786,7 @@ describe('HostReconciler wake fast-path — AP-1 fresh re-decide at the commit p expect(writes[0].lifecycle).toEqual({ state: 'active', wakeHandledGeneration: 5 }) expect(appsApi.patchNamespacedDeployment).toHaveBeenCalledTimes(1) const phases = logSpy.mock.calls - .map(args => String(args[0])) + .map(args => JSON.parse(String(args[0])).msg as string) .filter(line => line.startsWith('[StatelessWake]')) .map(line => /phase=(\w+)/.exec(line)?.[1]) expect(phases).toEqual(['wake_observed', 'status_flipped', 'replicas_patched']) @@ -814,7 +814,7 @@ describe('HostReconciler wake fast-path — AP-1 fresh re-decide at the commit p expect(writes.at(-1)?.lifecycle).toEqual({ state: 'active', wakeHandledGeneration: 2 }) expect(appsApi.patchNamespacedDeployment).toHaveBeenCalledTimes(1) const flipped = logSpy.mock.calls - .map(args => String(args[0])) + .map(args => JSON.parse(String(args[0])).msg as string) .filter(line => line.includes('phase=status_flipped')) expect(flipped).toHaveLength(1) } finally { @@ -839,7 +839,7 @@ describe('HostReconciler wake fast-path — AP-1 fresh re-decide at the commit p expect(customApi.patchNamespacedCustomObjectStatus).toHaveBeenCalledTimes(1) expect(appsApi.patchNamespacedDeployment).not.toHaveBeenCalled() const flipped = logSpy.mock.calls - .map(args => String(args[0])) + .map(args => JSON.parse(String(args[0])).msg as string) .filter(line => line.includes('phase=status_flipped')) expect(flipped).toEqual([]) } finally { diff --git a/host-context-controller/test/statelessLifecycleTracker.test.ts b/host-context-controller/test/statelessLifecycleTracker.test.ts index c6836057a6..01b6cfaa25 100644 --- a/host-context-controller/test/statelessLifecycleTracker.test.ts +++ b/host-context-controller/test/statelessLifecycleTracker.test.ts @@ -174,6 +174,150 @@ describe('StatelessLifecycleTracker — D8 idle rule', () => { }) }) + it('does not drain or suspend a stateless Host while channel authority is unavailable', async () => { + const port = makePort() + port.getEffectiveLifecycle.mockReturnValue({ + stateless: true, + state: 'active', + suspensionBlocked: true, + }) + const tracker = makeTracker({ port }) + for (let cycle = 0; cycle < 2; cycle++) { + expect(await tracker.handleHeartbeat(payload({ lastActivityTs: NOW - 10 * HOUR }))).toEqual({ + drain: false, + }) + } + expect(port.suspendHostFromHeartbeat).not.toHaveBeenCalled() + }) + + it('reverts a durable draining Host through a fresh read when a cached Host is stale', async () => { + const port = makePort() + const cachedHost = makeHost({ lifecycle: { state: 'active', wakeHandledGeneration: 3 } }) + port.getEffectiveLifecycle.mockReturnValue({ + stateless: true, + state: 'active', + suspensionBlocked: true, + }) + port.readFreshHost.mockResolvedValue( + makeHost({ lifecycle: { state: 'draining', wakeHandledGeneration: 3 } }) + ) + const tracker = makeTracker({ port, host: cachedHost }) + + await expect(tracker.handleHeartbeat(payload({ state: 'draining' }))).resolves.toEqual({ + drain: false, + }) + expect(port.markHostActiveFromHeartbeat).toHaveBeenCalledWith(cachedHost) + expect(port.suspendHostFromHeartbeat).not.toHaveBeenCalled() + }) + + it('fresh-checks a suspension-blocked Host once per channel authority loss epoch', async () => { + const port = makePort() + const blocked = { stateless: true, state: 'active', suspensionBlocked: true } + port.getEffectiveLifecycle.mockReturnValue(blocked) + const tracker = makeTracker({ port, host: makeHost() }) + + await expect(tracker.handleHeartbeat(payload())).resolves.toEqual({ drain: false }) + expect(port.getEffectiveLifecycle).toHaveBeenCalledTimes(1) + expect(port.markHostActiveFromHeartbeat).toHaveBeenCalledTimes(1) + + await expect(tracker.handleHeartbeat(payload())).resolves.toEqual({ drain: false }) + expect(port.getEffectiveLifecycle).toHaveBeenCalledTimes(2) + expect(port.markHostActiveFromHeartbeat).toHaveBeenCalledTimes(1) + + port.getEffectiveLifecycle.mockReturnValue({ stateless: true, state: 'active' }) + await expect(tracker.handleHeartbeat(payload())).resolves.toEqual({ drain: false }) + expect(port.getEffectiveLifecycle).toHaveBeenCalledTimes(3) + expect(port.markHostActiveFromHeartbeat).toHaveBeenCalledTimes(1) + + port.getEffectiveLifecycle.mockReturnValue(blocked) + await expect(tracker.handleHeartbeat(payload())).resolves.toEqual({ drain: false }) + expect(port.getEffectiveLifecycle).toHaveBeenCalledTimes(4) + expect(port.markHostActiveFromHeartbeat).toHaveBeenCalledTimes(2) + expect(port.suspendHostFromHeartbeat).not.toHaveBeenCalled() + }) + + it('repeats the suspension-blocked fresh check after the tracker is stopped', async () => { + const port = makePort() + port.getEffectiveLifecycle.mockReturnValue({ + stateless: true, + state: 'active', + suspensionBlocked: true, + }) + const tracker = makeTracker({ port, host: makeHost() }) + + await expect(tracker.handleHeartbeat(payload())).resolves.toEqual({ drain: false }) + await expect(tracker.handleHeartbeat(payload())).resolves.toEqual({ drain: false }) + // The epoch's single fresh check ran, and the second blocked beat skipped it. + expect(port.getEffectiveLifecycle).toHaveBeenCalledTimes(2) + expect(port.markHostActiveFromHeartbeat).toHaveBeenCalledTimes(1) + + tracker.stop() + await expect(tracker.handleHeartbeat(payload())).resolves.toEqual({ drain: false }) + expect(port.getEffectiveLifecycle).toHaveBeenCalledTimes(3) + expect(port.markHostActiveFromHeartbeat).toHaveBeenCalledTimes(2) + expect(port.suspendHostFromHeartbeat).not.toHaveBeenCalled() + }) + + it('reports a cancel-drain with nothing to revert as settled', async () => { + const port = makePort() + const tracker = makeTracker({ port }) + const cancelDrain = ( + tracker as unknown as { + cancelDrainOnEvidence(hostRef: string, host: HostCRD, bypass?: boolean): Promise + } + ).cancelDrainOnEvidence.bind(tracker) + const active = makeHost() + const suspended = makeHost({ lifecycle: { state: 'suspended', wakeHandledGeneration: 0 } }) + const draining = makeHost({ lifecycle: { state: 'draining', wakeHandledGeneration: 0 } }) + + // Cached active or suspended evidence needs no revert: settled, no write. + await expect(cancelDrain(active.name, active)).resolves.toBe(true) + await expect(cancelDrain(suspended.name, suspended)).resolves.toBe(true) + expect(port.markHostActiveFromHeartbeat).not.toHaveBeenCalled() + + // Liveness witness: cached draining evidence does reach the revert write. + await expect(cancelDrain(draining.name, draining)).resolves.toBe(true) + expect(port.markHostActiveFromHeartbeat).toHaveBeenCalledOnce() + expect(port.markHostActiveFromHeartbeat).toHaveBeenCalledWith(draining) + }) + + it('retries a failed suspension-blocked fresh check and still reverts cached draining evidence', async () => { + const port = makePort() + port.getEffectiveLifecycle.mockReturnValue({ + stateless: true, + state: 'active', + suspensionBlocked: true, + }) + port.markHostActiveFromHeartbeat.mockRejectedValueOnce(new Error('api down')) + let cachedHost = makeHost() + const tracker = new StatelessLifecycleTracker({ + idleMinutes: 30, + idleFloorMinutes: 15, + drainGraceMs: 60_000, + maxUptimeHours: 72, + reconciler: port as unknown as StatelessLifecycleReconcilerPort, + getHost: () => cachedHost, + now: () => NOW, + }) + const errors = vi.spyOn(console, 'error').mockImplementation(() => undefined) + try { + await tracker.handleHeartbeat(payload()) + await tracker.handleHeartbeat(payload()) + expect(port.markHostActiveFromHeartbeat).toHaveBeenCalledTimes(2) + expect(errors).toHaveBeenCalledOnce() + + await tracker.handleHeartbeat(payload()) + expect(port.markHostActiveFromHeartbeat).toHaveBeenCalledTimes(2) + + cachedHost = makeHost({ lifecycle: { state: 'draining', wakeHandledGeneration: 0 } }) + await tracker.handleHeartbeat(payload()) + expect(port.markHostActiveFromHeartbeat).toHaveBeenCalledTimes(3) + expect(port.markHostActiveFromHeartbeat).toHaveBeenLastCalledWith(cachedHost) + } finally { + errors.mockRestore() + } + }) + it('answers drain:false for an unknown host', async () => { const port = makePort() const tracker = new StatelessLifecycleTracker({ diff --git a/mcp-host/package-lock.json b/mcp-host/package-lock.json index fa85fc3a05..5778fe8962 100644 --- a/mcp-host/package-lock.json +++ b/mcp-host/package-lock.json @@ -1,12 +1,12 @@ { "name": "clerum-mcp-host", - "version": "0.1.471", + "version": "0.1.476", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "clerum-mcp-host", - "version": "0.1.471", + "version": "0.1.476", "license": "SEE LICENSE IN LICENSE", "dependencies": { "@anthropic-ai/sdk": "^0.32.0", diff --git a/mcp-host/package.json b/mcp-host/package.json index 2313280cd6..ece16c8289 100644 --- a/mcp-host/package.json +++ b/mcp-host/package.json @@ -1,6 +1,6 @@ { "name": "clerum-mcp-host", - "version": "0.1.471", + "version": "0.1.476", "description": "MCP Host - reads Host CRD and provides LLM access via OpenAI/Claude", "main": "dist/main.js", "engines": { diff --git a/mcp-host/src/__tests__/providers/zai.test.ts b/mcp-host/src/__tests__/providers/zai.test.ts index e24933353f..73b3a2516f 100644 --- a/mcp-host/src/__tests__/providers/zai.test.ts +++ b/mcp-host/src/__tests__/providers/zai.test.ts @@ -44,6 +44,7 @@ function toolCallResponse(toolName = 'search') { message: { role: 'assistant', content: null, + reasoning_content: 'Keep this reasoning across the tool call', tool_calls: [ { id: 'tc1', @@ -120,6 +121,42 @@ describe("ZaiProvider (OpenAICompatibleProvider, id='zai')", () => { expect(result.tool_calls![0].name).toBe('web_search') }) + it('preserves GLM reasoning_content on the assistant tool-call round trip', async () => { + const provider = makeZai('glm-5.3') + const mockClient = createMockOpenAIClient() + mockClient.chat.completions.create.mockResolvedValue(toolCallResponse('shell_exec')) + // eslint-disable-next-line @typescript-eslint/no-explicit-any + ;(provider as any)['client'] = mockClient + + const result = await provider.completeSingleTurnWithTools( + [ + { role: 'user', content: 'Run the probe' }, + { + role: 'assistant', + content: '', + reasoning_content: 'Prior GLM reasoning', + tool_calls: [ + { + id: 'previous-call', + name: 'shell_exec', + arguments: { command: 'printf prior' }, + }, + ], + }, + { role: 'tool', tool_call_id: 'previous-call', content: 'prior' }, + ], + [] + ) + + expect(result.reasoning_content).toBe('Keep this reasoning across the tool call') + const request = mockClient.chat.completions.create.mock.calls[0][0] + const assistant = request.messages.find( + (message: { role: string; tool_calls?: unknown }) => + message.role === 'assistant' && message.tool_calls + ) + expect(assistant.reasoning_content).toBe('Prior GLM reasoning') + }) + it('propagates API errors from the underlying client', async () => { const provider = makeZai() const mockClient = createMockOpenAIClient() diff --git a/mcp-host/src/core/orchestration/__tests__/toolUseLoop.test.ts b/mcp-host/src/core/orchestration/__tests__/toolUseLoop.test.ts index 6a3aa6ee46..d31ad928f7 100644 --- a/mcp-host/src/core/orchestration/__tests__/toolUseLoop.test.ts +++ b/mcp-host/src/core/orchestration/__tests__/toolUseLoop.test.ts @@ -8,8 +8,11 @@ import { VisualInputBudget } from '../../../visualInput/policy' import { makeFakeConversation } from '../../conversation/__testing__/makeFakeConversation' import { LlmError, LlmErrorCode } from '../../errors' import type { ReasoningPort, Tool, ToolRegistry } from '../../interfaces' +import { DefaultReasoningPort } from '../../reasoning/port' +import { DefaultPromptBuilder } from '../../reasoning/promptBuilder' import { BasicSafety } from '../../safety/safety' -import type { Attachment, RespondResult, ToolOutput } from '../../types' +import type { Attachment, ChatMessage, RespondResult, ToolOutput } from '../../types' +import { FinishReason } from '../../types' import { SimpleEventEmitter } from '../eventEmitter' import { buildLoopConfig } from '../loopConfig' import { @@ -153,6 +156,47 @@ describe('runToolUseLoop — loop control', () => { } }) + it('preserves provider reasoning_content from ReasoningPort through the next tool continuation', async () => { + const completeWithTools = vi + .fn() + .mockResolvedValueOnce({ + content: null, + tool_calls: [{ id: 'tc_reason', name: 'noop', arguments: {} }], + reasoning_content: 'GLM reasoning must survive the tool boundary', + usage: { input_tokens: 10, output_tokens: 5, total_tokens: 15 }, + finish_reason: FinishReason.ToolUse, + }) + .mockResolvedValueOnce({ + content: 'done after reasoning', + tool_calls: null, + usage: { input_tokens: 11, output_tokens: 2, total_tokens: 13 }, + finish_reason: FinishReason.Stop, + }) + const reasoning = new DefaultReasoningPort( + { complete: vi.fn(), completeWithTools, modelName: () => 'glm-5.3' }, + new DefaultPromptBuilder() + ) + + const config = buildLoopConfig({ + reasoning, + toolRegistry: createMockRegistry([createMockTool('noop')]), + safety: new BasicSafety(), + events: new SimpleEventEmitter(), + conversation: makeFakeConversation(), + }) + + const result = await runToolUseLoop(config, [{ role: 'user', content: 'Use the tool' }]) + + expect(result.type).toBe('response') + expect(completeWithTools).toHaveBeenCalledTimes(2) + const continuation = completeWithTools.mock.calls[1]?.[0] + const assistant = continuation?.messages.find( + (message: ChatMessage) => + message.role === 'assistant' && (message.tool_calls?.length ?? 0) > 0 + ) + expect(assistant?.reasoning_content).toBe('GLM reasoning must survive the tool boundary') + }) + it('should stop at maxIterations when reasoning always returns tool_calls (Risk 4.5a)', async () => { const toolCalls = { type: 'tool_calls' as const, @@ -590,6 +634,35 @@ describe('runToolUseLoop — loop control', () => { expect(workflowTriggerTool.execute).toHaveBeenCalledTimes(1) }) + it('recovers a bare hyphenated workflow slug when the LLM omits workflow_trigger', async () => { + const reasoning = createMockReasoning([ + { type: 'text', content: 'I can help with that.' }, + { + type: 'tool_calls', + calls: [{ id: 'tc_1', name: 'workflow_trigger', arguments: { name: 'risk-review' } }], + }, + { type: 'text', content: 'The workflow request is now being handled.' }, + ]) + const workflowTriggerTool = createMockTool('workflow_trigger', { + sanitize: false, + output: JSON.stringify({ workflowName: 'risk-review', phase: 'Pending' }), + }) + const config = buildLoopConfig({ + reasoning, + toolRegistry: createMockRegistry([workflowTriggerTool]), + safety: new BasicSafety(), + events: new SimpleEventEmitter(), + conversation: makeFakeConversation(), + maxIterations: 4, + }) + + const result = await runToolUseLoop(config, [{ role: 'user', content: 'Run risk-review' }]) + + expect(result.type).toBe('response') + expect(reasoning.respondWithTools).toHaveBeenCalledTimes(2) + expect(workflowTriggerTool.execute).toHaveBeenCalledTimes(1) + }) + it('synthesizes a workflow_list answer when the LLM omits returned recipe names', async () => { const reasoning = createMockReasoning([ { diff --git a/mcp-host/src/core/orchestration/toolUseLoop.ts b/mcp-host/src/core/orchestration/toolUseLoop.ts index b8926fb5ef..772bf375fb 100644 --- a/mcp-host/src/core/orchestration/toolUseLoop.ts +++ b/mcp-host/src/core/orchestration/toolUseLoop.ts @@ -333,6 +333,7 @@ export async function runToolUseLoop( role: 'assistant', content: result.content ?? '', tool_calls: result.calls, + reasoning_content: result.reasoning_content, }) for (const message of messages) { diff --git a/mcp-host/src/core/orchestration/toolUseLoopIntentRecovery.test.ts b/mcp-host/src/core/orchestration/toolUseLoopIntentRecovery.test.ts new file mode 100644 index 0000000000..d3fab8ce95 --- /dev/null +++ b/mcp-host/src/core/orchestration/toolUseLoopIntentRecovery.test.ts @@ -0,0 +1,102 @@ +import { describe, expect, it } from 'vitest' +import { shouldRecoverWorkflowTriggerTextResponse } from './toolUseLoopIntentRecovery' + +describe('shouldRecoverWorkflowTriggerTextResponse', () => { + it('does not classify a native shell command marker as a workflow recipe', () => { + expect( + shouldRecoverWorkflowTriggerTextResponse( + "Run exactly this command: printf 'PR849-SHELL-20260926'", + 'The command completed.' + ) + ).toBe(false) + }) + + it('recovers when a named workflow recipe request receives a text-only answer', () => { + expect( + shouldRecoverWorkflowTriggerTextResponse( + 'Run the quarterly-report workflow recipe', + 'I cannot find that report.' + ) + ).toBe(true) + }) + + it('recovers an explicit workflow recipe request without a hyphenated name', () => { + expect( + shouldRecoverWorkflowTriggerTextResponse( + 'Run the workflow recipe onboarding', + 'I cannot complete that request.' + ) + ).toBe(true) + }) + + it.each([ + 'Run "quarterly-report"', + "Run 'quarterly-report'", + 'Run “quarterly-report”', + 'Run ‘quarterly-report’', + 'Please run quarterly-report', + 'Launch quarterly-report', + 'Run quarterly-report.', + 'Please launch “quarterly-report”.', + ])('recovers a direct named recipe request: %s', userText => { + expect(shouldRecoverWorkflowTriggerTextResponse(userText, 'I can help with that.')).toBe(true) + }) + + // These all returned true on origin/dev (any hyphenated token plus a trigger verb). + it.each([ + 'run the daily-report', + 'can you run daily-report', + 'could you please run daily-report', + 'run daily-report now', + 'run daily-report please', + 'please run daily-report!', + 'run daily-report?', + 'trigger daily-report with region=eu', + 'Run "daily-report" now.', + 'execute daily-report using region=eu and dry-run=true', + 'run daily-report, please', + 'run daily-report,', + 'please run daily-report, now', + 'run daily-report;', + 'run daily-report: now', + ])('recovers a direct named recipe request with a natural tail: %s', userText => { + expect(shouldRecoverWorkflowTriggerTextResponse(userText, 'I can help with that.')).toBe(true) + }) + + it.each([ + 'Run a shell command', + 'run daily-report with', + 'run daily-report and delete it', + // Paired control for the trailing-comma positives: a comma followed by a + // non-filler tail is still not a trigger. + 'run daily-report, and delete it', + 'run daily-report now and then rm -rf /', + 'Run quarterly-report and delete it', + `Run ${'a'.repeat(129)}-report`, + ])('does not infer a recipe from unrelated or oversized text: %s', userText => { + expect(shouldRecoverWorkflowTriggerTextResponse(userText, 'I can help with that.')).toBe(false) + }) + + it.each([ + ['whitespace after the verb', `run${' '.repeat(200_000)}x`], + ['leading whitespace', `${' '.repeat(200_000)}run`], + ['one long token', `run ${'a'.repeat(200_000)}`], + ['repeated hyphen groups', `run ${'a-'.repeat(100_000)}`], + ['long tail of filler', `run daily-report${' now'.repeat(50_000)} !`], + ['whitespace before a rejected tail', `run daily-report${' '.repeat(200_000)}x`], + ['long punctuation run', `run daily-report${'!'.repeat(200_000)}x`], + ['repeated verbs', 'run '.repeat(50_000)], + ])('classifies adversarial input in linear time: %s', (_label, userText) => { + expect(userText.length).toBeGreaterThanOrEqual(150_000) + const started = performance.now() + const result = shouldRecoverWorkflowTriggerTextResponse(userText, 'I can help with that.') + const elapsedMs = performance.now() - started + // Liveness witness: a boolean means the classifier ran to completion. + expect(typeof result).toBe('boolean') + // The bound guards against pathological (super-linear) backtracking, not + // latency: linear classification of 200k characters takes a few ms, while + // quadratic backtracking takes seconds. 1000 ms leaves headroom for shared + // CI runners without hiding a regression. + expect(elapsedMs).toBeLessThan(1000) + }) +}) diff --git a/mcp-host/src/core/orchestration/toolUseLoopIntentRecovery.ts b/mcp-host/src/core/orchestration/toolUseLoopIntentRecovery.ts index 8260ef2734..128822290d 100644 --- a/mcp-host/src/core/orchestration/toolUseLoopIntentRecovery.ts +++ b/mcp-host/src/core/orchestration/toolUseLoopIntentRecovery.ts @@ -82,6 +82,43 @@ export function shouldRecoverWorkflowArtifactTextResponse(userText: string): boo return namesWorkflow && includesRecipeLikeName } +// A direct trigger is `[can you] [please] [the] ` followed only +// by filler (`now`, `please`, `again`, `immediately`), trailing punctuation +// (`.`, `!`, `?`, `,`, `;`, `:`), or `with|using` +// arguments. The head regex is anchored and every quantifier is bounded or +// separated from its neighbours by a literal, so it stays linear on adversarial +// input; the tail is checked with string operations instead of a second regex. +const DIRECT_TRIGGER_HEAD = + /^\s*(?:(?:can|could|would|will)\s+you\s+)?(?:please\s+)?(?:run|execute|trigger|start|launch)\s+(?:the\s+)?(\S{1,128})(?=\s|$)/i +const DIRECT_TRIGGER_FILLER = new Set(['now', 'again', 'please', 'immediately']) +const RECIPE_NAME = /^[a-z0-9]+(?:-[a-z0-9]+)+$/i + +function stripTrailingSentencePunctuation(value: string): string { + let end = value.length + while (end > 0 && '.!?,;:'.includes(value[end - 1])) end-- + return value.slice(0, end) +} + +function isDirectTriggerTail(tail: string): boolean { + const words = tail.trim().split(/\s+/) + for (let i = 0; i < words.length; i++) { + const word = stripTrailingSentencePunctuation(words[i]).toLowerCase() + if (word === '' || DIRECT_TRIGGER_FILLER.has(word)) continue + return (word === 'with' || word === 'using') && i + 1 < words.length + } + return true +} + +function isDirectRecipeTrigger(userText: string): boolean { + const head = DIRECT_TRIGGER_HEAD.exec(userText) + if (head === null) return false + const candidate = stripTrailingSentencePunctuation( + head[1].replace(/[\u0060\u0022\u0027\u201C\u201D\u2018\u2019]/g, '') + ) + if (!RECIPE_NAME.test(candidate)) return false + return isDirectTriggerTail(userText.slice(head[0].length)) +} + export function shouldRecoverWorkflowTriggerTextResponse( userText: string, responseText: string @@ -89,9 +126,13 @@ export function shouldRecoverWorkflowTriggerTextResponse( if (isWorkflowListIntent(userText)) return false const normalizedUser = userText.toLowerCase() + const namesDirectRecipe = isDirectRecipeTrigger(userText) const asksToTrigger = /\b(run|start|trigger|execute|launch)\b/.test(normalizedUser) const includesRecipeLikeName = /\b[a-z0-9]+(?:-[a-z0-9]+)+\b/.test(normalizedUser) - const namesWorkflow = /\bworkflow\s+recipe\b/.test(normalizedUser) || includesRecipeLikeName + const namesWorkflow = + /\bworkflow\s+recipe\b/.test(normalizedUser) || + namesDirectRecipe || + (includesRecipeLikeName && /\b(?:workflow|recipe)\b/.test(normalizedUser)) if (!asksToTrigger || !namesWorkflow) return false const normalizedResponse = responseText.toLowerCase() diff --git a/mcp-host/src/core/reasoning/port.ts b/mcp-host/src/core/reasoning/port.ts index d80a7a26b4..6b0cf14e69 100644 --- a/mcp-host/src/core/reasoning/port.ts +++ b/mcp-host/src/core/reasoning/port.ts @@ -205,6 +205,7 @@ export class DefaultReasoningPort implements ReasoningPort { type: 'tool_calls', calls: response.tool_calls, content: response.content ?? undefined, + reasoning_content: response.reasoning_content, usage: response.usage, } } diff --git a/mcp-host/src/core/types.ts b/mcp-host/src/core/types.ts index 922967561c..191244f45e 100644 --- a/mcp-host/src/core/types.ts +++ b/mcp-host/src/core/types.ts @@ -63,6 +63,13 @@ export interface ChatMessage { tool_call_id?: string name?: string tool_calls?: ToolCall[] | null + /** + * Provider reasoning that must accompany the assistant tool call on the next + * OpenAI-compatible request. ZAI GLM reasoning models require this field to + * remain on the same tool-call message; it is absent for providers that do + * not return reasoning. + */ + reasoning_content?: string | null /** * IronClaw invariant #2 (P.3, Opción D): lateral field that points at a * spillover blob written by T1.5. `content` always stays a string (a rich @@ -264,6 +271,7 @@ export interface ToolCompletionRequest { export interface ToolCompletionResponse { content: string | null tool_calls: ToolCall[] | null + reasoning_content?: string | null usage: TokenUsage /** True only when the provider response carried authoritative token counters. */ usage_reported?: boolean @@ -283,7 +291,13 @@ export interface ReasoningContext { export type RespondResult = | { type: 'text'; content: string } - | { type: 'tool_calls'; calls: ToolCall[]; content?: string; usage?: TokenUsage } + | { + type: 'tool_calls' + calls: ToolCall[] + content?: string + reasoning_content?: string | null + usage?: TokenUsage + } | { type: 'need_approval'; approval: PendingApproval } | { type: 'error'; error: Error } diff --git a/mcp-host/src/llm/__tests__/openai.singleTurn.test.ts b/mcp-host/src/llm/__tests__/openai.singleTurn.test.ts index 7827e34edf..a8a65c2dd3 100644 --- a/mcp-host/src/llm/__tests__/openai.singleTurn.test.ts +++ b/mcp-host/src/llm/__tests__/openai.singleTurn.test.ts @@ -209,6 +209,30 @@ describe('OpenAI message conversion', () => { expect(assistantMsg.tool_calls[0].function.arguments).toBe('{"query":"X"}') expect(typeof assistantMsg.tool_calls[0].function.arguments).toBe('string') }) + + it('should omit ZAI reasoning_content from native OpenAI requests', async () => { + const mockClient = createMockOpenAIClient() + mockClient.chat.completions.create.mockResolvedValue(openAITextResponse('Done')) + const provider = new OpenAIProvider(mockClient as any, 'gpt-4o') + + const messages: ChatMessage[] = [ + { role: 'user', content: 'Continue after a ZAI fallback' }, + { + role: 'assistant', + content: '', + reasoning_content: 'Prior ZAI reasoning', + tool_calls: [{ id: 'tc_zai', name: 'search', arguments: { query: 'fallback' } }], + }, + { role: 'tool', content: 'result', tool_call_id: 'tc_zai' }, + ] + + await provider.completeSingleTurnWithTools(messages, []) + + const callArgs = mockClient.chat.completions.create.mock.calls[0][0] + const assistantMsg = callArgs.messages.find((m: any) => m.role === 'assistant' && m.tool_calls) + expect(assistantMsg).toBeDefined() + expect(assistantMsg).not.toHaveProperty('reasoning_content') + }) }) describe('OpenAI contentParts in user messages (screenshot images)', () => { diff --git a/mcp-host/src/llm/openai.ts b/mcp-host/src/llm/openai.ts index 3659070594..e518177362 100644 --- a/mcp-host/src/llm/openai.ts +++ b/mcp-host/src/llm/openai.ts @@ -47,6 +47,15 @@ export class OpenAIProvider implements SingleTurnProvider { return /^(?:gpt-5(?:[.-]|$)|o[1-9](?:[.-]|$))/i.test(this.defaultModel) } + /** + * ZAI requires reasoning to remain on assistant tool-call messages across a + * tool turn. Native OpenAI and Azure keep the standard message shape, so a + * provider-specific field from prior ZAI history must not leak during failover. + */ + protected supportsReasoningContent(): boolean { + return false + } + private tokenLimitOptions(maxTokens: number | undefined): { max_tokens?: number max_completion_tokens?: number @@ -149,6 +158,7 @@ export class OpenAIProvider implements SingleTurnProvider { name: tc.function.name, arguments: this.parseArguments(tc.function.arguments), })) ?? null + const reasoningContent = (message as { reasoning_content?: unknown }).reasoning_content const usageReported = response.usage != null && @@ -159,6 +169,7 @@ export class OpenAIProvider implements SingleTurnProvider { return { content: message.content, tool_calls: toolCalls && toolCalls.length > 0 ? toolCalls : null, + reasoning_content: typeof reasoningContent === 'string' ? reasoningContent : null, usage: { input_tokens: response.usage?.prompt_tokens ?? 0, output_tokens: response.usage?.completion_tokens ?? 0, @@ -197,7 +208,7 @@ export class OpenAIProvider implements SingleTurnProvider { } if (m.role === 'assistant') { if (m.tool_calls) { - return { + const assistantMessage = { role: 'assistant' as const, content: m.content || null, tool_calls: m.tool_calls.map(tc => ({ @@ -208,7 +219,11 @@ export class OpenAIProvider implements SingleTurnProvider { arguments: JSON.stringify(tc.arguments), }, })), + } as Record + if (m.reasoning_content && this.supportsReasoningContent()) { + assistantMessage.reasoning_content = m.reasoning_content } + return assistantMessage as unknown as OpenAI.ChatCompletionAssistantMessageParam } return { role: 'assistant' as const, content: m.content } } diff --git a/mcp-host/src/llm/openaiCompatible.ts b/mcp-host/src/llm/openaiCompatible.ts index 975d0bddc5..9a63d0c236 100644 --- a/mcp-host/src/llm/openaiCompatible.ts +++ b/mcp-host/src/llm/openaiCompatible.ts @@ -152,6 +152,10 @@ export class OpenAICompatibleProvider extends OpenAIProvider { return false } + protected override supportsReasoningContent(): boolean { + return this.cfg.id === 'zai' + } + override completeSingleTurn( messages: CoreChatMessage[], options?: { max_tokens?: number; temperature?: number; signal?: AbortSignal } diff --git a/mcp-host/src/server/__tests__/sessionRouteHandlers.test.ts b/mcp-host/src/server/__tests__/sessionRouteHandlers.test.ts index e71a4777dc..7cccb357e2 100644 --- a/mcp-host/src/server/__tests__/sessionRouteHandlers.test.ts +++ b/mcp-host/src/server/__tests__/sessionRouteHandlers.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it } from 'vitest' +import { describe, expect, it, vi } from 'vitest' import { ConversationManager } from '../../core/conversation/conversation' import { buildConnectRequiredApproval } from '../../core/extensions/mcpApprovalGateController' import { BasicSafety } from '../../core/safety/safety' @@ -112,6 +112,54 @@ describe('createSessionRouteHandlers — handleSessionsList (R1-L2)', () => { }) }) +describe('createSessionRouteHandlers — handleSessionsList cursor validation', () => { + it('serves the next page for a valid cursor with exactly one store read (control)', async () => { + const { convManager, handleSessionsList } = makeHandlersUnderTest() + await seed(convManager, 'user-A:rpc:agent-x:chat-1') + await seed(convManager, 'user-A:rpc:agent-x:chat-2') + const first = await handleSessionsList('user-A', { agent: 'agent-x', limit: 1 }) + expect(first.items).toHaveLength(1) + expect(first.nextCursor).toEqual(expect.any(String)) + + const listSpy = vi.spyOn(convManager, 'listSessionSummariesForUserAsync') + const second = await handleSessionsList('user-A', { + agent: 'agent-x', + limit: 1, + cursor: first.nextCursor, + }) + + expect(listSpy).toHaveBeenCalledTimes(1) + expect(second.items).toHaveLength(1) + expect(second.items[0]?.chatId).not.toBe(first.items[0]?.chatId) + }) + + it.each([ + ['a malformed cursor', () => 'not-a-cursor'], + ['a cursor minted for another agent scope', (otherScopeCursor: string) => otherScopeCursor], + ])( + 'rejects %s instead of silently serving page 1, without reading the store', + async (_label, pick) => { + const { convManager, handleSessionsList } = makeHandlersUnderTest() + await seed(convManager, 'user-A:rpc:agent-x:chat-1') + await seed(convManager, 'user-A:rpc:agent-x:chat-2') + await seed(convManager, 'user-A:rpc:agent-y:chat-3') + await seed(convManager, 'user-A:rpc:agent-y:chat-4') + const otherScope = await handleSessionsList('user-A', { agent: 'agent-y', limit: 1 }) + expect(otherScope.nextCursor).toEqual(expect.any(String)) + + const listSpy = vi.spyOn(convManager, 'listSessionSummariesForUserAsync') + await expect( + handleSessionsList('user-A', { + agent: 'agent-x', + limit: 1, + cursor: pick(otherScope.nextCursor as string), + }) + ).rejects.toThrow('Invalid sessions cursor') + expect(listSpy).toHaveBeenCalledTimes(0) + } + ) +}) + describe('createSessionRouteHandlers — handleSessionsList title projection (spec 15 A12)', () => { async function seedWithTitle( convManager: ConversationManager, diff --git a/mcp-host/src/server/sessionRouteHandlers.ts b/mcp-host/src/server/sessionRouteHandlers.ts index 65c44cc502..90dd2646ee 100644 --- a/mcp-host/src/server/sessionRouteHandlers.ts +++ b/mcp-host/src/server/sessionRouteHandlers.ts @@ -91,6 +91,12 @@ export function createSessionRouteHandlers(deps: SessionRouteHandlerDeps) { const keyPrefix = agent ? `${userSub}:rpc:${agent}:` : `${userSub}:rpc:` const cursorScope = sessionsCursorScope(userSub, agent) const cursor = decodeSessionsCursor(query.cursor, cursorScope) + // A cursor that does not decode for this scope is a caller error. Treating + // it as "no cursor" would silently restart the listing at page 1. The HTTP + // route rejects it with 400 before reaching here; this guards other callers. + if (query.cursor !== undefined && cursor === null) { + throw new Error('Invalid sessions cursor') + } const entries = await convManager.listSessionSummariesForUserAsync(keyPrefix, { limit: query.limit === undefined ? undefined : query.limit + 1, cursor: cursor diff --git a/mcp-host/src/workflow/internalTools.ts b/mcp-host/src/workflow/internalTools.ts index 4b2623b522..64122da9ba 100644 --- a/mcp-host/src/workflow/internalTools.ts +++ b/mcp-host/src/workflow/internalTools.ts @@ -11,28 +11,10 @@ * chart.js + @napi-rs/canvas (PNG charts), built-in string write * for markdown and the dashboard HTML wrapper. */ -import { type SKRSContext2D, createCanvas } from '@napi-rs/canvas' -import { Chart, type ChartConfiguration, type ChartType, registerables } from 'chart.js' -import { - AlignmentType, - BorderStyle, - Document as DocxDocument, - Footer as DocxFooter, - Header as DocxHeader, - Table as DocxTable, - TableCell as DocxTableCell, - TableRow as DocxTableRow, - HeadingLevel, - ImageRun, - LevelFormat, - Packer, - PageNumber, - Paragraph, - ShadingType, - TextRun, - WidthType, -} from 'docx' -import ExcelJS from 'exceljs' +import type { SKRSContext2D } from '@napi-rs/canvas' +import type { Chart, ChartConfiguration, ChartType } from 'chart.js' +import type { Table as DocxTable, Paragraph, TextRun } from 'docx' +import type ExcelJS from 'exceljs' import * as fs from 'fs' import * as path from 'path' import type { @@ -49,16 +31,14 @@ import { WorkflowTriggerTool } from '../core/tools/workflowTriggerTool' import { CONTEXT_FILES_TOOLS, loadContextFilesMounts } from './contextFiles' import type { ArtifactMetadata, InternalToolDefinition, InternalToolResult } from './types' -// eslint-disable-next-line @typescript-eslint/no-require-imports -const PdfPrinter = require('pdfmake') - -// eslint-disable-next-line @typescript-eslint/no-require-imports -const PptxGenJS = require('pptxgenjs') - -// Register Chart.js controllers/scales/elements/plugins once. Chart.js v4 ships -// these as separate exports so we have to opt in. `registerables` includes all -// chart types we expose (line, bar, pie, doughnut, etc.) plus axes/legend/title. -Chart.register(...registerables) +type DocxModule = typeof import('docx') +// pptxgenjs exposes these lookup tables on instances at runtime, though its +// declaration omits them from the instance type. +type PptxGenJS = InstanceType & { + charts: Record + shapes: Record +} +let chartRegistered = false // Accessor to the current Host CRD, injected by main (avoids a circular import). // Re-read on every getOutputDir() call because `currentHost` is hydrated async @@ -746,6 +726,15 @@ const generateChart: InternalToolDefinition = { ], } + const [{ createCanvas }, { Chart, registerables }] = await Promise.all([ + import('@napi-rs/canvas'), + import('chart.js'), + ]) + // Chart.js controllers, scales, and plugins are needed only for PNG charts. + if (!chartRegistered) { + Chart.register(...registerables) + chartRegistered = true + } const canvas = createCanvas(width, height) const ctx = canvas.getContext('2d') as SKRSContext2D // Chart.js types target a browser CanvasRenderingContext2D; the Skia @@ -1368,6 +1357,11 @@ const generatePdf: InternalToolDefinition = { content, } + // The published pdfmake types describe its browser API; this tool uses + // the package's Node printer constructor, as the previous require did. + const PdfPrinter = (await import('pdfmake')).default as unknown as new ( + fonts: TFontDictionary + ) => { createPdfKitDocument: (definition: TDocumentDefinitions) => import('stream').Duplex } const printer = new PdfPrinter(PDF_FONTS) const pdfDoc = printer.createPdfKitDocument(docDef) @@ -1477,7 +1471,12 @@ interface DocxTableSpec { * Parse inline markdown in a single line into docx TextRun objects. * Supports **bold**, *italic*, `code`. Falls through as plain text otherwise. */ -function parseInlineMarkdownToRuns(text: string, baseColor?: string): TextRun[] { +function parseInlineMarkdownToRuns( + text: string, + baseColor: string | undefined, + docx: DocxModule +): TextRun[] { + const { TextRun } = docx // Same logic as parseInlineMarkdown above: non-greedy bold so nested // single asterisks (italic) don't break `**bold *italic***`. const runs: TextRun[] = [] @@ -1521,8 +1520,19 @@ function buildDocxTable( headers: string[], rows: string[][], palette: DocxPalette, - layout: 'striped' | 'minimal' | 'grid' = 'striped' + layout: 'striped' | 'minimal' | 'grid', + docx: DocxModule ): DocxTable { + const { + BorderStyle, + Paragraph, + ShadingType, + Table: DocxTable, + TableCell: DocxTableCell, + TableRow: DocxTableRow, + TextRun, + WidthType, + } = docx // Pad/truncate each row to headers.length so ragged input doesn't // produce a misaligned table that Word renders silently wrong. const normalizedRows = rows.map(row => normalizeRowLength(row, headers.length)) @@ -1554,7 +1564,7 @@ function buildDocxTable( : {}), children: [ new Paragraph({ - children: parseInlineMarkdownToRuns(cell ?? '', docxHex(palette.text)), + children: parseInlineMarkdownToRuns(cell ?? '', docxHex(palette.text), docx), }), ], margins: { top: 60, bottom: 60, left: 120, right: 120 }, @@ -1624,7 +1634,12 @@ function splitTableRowDocx(line: string): string[] { * Convert markdown body into docx Paragraph + Table objects, applying palette * colors to headings. Mirrors the GFM subset supported by the PDF tool. */ -function bodyToDocxChildren(body: string, palette: DocxPalette): (Paragraph | DocxTable)[] { +function bodyToDocxChildren( + body: string, + palette: DocxPalette, + docx: DocxModule +): (Paragraph | DocxTable)[] { + const { BorderStyle, HeadingLevel, Paragraph, TextRun } = docx const out: (Paragraph | DocxTable)[] = [] const lines = body.split('\n') let i = 0 @@ -1700,7 +1715,7 @@ function bodyToDocxChildren(body: string, palette: DocxPalette): (Paragraph | Do rows.push(splitTableRowDocx(lines[i])) i++ } - out.push(buildDocxTable(headers, rows, palette, 'striped')) + out.push(buildDocxTable(headers, rows, palette, 'striped', docx)) // Trailing empty paragraph for spacing. out.push(new Paragraph({ children: [new TextRun({ text: '' })], spacing: { before: 60 } })) continue @@ -1714,7 +1729,8 @@ function bodyToDocxChildren(body: string, palette: DocxPalette): (Paragraph | Do numbering: { reference: 'doc-bullets', level: 0 }, children: parseInlineMarkdownToRuns( lines[i].trimStart().slice(2), - docxHex(palette.text) + docxHex(palette.text), + docx ), }) ) @@ -1754,7 +1770,7 @@ function bodyToDocxChildren(body: string, palette: DocxPalette): (Paragraph | Do out.push( new Paragraph({ spacing: { after: 80 }, - children: parseInlineMarkdownToRuns(line, docxHex(palette.text)), + children: parseInlineMarkdownToRuns(line, docxHex(palette.text), docx), }) ) i++ @@ -1847,6 +1863,20 @@ const generateDocx: InternalToolDefinition = { const branding = (args.branding ?? {}) as DocxBranding const images = (args.images ?? []) as DocxImageRef[] const tables = (args.tables ?? []) as DocxTableSpec[] + const docx = await import('docx') + const { + AlignmentType, + Document: DocxDocument, + Footer: DocxFooter, + Header: DocxHeader, + HeadingLevel, + ImageRun, + LevelFormat, + Packer, + PageNumber, + Paragraph, + TextRun, + } = docx const children: (Paragraph | DocxTable)[] = [] @@ -1905,13 +1935,13 @@ const generateDocx: InternalToolDefinition = { } // Body. - children.push(...bodyToDocxChildren(body, palette)) + children.push(...bodyToDocxChildren(body, palette, docx)) // Explicit tables (after body). for (const t of tables) { if (!t || !Array.isArray(t.headers) || !Array.isArray(t.rows)) continue const stringRows = t.rows.map(r => r.map(c => (c == null ? '' : String(c)))) - children.push(buildDocxTable(t.headers, stringRows, palette, t.layout ?? 'striped')) + children.push(buildDocxTable(t.headers, stringRows, palette, t.layout ?? 'striped', docx)) children.push(new Paragraph({ children: [new TextRun({ text: '' })] })) } @@ -2418,6 +2448,7 @@ const generateXlsx: InternalToolDefinition = { return { success: false, error: 'sheets must be a non-empty array' } } + const { default: ExcelJS } = await import('exceljs') const workbook = new ExcelJS.Workbook() workbook.creator = branding.companyName ?? '' workbook.company = branding.companyName ?? '' @@ -4987,7 +5018,7 @@ function loadSafeImage(p: string | undefined, outputDir: string): string | null * Map our chart type names (matching clerum__generate_chart) to * pptxgenjs's internal chart-type strings. */ -function pptxChartTypeFromName(t: PptxChartType, pptx: typeof PptxGenJS): string { +function pptxChartTypeFromName(t: PptxChartType, pptx: PptxGenJS): string { const c = pptx.charts ?? {} switch (t) { case 'line': @@ -5032,7 +5063,7 @@ function pptxBodyRegion(dims: PptxDimensions): { /** Add the standard footer (company / footerText / page number) to every slide. */ function addPptxFooter( - slide: Record & { addText: (...args: unknown[]) => void }, + slide: ReturnType, region: ReturnType, dims: PptxDimensions, branding: PptxBranding, @@ -5563,7 +5594,8 @@ const generatePptxTool: InternalToolDefinition = { } } - const pptx = new PptxGenJS() + const { default: PptxGenJS } = await import('pptxgenjs') + const pptx = new PptxGenJS() as PptxGenJS pptx.layout = layoutDef.name const dims = layoutDef.dims const region = pptxBodyRegion(dims) @@ -5623,7 +5655,7 @@ interface RenderCtx { branding: PptxBranding safeLogoPath: string | null outputDir: string - pptx: typeof PptxGenJS + pptx: PptxGenJS } // eslint-disable-next-line @typescript-eslint/no-explicit-any diff --git a/rpc-proxy/package-lock.json b/rpc-proxy/package-lock.json index fa235616b6..5bb586f388 100644 --- a/rpc-proxy/package-lock.json +++ b/rpc-proxy/package-lock.json @@ -1,12 +1,12 @@ { "name": "clerum-rpc-proxy", - "version": "0.1.94", + "version": "0.1.111", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "clerum-rpc-proxy", - "version": "0.1.94", + "version": "0.1.111", "dependencies": { "cors": "^2.8.5", "express": "^4.21.2", diff --git a/rpc-proxy/package.json b/rpc-proxy/package.json index cbaa8cf892..7544d37ea6 100644 --- a/rpc-proxy/package.json +++ b/rpc-proxy/package.json @@ -1,6 +1,6 @@ { "name": "clerum-rpc-proxy", - "version": "0.1.94", + "version": "0.1.111", "description": "External-facing RPC proxy for tenant-scoped MCP access", "main": "dist/main.js", "engines": { diff --git a/rpc-proxy/src/__tests__/approval-artifacts.test.ts b/rpc-proxy/src/__tests__/approval-artifacts.test.ts index b3d9accf98..f29b3f84fa 100644 --- a/rpc-proxy/src/__tests__/approval-artifacts.test.ts +++ b/rpc-proxy/src/__tests__/approval-artifacts.test.ts @@ -2,6 +2,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import express from 'express' import request from 'supertest' import { config } from '../config.js' +import { apiErrorHandler } from '../errorHandler.js' import { createRpcRouter } from '../routes/rpc.js' // ── Hoisted mocks ─────────────────────────────────────────────────── @@ -76,12 +77,7 @@ function makeApp() { const app = express() app.use(express.json()) app.use(createRpcRouter()) - // Generic error handler to avoid unhandled rejections in tests - app.use( - (err: unknown, _req: express.Request, res: express.Response, _next: express.NextFunction) => { - res.status(500).json({ error: err instanceof Error ? err.message : 'Internal error' }) - } - ) + app.use(apiErrorHandler) return app } @@ -104,6 +100,8 @@ function mockFetchResponse( // ── Setup / Teardown ──────────────────────────────────────────────── const originalFetch = globalThis.fetch const originalArtifactDownloadMaxBytes = config.artifactDownloadMaxBytes +const originalArtifactDownloadTimeoutMs = config.artifactDownloadTimeoutMs +const originalUpstreamTimeoutMs = config.upstreamTimeoutMs beforeEach(() => { authTokenMock.verifyRpcToken.mockReset() @@ -123,12 +121,44 @@ beforeEach(() => { afterEach(() => { globalThis.fetch = originalFetch config.artifactDownloadMaxBytes = originalArtifactDownloadMaxBytes + config.artifactDownloadTimeoutMs = originalArtifactDownloadTimeoutMs + config.upstreamTimeoutMs = originalUpstreamTimeoutMs }) // ===================================================================== // Approval Routes // ===================================================================== describe('POST /rpc/hosts/:hostRef/approvals/approve', () => { + // R2-M3: approve mutates state on the host. A client-side abort of the first + // attempt would answer 504 for an approval the host may already have applied. + // Real-socket coverage (slow upstream, hung retry) lives in + // route-upstream-deadlines.real-http.test.ts. + it('sends the first approval attempt without a client-side deadline', async () => { + config.upstreamTimeoutMs = 20 + // The first attempt carries only the client-disconnect signal (R3-L6): it + // must outlive several upstream timeouts without being aborted. + let firstSignal: AbortSignal | undefined + let abortedWhenAnswered: boolean | undefined + const fetchMock = vi.fn().mockImplementation(async (_url: string, init: RequestInit) => { + firstSignal = init.signal as AbortSignal + await new Promise(resolve => setTimeout(resolve, config.upstreamTimeoutMs * 3)) + abortedWhenAnswered = firstSignal.aborted + return mockFetchResponse(200, JSON.stringify({ ok: true })) + }) + globalThis.fetch = fetchMock + + await request(makeApp()) + .post('/rpc/hosts/chatllm/approvals/approve') + .set('authorization', 'Bearer token') + .send({ toolCallId: 'tc-no-deadline' }) + .expect(200) + + expect(fetchMock).toHaveBeenCalledTimes(1) + expect(firstSignal).toBeInstanceOf(AbortSignal) + expect(abortedWhenAnswered).toBe(false) + expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() + }) + it('translates toolCallId to requestId in upstream body', async () => { let capturedBody: string | undefined globalThis.fetch = vi.fn().mockImplementation(async (_url: string, init: RequestInit) => { @@ -222,7 +252,10 @@ describe('POST /rpc/hosts/:hostRef/approvals/approve', () => { }) it('returns 403 when host is not accessible', async () => { - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) const app = makeApp() await request(app) @@ -341,7 +374,10 @@ describe('POST /rpc/hosts/:hostRef/approvals/deny', () => { }) it('returns 403 when host is not accessible', async () => { - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) const app = makeApp() await request(app) @@ -419,7 +455,10 @@ describe('GET /rpc/hosts/:hostRef/artifacts', () => { }) it('returns 403 when host is not accessible', async () => { - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) const app = makeApp() await request(app) @@ -490,7 +529,8 @@ describe('POST /rpc/hosts/:hostRef/tasks/:taskId/cancel', () => { expect(serviceMock.forwardCancelToHost).toHaveBeenCalledWith( expect.objectContaining({ name: 'chatllm' }), 'abc', - VALID_CLAIMS.sub + VALID_CLAIMS.sub, + expect.anything() ) }) @@ -547,7 +587,10 @@ describe('POST /rpc/hosts/:hostRef/tasks/:taskId/cancel', () => { ...VALID_CLAIMS, scopes: ['host:message:invoke'], }) - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) const app = makeApp() await request(app) @@ -614,6 +657,34 @@ describe('POST /rpc/hosts/:hostRef/tasks/:taskId/cancel', () => { }) describe('GET /rpc/hosts/:hostRef/artifacts/:filename/download', () => { + it('allows the bounded body transfer to outlast the upstream header timeout', async () => { + config.upstreamTimeoutMs = 5 + config.artifactDownloadTimeoutMs = 1_000 + let signal: AbortSignal | undefined + globalThis.fetch = vi.fn().mockImplementation((_url: string, init: RequestInit) => { + signal = init.signal as AbortSignal + return Promise.resolve({ + ok: true, + status: 200, + headers: new Headers({ 'content-type': 'application/octet-stream' }), + arrayBuffer: async () => { + await new Promise(resolve => setTimeout(resolve, 25)) + expect(signal?.aborted).toBe(false) + return new TextEncoder().encode('slow-artifact').buffer as ArrayBuffer + }, + } as unknown as Response) + }) + + const app = makeApp() + const response = await request(app) + .get('/rpc/hosts/chatllm/artifacts/slow.bin/download') + .set('authorization', 'Bearer token') + .expect(200) + + expect(Buffer.from(response.body).toString()).toBe('slow-artifact') + expect(signal?.aborted).toBe(false) + }) + it("rejects filename containing '..'", async () => { // Express normalizes "../" in paths, so use URL-encoded dots to test // the validation logic when the param actually contains ".." @@ -833,7 +904,10 @@ describe('GET /rpc/hosts/:hostRef/artifacts/:filename/download', () => { }) it('returns 403 when host is not accessible', async () => { - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) const app = makeApp() await request(app) diff --git a/rpc-proxy/src/__tests__/config.test.ts b/rpc-proxy/src/__tests__/config.test.ts index 6c28f4d0f5..b1dd95d29f 100644 --- a/rpc-proxy/src/__tests__/config.test.ts +++ b/rpc-proxy/src/__tests__/config.test.ts @@ -1,8 +1,13 @@ -import { describe, expect, it } from 'vitest' +import { afterEach, describe, expect, it, vi } from 'vitest' +import { readFileSync } from 'node:fs' +import { resolve } from 'node:path' import { + MAX_REQUEST_HOLD_MS, + MAX_TIMER_DELAY_MS, parseArtifactDownloadMaxBytes, parsePositiveIntMs, parseSandboxUiAllowedPorts, + parseWakeMaxHoldMs, } from '../config.js' describe('parseSandboxUiAllowedPorts', () => { @@ -73,3 +78,135 @@ describe('parsePositiveIntMs (wake-and-hold intervals)', () => { ) }) }) + +// NEW-rpx-2: setTimeout / AbortSignal.timeout treat a delay above 2^31-1 as 1 ms, +// so an oversized value would turn a generous deadline into an instant abort. +describe('parsePositiveIntMs timer bound', () => { + it('accepts the largest delay a timer can honour', () => { + expect(MAX_TIMER_DELAY_MS).toBe(2_147_483_647) + expect(parsePositiveIntMs('RPC_PROXY_UPSTREAM_TIMEOUT_MS', '2147483647')).toBe(2_147_483_647) + }) + + it('rejects a value one above the timer bound', () => { + expect(() => parsePositiveIntMs('RPC_PROXY_UPSTREAM_TIMEOUT_MS', '2147483648')).toThrow( + /RPC_PROXY_UPSTREAM_TIMEOUT_MS.*at most 2147483647/ + ) + }) +}) + +describe('parseWakeMaxHoldMs', () => { + afterEach(() => { + vi.restoreAllMocks() + }) + + it('honours a value within the request hold cap without warning', () => { + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + expect(parseWakeMaxHoldMs('30000')).toBe(30_000) + expect(parseWakeMaxHoldMs(String(MAX_REQUEST_HOLD_MS))).toBe(MAX_REQUEST_HOLD_MS) + expect(warn).not.toHaveBeenCalled() + }) + + it('clamps a larger value to the cap and says so loudly at config load', () => { + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + expect(parseWakeMaxHoldMs('90000')).toBe(MAX_REQUEST_HOLD_MS) + expect(warn).toHaveBeenCalledTimes(1) + expect(String(warn.mock.calls[0]![0])).toMatch( + /RPC_PROXY_WAKE_MAX_HOLD_MS=90000.*clamped to 48000/ + ) + }) + + it('still rejects an invalid value instead of clamping it', () => { + expect(() => parseWakeMaxHoldMs('0')).toThrow(/RPC_PROXY_WAKE_MAX_HOLD_MS.*positive integer/) + expect(() => parseWakeMaxHoldMs('2147483648')).toThrow(/at most 2147483647/) + }) +}) + +describe('config timeouts loaded from the environment', () => { + const ENV_KEYS = [ + 'RPC_PROXY_UPSTREAM_TIMEOUT_MS', + 'RPC_PROXY_ARTIFACT_DOWNLOAD_TIMEOUT_MS', + 'RPC_PROXY_WAKE_MAX_HOLD_MS', + ] as const + const saved = Object.fromEntries(ENV_KEYS.map(key => [key, process.env[key]])) + + afterEach(() => { + for (const key of ENV_KEYS) { + if (saved[key] === undefined) delete process.env[key] + else process.env[key] = saved[key] + } + vi.resetModules() + vi.restoreAllMocks() + }) + + async function loadConfig() { + vi.resetModules() + return (await import('../config.js')).config + } + + it('loads RPC_PROXY_UPSTREAM_TIMEOUT_MS as an integer', async () => { + process.env.RPC_PROXY_UPSTREAM_TIMEOUT_MS = '45000' + expect((await loadConfig()).upstreamTimeoutMs).toBe(45_000) + }) + + it.each(['abc', '0', '-1', '1.5', '2147483648'])( + 'fails loud on RPC_PROXY_UPSTREAM_TIMEOUT_MS=%s', + async value => { + process.env.RPC_PROXY_UPSTREAM_TIMEOUT_MS = value + await expect(loadConfig()).rejects.toThrow(/RPC_PROXY_UPSTREAM_TIMEOUT_MS/) + } + ) + + it('loads RPC_PROXY_ARTIFACT_DOWNLOAD_TIMEOUT_MS and rejects an oversized one', async () => { + process.env.RPC_PROXY_ARTIFACT_DOWNLOAD_TIMEOUT_MS = '300000' + expect((await loadConfig()).artifactDownloadTimeoutMs).toBe(300_000) + + process.env.RPC_PROXY_ARTIFACT_DOWNLOAD_TIMEOUT_MS = '2147483648' + await expect(loadConfig()).rejects.toThrow(/RPC_PROXY_ARTIFACT_DOWNLOAD_TIMEOUT_MS/) + }) + + it('clamps RPC_PROXY_WAKE_MAX_HOLD_MS above the cap in the loaded config', async () => { + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + process.env.RPC_PROXY_WAKE_MAX_HOLD_MS = '90000' + expect((await loadConfig()).wakeMaxHoldMs).toBe(MAX_REQUEST_HOLD_MS) + expect(warn).toHaveBeenCalledWith(expect.stringContaining('clamped to 48000')) + }) + + it('the built-in default is the hold cap itself, so it needs no clamp and no boot-time warning', async () => { + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined) + delete process.env.RPC_PROXY_WAKE_MAX_HOLD_MS + expect((await loadConfig()).wakeMaxHoldMs).toBe(48_000) + expect(MAX_REQUEST_HOLD_MS).toBe(48_000) + // The literal control-api's host-wake rate-limit guard reads must be the + // effective default (not a larger value that is clamped at load). + const source = readFileSync(new URL('../config.ts', import.meta.url), 'utf-8') + expect(source).toMatch(/RPC_PROXY_WAKE_MAX_HOLD_MS\s*\|\|\s*'48000'/) + expect(warn).not.toHaveBeenCalled() + }) +}) + +describe('minikube rpc-proxy ConfigMap', () => { + const overlay = readFileSync( + resolve(__dirname, '../../../deploy/overlays/minikube/configmaps/rpc-proxy-config.yaml'), + 'utf8' + ) + + function value(key: string): number { + const match = overlay.match(new RegExp(`^\\s+${key}:\\s+"(\\d+)"\\s*$`, 'm')) + if (!match) throw new Error(`${key} is missing from the minikube rpc-proxy ConfigMap`) + return Number(match[1]) + } + + it('sets an artifact download timeout sized for the 250 MB cap', () => { + const timeoutMs = value('RPC_PROXY_ARTIFACT_DOWNLOAD_TIMEOUT_MS') + const maxMb = value('RPC_PROXY_ARTIFACT_DOWNLOAD_MAX_MB') + expect(maxMb).toBe(250) + // The 60 s default needs 4.2 MB/s to move 250 MB; the overlay must not. + const requiredMbPerSecond = maxMb / (timeoutMs / 1000) + expect(requiredMbPerSecond).toBeLessThanOrEqual(1) + expect(timeoutMs).toBeGreaterThan(60_000) + // ...and it must load through the same parser as production. + expect(parsePositiveIntMs('RPC_PROXY_ARTIFACT_DOWNLOAD_TIMEOUT_MS', String(timeoutMs))).toBe( + timeoutMs + ) + }) +}) diff --git a/rpc-proxy/src/__tests__/errorHandler.test.ts b/rpc-proxy/src/__tests__/errorHandler.test.ts new file mode 100644 index 0000000000..ae923027b4 --- /dev/null +++ b/rpc-proxy/src/__tests__/errorHandler.test.ts @@ -0,0 +1,54 @@ +import { describe, expect, it } from 'vitest' +import express from 'express' +import request from 'supertest' +import { createApp } from '../app.js' +import { apiErrorHandler } from '../errorHandler.js' + +function appThrowing(error: unknown) { + const app = express() + app.get('/boom', (_req, _res, next) => next(error)) + app.use(apiErrorHandler) + return app +} + +function named(name: string): Error { + const error = new Error(name) + error.name = name + return error +} + +describe('apiErrorHandler', () => { + it.each(['AbortError', 'TimeoutError'])('maps %s to 504 Gateway Timeout', async name => { + const res = await request(appThrowing(named(name))).get('/boom') + expect(res.status).toBe(504) + expect(res.body).toEqual({ error: 'Gateway Timeout' }) + }) + + it('maps a DOMException abort (what fetch rejects with) to 504', async () => { + const res = await request(appThrowing(new DOMException('aborted', 'AbortError'))).get('/boom') + expect(res.status).toBe(504) + }) + + it('maps body-parser entity.too.large to 413', async () => { + const res = await request( + appThrowing(Object.assign(new Error('big'), { type: 'entity.too.large' })) + ).get('/boom') + expect(res.status).toBe(413) + expect(res.body).toEqual({ error: 'Payload Too Large' }) + }) + + it('maps anything else to 500 with the message', async () => { + const res = await request(appThrowing(new Error('kaput'))).get('/boom') + expect(res.status).toBe(500) + expect(res.body).toEqual({ error: 'Internal Server Error', message: 'kaput' }) + }) + + it('is the terminal handler of the real app (tests and production share it)', () => { + // Express 4 keeps its layer stack on `_router`; the public `router` getter + // throws a deprecation error. + const stack = (createApp() as unknown as { _router?: { stack: Array<{ handle: unknown }> } }) + ._router?.stack + if (!stack) throw new Error('express app exposes no _router stack; update this guard') + expect(stack.at(-1)!.handle).toBe(apiErrorHandler) + }) +}) diff --git a/rpc-proxy/src/__tests__/host-access-denial-code.route.test.ts b/rpc-proxy/src/__tests__/host-access-denial-code.route.test.ts new file mode 100644 index 0000000000..6d4a641050 --- /dev/null +++ b/rpc-proxy/src/__tests__/host-access-denial-code.route.test.ts @@ -0,0 +1,435 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import express from 'express' +import request from 'supertest' +import { apiErrorHandler } from '../errorHandler.js' +import { createRpcRouter } from '../routes/rpc.js' +import { createRpcHostActivityStreamRouter } from '../routes/rpcHostActivityStream.js' +import { createRpcHostProgressStreamRouter } from '../routes/rpcHostProgressStream.js' +import { createRpcHostStatusStreamRouter } from '../routes/rpcHostStatusStream.js' +import { hostAccessDenialCodeForReason } from '../services/hostAccessDenial.js' + +// PR #849 R1-M6 producer side (also R1-L7 on the rpc-proxy side). +// +// rpc-proxy's OWN Host-authorization 403 carries `code` +// (`host_access_revoked` | `host_access_denied`) at every site. Only the JWT +// verifier is mocked: the real routers, the real mcpProxyService and the real +// controlApiRestService run, and the only stubbed boundary is `fetch`. That +// makes the control-api reason -> code mapping observable end to end and lets a +// passthrough mcp-host 403 prove it is NOT rewritten. + +const authTokenMock = vi.hoisted(() => ({ verifyRpcToken: vi.fn() })) +vi.mock('../authToken.js', () => authTokenMock) + +const DENIED_ERROR = 'Forbidden: user cannot access this host' + +// control-api reports why it denied Host access in this response header; its 403 +// body is always exactly {"error":"Forbidden"}. Pinned literally so the test +// fails if rpc-proxy and control-api ever disagree on the name. +const REASON_HEADER = 'x-host-access-denial-reason' +const CONTROL_API_403_BODY = { error: 'Forbidden' } + +const CLAIMS = { + sub: 'user-uuid-123', + typ: 'user' as const, + accessScope: 'team' as const, + teamId: 'team-1', + scopes: [ + 'host:message:invoke', + 'host:wake:write', + 'host:approval:write', + 'host:session:read', + 'host:session:write', + 'host:model:write', + 'host:task:read', + 'host:activity:read', + 'host:status:read', + 'host:health:read', + ], + hostRefs: ['chatllm'], + jti: 'j1', + iat: 1, + exp: 9999999999, +} + +function makeApp() { + const app = express() + app.use(express.json()) + app.use(createRpcHostStatusStreamRouter()) + app.use(createRpcHostActivityStreamRouter()) + app.use(createRpcHostProgressStreamRouter()) + app.use(createRpcRouter()) + app.use(apiErrorHandler) + return app +} + +type ControlApiAnswer = { status: number; body?: unknown; headers?: Record } +const originalFetch = globalThis.fetch +let controlApiAnswer: ControlApiAnswer +let controlApiCalls: string[] +let hostCalls: string[] +let hostAnswer: { status: number; body: string } + +beforeEach(() => { + vi.clearAllMocks() + authTokenMock.verifyRpcToken.mockReturnValue({ ...CLAIMS }) + controlApiCalls = [] + hostCalls = [] + hostAnswer = { status: 200, body: '{}' } + controlApiAnswer = { + status: 403, + body: CONTROL_API_403_BODY, + headers: { [REASON_HEADER]: 'directory_grant_missing' }, + } + globalThis.fetch = vi.fn(async (input: string | URL | Request) => { + const url = String(input) + if (url.includes('/rpc/access/users/')) { + controlApiCalls.push(url) + return new Response( + controlApiAnswer.body === undefined ? null : JSON.stringify(controlApiAnswer.body), + { + status: controlApiAnswer.status, + headers: { 'content-type': 'application/json', ...controlApiAnswer.headers }, + } + ) + } + hostCalls.push(url) + return new Response(hostAnswer.body, { + status: hostAnswer.status, + headers: { 'content-type': 'application/json' }, + }) + }) as unknown as typeof fetch +}) + +afterEach(() => { + globalThis.fetch = originalFetch +}) + +type RouteCase = { + label: string + method: 'get' | 'post' | 'patch' + path: string + body?: Record +} + +// Every rpc-proxy route that resolves a Host and answers its own authorization +// 403: the 15 in rpc.ts plus the three streams. +const DENYING_ROUTES: RouteCase[] = [ + { + label: 'messages', + method: 'post', + path: '/rpc/hosts/chatllm/messages', + body: { content: 'hi' }, + }, + { label: 'wake', method: 'post', path: '/rpc/hosts/chatllm/wake' }, + { + label: 'approve', + method: 'post', + path: '/rpc/hosts/chatllm/approvals/approve', + body: { toolCallId: 'tc-1' }, + }, + { + label: 'deny', + method: 'post', + path: '/rpc/hosts/chatllm/approvals/deny', + body: { toolCallId: 'tc-1' }, + }, + { label: 'sessions', method: 'get', path: '/rpc/hosts/chatllm/sessions' }, + { + label: 'session messages', + method: 'get', + path: '/rpc/hosts/chatllm/sessions/agent-a/chat-1/messages', + }, + { + label: 'context breakdown', + method: 'get', + path: '/rpc/hosts/chatllm/sessions/agent-a/chat-1/context-breakdown', + }, + { + label: 'session rename', + method: 'patch', + path: '/rpc/hosts/chatllm/sessions/agent-a/chat-1/name', + body: { name: 'renamed' }, + }, + { label: 'models', method: 'get', path: '/rpc/hosts/chatllm/models' }, + { + label: 'set model', + method: 'post', + path: '/rpc/hosts/chatllm/model', + body: { chatId: 'c1', model: 'claude-haiku-4-5' }, + }, + { label: 'artifacts list', method: 'get', path: '/rpc/hosts/chatllm/artifacts' }, + { + label: 'artifact download', + method: 'get', + path: '/rpc/hosts/chatllm/artifacts/report.pdf/download', + }, + { label: 'activity', method: 'get', path: '/rpc/hosts/chatllm/activity' }, + { label: 'status', method: 'get', path: '/rpc/hosts/chatllm/status' }, + { label: 'health', method: 'get', path: '/rpc/hosts/chatllm/health' }, + { label: 'status stream', method: 'get', path: '/rpc/hosts/chatllm/status/stream' }, + { label: 'activity stream', method: 'get', path: '/rpc/hosts/chatllm/activity/stream' }, + { + label: 'progress stream', + method: 'get', + path: '/rpc/hosts/chatllm/tasks/task-1/progress/stream', + }, +] + +async function send(route: RouteCase) { + const pending = request(makeApp())[route.method](route.path).set('authorization', 'Bearer tok') + return route.body ? pending.send(route.body) : pending +} + +describe('host authorization 403 carries a machine-readable code at every site', () => { + it.each(DENYING_ROUTES)( + '$label: membership/grant revocation -> host_access_revoked', + async route => { + const res = await send(route) + + // Witness: the request reached control-api and no host was contacted. + expect(controlApiCalls).toHaveLength(1) + expect(hostCalls).toEqual([]) + expect(res.status).toBe(403) + expect(res.body).toEqual({ error: DENIED_ERROR, code: 'host_access_revoked' }) + } + ) + + it.each(DENYING_ROUTES)('$label: a missing host (404) -> host_access_denied', async route => { + controlApiAnswer = { status: 404, body: { error: 'not found' } } + + const res = await send(route) + + expect(controlApiCalls).toHaveLength(1) + expect(hostCalls).toEqual([]) + expect(res.status).toBe(403) + expect(res.body).toEqual({ error: DENIED_ERROR, code: 'host_access_denied' }) + }) +}) + +describe('R3-L8 hostAccessDenialCodeForReason', () => { + it('subject_mismatch is a denial; removed membership and grants are revocations', () => { + // Positive controls: the revoked set is live. + expect(hostAccessDenialCodeForReason('team_membership_missing')).toBe('host_access_revoked') + expect(hostAccessDenialCodeForReason('directory_grant_missing')).toBe('host_access_revoked') + expect(hostAccessDenialCodeForReason('subject_mismatch')).toBe('host_access_denied') + }) +}) + +describe('control-api 403 reason -> code mapping', () => { + const status = DENYING_ROUTES.find(route => route.label === 'status')! + + it.each([ + ['team_membership_missing', 'host_access_revoked'], + ['directory_grant_missing', 'host_access_revoked'], + // R3-L8: the route's userId is the token's own sub, so a mismatch is a + // request-shape error, not a removed grant. + ['subject_mismatch', 'host_access_denied'], + ['host_disabled', 'host_access_denied'], + ['host_missing', 'host_access_denied'], + ['host_claim_missing', 'host_access_denied'], + ['a_reason_added_later', 'host_access_denied'], + ])('reason %s -> %s', async (reason, code) => { + controlApiAnswer = { + status: 403, + body: CONTROL_API_403_BODY, + headers: { [REASON_HEADER]: reason }, + } + + const res = await send(status) + + expect(controlApiCalls).toHaveLength(1) + expect(res.status).toBe(403) + expect(res.body).toEqual({ error: DENIED_ERROR, code }) + }) + + it.each([ + ['the header is absent', {}], + ['the header is empty', { [REASON_HEADER]: '' }], + ])( + '%s -> host_access_denied (an unreadable reason never proves revocation)', + async (_l, headers) => { + controlApiAnswer = { status: 403, body: CONTROL_API_403_BODY, headers } + + const res = await send(status) + + expect(controlApiCalls).toHaveLength(1) + expect(res.body).toEqual({ error: DENIED_ERROR, code: 'host_access_denied' }) + } + ) + + // Only reasons that map to host_access_revoked as a header give this test + // power: a denied-anyway reason would pass even if the body were read. + it.each(['directory_grant_missing', 'team_membership_missing'])( + 'a legacy body-only reason %s (no header) is ignored -> host_access_denied', + async reason => { + // Proves the 403 body is never read: the same reason that maps to + // host_access_revoked when sent as the header must not count from the body. + controlApiAnswer = { status: 403, body: { error: 'Forbidden', reason }, headers: {} } + + const res = await send(status) + + expect(controlApiCalls).toHaveLength(1) + expect(res.status).toBe(403) + expect(res.body).toEqual({ error: DENIED_ERROR, code: 'host_access_denied' }) + } + ) +}) + +describe('passthrough and non-authorization responses are left untouched', () => { + beforeEach(() => { + controlApiAnswer = { + status: 200, + body: { + userId: CLAIMS.sub, + hostRef: 'chatllm', + url: 'http://chatllm.mcp-host.svc.cluster.local:8080', + }, + } + }) + + it('an mcp-host 403 body is relayed verbatim, without a `code`', async () => { + hostAnswer = { status: 403, body: JSON.stringify({ error: 'Forbidden by mcp-host policy' }) } + + const res = await send(DENYING_ROUTES.find(route => route.label === 'approve')!) + + // Witness: the host was contacted, so this 403 is the host's, not ours. + expect(controlApiCalls).toHaveLength(1) + expect(hostCalls).toHaveLength(1) + expect(res.status).toBe(403) + expect(JSON.parse(res.text)).toEqual({ error: 'Forbidden by mcp-host policy' }) + expect(res.text).not.toContain('"code"') + }) + + it('an mcp-host 403 on the artifact download is relayed verbatim, without a `code`', async () => { + hostAnswer = { status: 403, body: JSON.stringify({ error: 'artifact not readable' }) } + + const res = await send(DENYING_ROUTES.find(route => route.label === 'artifact download')!) + + expect(hostCalls).toHaveLength(1) + expect(res.status).toBe(403) + expect(JSON.parse(res.text)).toEqual({ error: 'artifact not readable' }) + }) + + it('an mcp-host 403 text body naming a reserved code is relayed verbatim', async () => { + hostAnswer = { status: 403, body: 'host_access_revoked' } + + const res = await send(DENYING_ROUTES.find(route => route.label === 'approve')!) + + expect(hostCalls).toHaveLength(1) + expect(res.status).toBe(403) + expect(res.text).toBe('host_access_revoked') + }) + + it('a non-403 mcp-host body is relayed verbatim even when it carries a reserved code', async () => { + const body = JSON.stringify({ ok: true, code: 'host_access_revoked' }) + hostAnswer = { status: 200, body } + + const res = await send(DENYING_ROUTES.find(route => route.label === 'approve')!) + + expect(hostCalls).toHaveLength(1) + expect(res.status).toBe(200) + expect(res.text).toBe(body) + }) + + it('control-api 401 is not a Host denial and gets no `code`', async () => { + controlApiAnswer = { status: 401, body: { error: 'token rejected' } } + authTokenMock.verifyRpcToken.mockReturnValue({ ...CLAIMS }) + + const res = await send(DENYING_ROUTES.find(route => route.label === 'status')!) + + expect(controlApiCalls).toHaveLength(1) + expect(res.body).not.toHaveProperty('code') + }) +}) + +// PR #849 ADV-SEC-1. Desktop reads the `code` of any 403 JSON body as +// rpc-proxy's own verdict, so an mcp-host 403 carrying a reserved code would +// forge a confirmed revocation (or denial). Every route that relays an mcp-host +// body must drop that field while keeping the status and the other fields. +describe('an mcp-host 403 never reaches the client with a reserved Host-access code', () => { + type PassthroughCase = RouteCase & { + /** The mcp-host path the route forwards to (witness the host was called). */ + hostPath: string + /** A route that answers rpc-proxy's own 403 (cancel answers its own denial as 404). */ + ownDenial?: RouteCase + } + + const byLabel = (label: string) => DENYING_ROUTES.find(route => route.label === label)! + + // Every site in rpc.ts that relays an mcp-host response body. + const PASSTHROUGH_ROUTES: PassthroughCase[] = [ + { ...byLabel('approve'), hostPath: '/v1/runtime/approvals/approve' }, + { ...byLabel('deny'), hostPath: '/v1/runtime/approvals/deny' }, + { ...byLabel('sessions'), hostPath: '/v1/runtime/sessions' }, + { + ...byLabel('session messages'), + hostPath: '/v1/runtime/sessions/agent-a/chat-1/messages', + }, + { + ...byLabel('context breakdown'), + hostPath: '/v1/runtime/sessions/agent-a/chat-1/context-breakdown', + }, + { ...byLabel('session rename'), hostPath: '/v1/runtime/sessions/agent-a/chat-1/name' }, + { ...byLabel('models'), hostPath: '/v1/runtime/models' }, + { ...byLabel('set model'), hostPath: '/v1/runtime/model' }, + { + label: 'task cancel', + method: 'post', + path: '/rpc/hosts/chatllm/tasks/task-1/cancel', + hostPath: '/v1/runtime/tasks/task-1/cancel', + ownDenial: byLabel('approve'), + }, + { ...byLabel('artifacts list'), hostPath: '/v1/runtime/artifacts' }, + { + ...byLabel('artifact download'), + hostPath: '/v1/runtime/artifacts/report.pdf/download', + }, + ] + + // control-api reasons that make rpc-proxy itself emit each reserved code. + const OWN_DENIAL_REASON = { + host_access_revoked: 'directory_grant_missing', + host_access_denied: 'host_disabled', + } as const + + const cases = PASSTHROUGH_ROUTES.flatMap(route => + (['host_access_revoked', 'host_access_denied'] as const).map(code => ({ ...route, code })) + ) + + it.each(cases)('$label: an upstream 403 with code=$code loses the code', async route => { + controlApiAnswer = { + status: 200, + body: { + userId: CLAIMS.sub, + hostRef: 'chatllm', + url: 'http://chatllm.mcp-host.svc.cluster.local:8080', + }, + } + hostAnswer = { + status: 403, + body: JSON.stringify({ error: 'forged by host', code: route.code, detail: 'kept' }), + } + + const forged = await send(route) + + // Witness: control-api authorized the call and the host answered this 403. + expect(controlApiCalls).toHaveLength(1) + expect(hostCalls).toHaveLength(1) + expect(new URL(hostCalls[0]!).pathname).toBe(route.hostPath) + expect(forged.status).toBe(403) + expect(JSON.parse(forged.text)).toEqual({ error: 'forged by host', detail: 'kept' }) + + // Witness: rpc-proxy's own denial still carries the same reserved code. + controlApiAnswer = { + status: 403, + body: CONTROL_API_403_BODY, + headers: { [REASON_HEADER]: OWN_DENIAL_REASON[route.code] }, + } + const own = await send(route.ownDenial ?? route) + + expect(controlApiCalls).toHaveLength(2) + expect(hostCalls).toHaveLength(1) + expect(own.status).toBe(403) + expect(own.body).toEqual({ error: DENIED_ERROR, code: route.code }) + }) +}) diff --git a/rpc-proxy/src/__tests__/host-access-denial.cross-service.test.ts b/rpc-proxy/src/__tests__/host-access-denial.cross-service.test.ts new file mode 100644 index 0000000000..095d0eac81 --- /dev/null +++ b/rpc-proxy/src/__tests__/host-access-denial.cross-service.test.ts @@ -0,0 +1,120 @@ +import { describe, expect, it } from 'vitest' +import { readFileSync } from 'node:fs' +import { hostAccessDenialCodeForReason } from '../services/hostAccessDenial.js' + +/** + * PR #849 R3-L9: cross-service guard for the Host-access denial contract. + * + * control-api names the denial reason (`RpcHostAccessDenialReason`) and sends it + * in a response header; rpc-proxy reads that header, maps the reason to a + * `HostAccessDenialCode`, and Desktop matches the code against its own + * constants. The three services share no module, so each hop is read here as + * source and compared. The rpc-proxy literals are also checked against the + * running mapping, so a parse that drifted from the code fails instead of + * comparing stale text. + */ + +function read(relativeFromThisFile: string): string { + return readFileSync(new URL(relativeFromThisFile, import.meta.url), 'utf-8') +} + +/** Fail-loud single-match extraction — a miss means the source moved. */ +function extractOne(source: string, pattern: RegExp, label: string): string { + const match = source.match(pattern) + if (!match || match[1] === undefined) { + throw new Error(`Could not extract ${label} with ${pattern} — re-derive the guard`) + } + return match[1] +} + +/** Every single-quoted literal of a union or array body; empty is a failure. */ +function quotedLiterals(body: string, label: string): string[] { + const literals = [...body.matchAll(/'([^']+)'/g)].map(match => match[1]!) + if (literals.length === 0) throw new Error(`No literals found in ${label}`) + return literals +} + +const controlApiAuthorizer = read( + '../../../control-api/src/services/access/rpcHostAccessAuthorizer.ts' +) +const controlApiRoute = read('../../../control-api/src/routes/rpc-access/users.ts') +const rpcProxyRestService = read('../services/controlApiRestService.ts') +const rpcProxyDenial = read('../services/hostAccessDenial.ts') +const desktopUpstreamErrors = read('../../../desktop-app/src/upstreamErrors.ts') + +const controlApiReasons = quotedLiterals( + extractOne( + controlApiAuthorizer, + /export type RpcHostAccessDenialReason =([\s\S]*?)\n\n/, + 'control-api RpcHostAccessDenialReason' + ), + 'control-api RpcHostAccessDenialReason' +) +const revokedReasons = quotedLiterals( + extractOne( + rpcProxyDenial, + /const REVOKED_CONTROL_API_REASONS: ReadonlySet = new Set\(\[([\s\S]*?)\]\)/, + 'rpc-proxy REVOKED_CONTROL_API_REASONS' + ), + 'rpc-proxy REVOKED_CONTROL_API_REASONS' +) +const rpcProxyCodes = quotedLiterals( + extractOne( + rpcProxyDenial, + /export type HostAccessDenialCode = ([^\n]+)/, + 'rpc-proxy HostAccessDenialCode' + ), + 'rpc-proxy HostAccessDenialCode' +) +const desktopRevokedCode = extractOne( + desktopUpstreamErrors, + /export const HOST_ACCESS_REVOKED_CODE = '([^']+)'/, + 'desktop HOST_ACCESS_REVOKED_CODE' +) +const desktopDeniedCode = extractOne( + desktopUpstreamErrors, + /export const HOST_ACCESS_DENIED_CODE = '([^']+)'/, + 'desktop HOST_ACCESS_DENIED_CODE' +) + +describe('R3-L9 Host-access denial contract across control-api, rpc-proxy and Desktop', () => { + it('rpc-proxy reads the denial-reason header under the name control-api sends it', () => { + const sent = extractOne( + controlApiRoute, + /export const HOST_ACCESS_DENIAL_REASON_HEADER = '([^']+)'/, + 'control-api HOST_ACCESS_DENIAL_REASON_HEADER' + ) + const received = extractOne( + rpcProxyRestService, + /const reason = response\.headers\.get\('([^']+)'\)/, + 'rpc-proxy denial-reason header read' + ) + + expect(received).toBe(sent) + }) + + it('every revoking reason in rpc-proxy is a reason control-api can send', () => { + // Witness: the parsed set is the running set. + for (const reason of revokedReasons) { + expect(hostAccessDenialCodeForReason(reason)).toBe('host_access_revoked') + } + for (const reason of revokedReasons) { + expect(controlApiReasons).toContain(reason) + } + // And every other control-api reason is a plain denial. + const deniedReasons = controlApiReasons.filter(r => !revokedReasons.includes(r)) + expect(deniedReasons.length).toBeGreaterThan(0) + for (const reason of deniedReasons) { + expect(hostAccessDenialCodeForReason(reason)).toBe('host_access_denied') + } + }) + + it('Desktop matches exactly the codes rpc-proxy emits', () => { + expect([...rpcProxyCodes].sort()).toEqual([desktopDeniedCode, desktopRevokedCode].sort()) + // The running mapping emits Desktop's literals for both outcomes. + for (const reason of revokedReasons) { + expect(hostAccessDenialCodeForReason(reason)).toBe(desktopRevokedCode) + } + expect(hostAccessDenialCodeForReason('a_reason_no_service_sends')).toBe(desktopDeniedCode) + }) +}) diff --git a/rpc-proxy/src/__tests__/host-wake-route.test.ts b/rpc-proxy/src/__tests__/host-wake-route.test.ts index 8527cf66da..b12a9ea26e 100644 --- a/rpc-proxy/src/__tests__/host-wake-route.test.ts +++ b/rpc-proxy/src/__tests__/host-wake-route.test.ts @@ -109,12 +109,19 @@ beforeEach(() => { describe('POST /rpc/hosts/:hostRef/wake auth', () => { it('rejects a token that cannot access :hostRef exactly like the messages route (403) and never calls the wake plane', async () => { // Same enforcement point as the messages route: control-api refuses the - // host resolution for this user/token → resolveHostConnectionForUser null. - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + // host resolution for this user/token → resolveHostConnectionForUser + // returns the typed denial. + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) const response = await postWake(makeApp()).expect(403) - expect(response.body).toEqual({ error: 'Forbidden: user cannot access this host' }) + expect(response.body).toEqual({ + error: 'Forbidden: user cannot access this host', + code: 'host_access_denied', + }) expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() }) diff --git a/rpc-proxy/src/__tests__/model-selection.test.ts b/rpc-proxy/src/__tests__/model-selection.test.ts index 52886365f0..d206728474 100644 --- a/rpc-proxy/src/__tests__/model-selection.test.ts +++ b/rpc-proxy/src/__tests__/model-selection.test.ts @@ -135,7 +135,10 @@ describe('GET /rpc/hosts/:hostRef/models — passthrough to mcp-host', () => { }) it('returns 403 if the user cannot access the host', async () => { - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) await request(makeApp()) .get('/rpc/hosts/other-host/models') .set('authorization', 'Bearer user-token') @@ -241,7 +244,10 @@ describe('POST /rpc/hosts/:hostRef/model — set per-session model', () => { }) it('returns 403 if the user cannot access the host', async () => { - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) await request(makeApp()) .post('/rpc/hosts/other-host/model') .set('authorization', 'Bearer user-token') diff --git a/rpc-proxy/src/__tests__/route-upstream-deadlines.real-http.test.ts b/rpc-proxy/src/__tests__/route-upstream-deadlines.real-http.test.ts new file mode 100644 index 0000000000..b0c08cfa9b --- /dev/null +++ b/rpc-proxy/src/__tests__/route-upstream-deadlines.real-http.test.ts @@ -0,0 +1,931 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import express from 'express' +import { + type IncomingMessage, + type Server, + type ServerResponse, + createServer, + request as httpRequest, +} from 'node:http' +import type { AddressInfo } from 'node:net' +import request from 'supertest' +import { config } from '../config.js' +import { apiErrorHandler } from '../errorHandler.js' +import { createRpcRouter } from '../routes/rpc.js' +import { + forwardCancelToHost, + forwardHostMessageToHost, + forwardTaskResultFromHost, +} from '../services/mcpHostRestService.js' + +// PR #849 R2-M3 / R1-M12 / R1-M13 / NEW-rpx-1 / NEW-sec-4. +// +// Every test here talks to a REAL http server through the real `fetch`, so the +// timers under test are the production timers, not a mocked fetch that reacts +// to an AbortSignal. Each upstream handler records the requests it saw: that is +// the liveness witness proving the request reached the upstream before the +// asserted outcome, so a 504 cannot be explained by "the request never left". + +const authTokenMock = vi.hoisted(() => ({ verifyRpcToken: vi.fn() })) +const serviceMock = vi.hoisted(() => ({ + listAllowedServersForUser: vi.fn(), + resolveServerConnectionForUser: vi.fn(), + resolveHostConnectionForUser: vi.fn(), + validateRpcRequest: vi.fn(), + forwardRpcToServer: vi.fn(), + forwardHostMessageToHost: vi.fn(), + forwardHostActivity: vi.fn(), + forwardHostStatus: vi.fn(), + forwardHostHealth: vi.fn(), + forwardTaskResultFromHost: vi.fn(), + forwardCancelToHost: vi.fn(), + UpstreamHostError: class UpstreamHostError extends Error { + constructor( + public readonly status: number, + public readonly bodySnippet: string + ) { + super(`Upstream host returned ${status}: ${bodySnippet}`) + this.name = 'UpstreamHostError' + } + }, +})) +const controlApiMock = vi.hoisted(() => ({ + fetchUserAllowedServersFromControlApi: vi.fn(), + fetchHostConnectionFromControlApi: vi.fn(), + requestHostWakeFromControlApi: vi.fn(), +})) + +vi.mock('../authToken.js', () => authTokenMock) +vi.mock('../services/mcpProxyService.js', () => serviceMock) +vi.mock('../services/controlApiRestService.js', () => controlApiMock) + +const ALL_SCOPES = [ + 'host:approval:write', + 'host:model:write', + 'host:session:read', + 'host:session:write', + 'host:task:read', + 'host:message:invoke', + 'host:wake:write', +] + +function claims() { + return { + sub: 'user-uuid-123', + typ: 'user' as const, + accessScope: 'team' as const, + teamId: 'team-1', + scopes: ALL_SCOPES, + hostRefs: ['chatllm'], + jti: 'j1', + iat: 1, + exp: 9999999999, + } +} + +function makeApp() { + const app = express() + app.use(express.json()) + app.use(createRpcRouter()) + app.use(apiErrorHandler) + return app +} + +type SeenRequest = { method: string; url: string; closedEarly: boolean } +type Handler = (req: IncomingMessage, res: ServerResponse, index: number) => void + +let server: Server | undefined +let seen: SeenRequest[] = [] + +async function startUpstream(handler: Handler): Promise { + seen = [] + server = createServer((req, res) => { + const entry: SeenRequest = { method: req.method ?? '', url: req.url ?? '', closedEarly: false } + seen.push(entry) + res.once('close', () => { + if (!res.writableFinished) entry.closedEarly = true + }) + handler(req, res, seen.length - 1) + }) + await new Promise((resolve, reject) => { + server!.once('error', reject) + server!.listen(0, '127.0.0.1', resolve) + }) + const { port } = server.address() as AddressInfo + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + name: 'chatllm', + url: `http://127.0.0.1:${port}`, + headers: {}, + }) +} + +/** Ends the TCP connection without a response: undici reports it as `fetch failed`. */ +function dropConnection(req: IncomingMessage): void { + req.socket.destroy() +} + +/** + * Sends the 200 headers and part of the body, then kills the socket: undici + * resolves `fetch()` and the body read fails with `terminated` (UND_ERR_SOCKET). + */ +function sendHeadersThenDestroy(req: IncomingMessage, res: ServerResponse): void { + res.writeHead(200, { 'content-type': 'application/json' }) + res.write('{"applied":', () => { + setTimeout(() => req.socket.destroy(), 20) + }) +} + +// The symbol undici's setGlobalDispatcher writes; Node's fetch reads it per call. +const UNDICI_GLOBAL_DISPATCHER = Symbol.for('undici.globalDispatcher.1') +type UndiciDispatcher = { close(): Promise } +type UndiciAgentClass = new (options: { + headersTimeout: number + bodyTimeout: number +}) => UndiciDispatcher + +/** + * Runs `fn` with Node's own undici Agent swapped for one whose header/body + * timers are 200 ms instead of 300 s, so undici's real UND_ERR_HEADERS_TIMEOUT / + * UND_ERR_BODY_TIMEOUT errors are produced by the production fetch calls. + */ +async function withShortUndiciTimeouts(fn: () => Promise): Promise { + const globals = globalThis as unknown as Record + if (!globals[UNDICI_GLOBAL_DISPATCHER]) { + // Node creates its global Agent on the first fetch: make one to a + // throwaway server so the upstream under test sees no extra request. + const warmup = createServer((_req, res) => res.end()) + await new Promise(resolve => warmup.listen(0, '127.0.0.1', resolve)) + const { port } = warmup.address() as AddressInfo + await (await fetch(`http://127.0.0.1:${port}/`)).text() + await new Promise(resolve => warmup.close(() => resolve())) + } + const original = globals[UNDICI_GLOBAL_DISPATCHER] + if (!original) throw new Error('Node fetch did not install an undici global dispatcher') + const Agent = original.constructor as UndiciAgentClass + const shortAgent = new Agent({ headersTimeout: 200, bodyTimeout: 200 }) + globals[UNDICI_GLOBAL_DISPATCHER] = shortAgent + try { + return await fn() + } finally { + globals[UNDICI_GLOBAL_DISPATCHER] = original + await shortAgent.close() + } +} + +const original = { + upstreamTimeoutMs: config.upstreamTimeoutMs, + artifactDownloadTimeoutMs: config.artifactDownloadTimeoutMs, + wakeMaxHoldMs: config.wakeMaxHoldMs, +} + +beforeEach(() => { + vi.clearAllMocks() + authTokenMock.verifyRpcToken.mockReturnValue(claims()) + controlApiMock.requestHostWakeFromControlApi.mockResolvedValue({ + kind: 'active', + wakeGeneration: null, + }) + serviceMock.forwardHostHealth.mockResolvedValue({ ok: true }) + serviceMock.forwardHostMessageToHost.mockImplementation( + forwardHostMessageToHost as typeof serviceMock.forwardHostMessageToHost + ) +}) + +afterEach(async () => { + config.upstreamTimeoutMs = original.upstreamTimeoutMs + config.artifactDownloadTimeoutMs = original.artifactDownloadTimeoutMs + config.wakeMaxHoldMs = original.wakeMaxHoldMs + if (server) { + server.closeAllConnections() + await new Promise(resolve => server!.close(() => resolve())) + server = undefined + } +}) + +const MUTATING_ROUTES = [ + { + label: 'approve', + path: '/rpc/hosts/chatllm/approvals/approve', + body: { toolCallId: 'tc-1' }, + upstreamPath: '/v1/runtime/approvals/approve', + }, + { + label: 'deny', + path: '/rpc/hosts/chatllm/approvals/deny', + body: { toolCallId: 'tc-1' }, + upstreamPath: '/v1/runtime/approvals/deny', + }, + { + label: 'set model', + path: '/rpc/hosts/chatllm/model', + body: { chatId: 'c1', model: 'claude-haiku-4-5' }, + upstreamPath: '/v1/runtime/model', + }, +] as const + +function post(route: (typeof MUTATING_ROUTES)[number]) { + return request(makeApp()).post(route.path).set('authorization', 'Bearer tok').send(route.body) +} + +describe.each(MUTATING_ROUTES)('R2-M3 / R1-M12 $label route deadlines', route => { + it('waits for a slow upstream past upstreamTimeoutMs and relays its real answer', async () => { + config.upstreamTimeoutMs = 60 + await startUpstream((_req, res) => { + setTimeout(() => { + res.writeHead(200, { 'content-type': 'application/json' }) + res.end(JSON.stringify({ applied: true })) + }, 400) + }) + + const startedAt = Date.now() + const res = await post(route).expect(200) + + // Witness: the upstream received exactly this one request, was still working + // on it well after the 60 ms deadline, and completed it (not closed early). + expect(seen).toEqual([{ method: 'POST', url: route.upstreamPath, closedEarly: false }]) + expect(Date.now() - startedAt).toBeGreaterThanOrEqual(350) + expect(JSON.parse(res.text)).toEqual({ applied: true }) + expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() + }) + + it('a wake retry that hangs ends in 504 by its own bounded timeout', async () => { + config.upstreamTimeoutMs = 150 + await startUpstream((req, _res, index) => { + if (index === 0) dropConnection(req) // first attempt: host down + // second attempt (the wake retry): never answered + }) + + const startedAt = Date.now() + const res = await post(route).expect(504) + + expect(res.body).toEqual({ error: 'Gateway Timeout' }) + // Witness: two requests reached the upstream. The second is the retry; the + // proxy aborted it (closedEarly) at ~upstreamTimeoutMs, not at some later + // point, so the 504 comes from the retry deadline. + expect(seen.map(entry => entry.url)).toEqual([route.upstreamPath, route.upstreamPath]) + await vi.waitFor(() => expect(seen[1]!.closedEarly).toBe(true)) + expect(Date.now() - startedAt).toBeLessThan(2_000) + expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalledTimes(1) + }) + + it('a wake retry is bounded by the remaining hold budget, not by upstreamTimeoutMs', async () => { + config.upstreamTimeoutMs = 30_000 + config.wakeMaxHoldMs = 1_300 + await startUpstream((req, _res, index) => { + if (index === 0) dropConnection(req) + }) + + const startedAt = Date.now() + await post(route).expect(504) + const elapsed = Date.now() - startedAt + + expect(seen).toHaveLength(2) + await vi.waitFor(() => expect(seen[1]!.closedEarly).toBe(true)) + // The hold budget (1.3 s) ended it, far below the 30 s upstream timeout. + expect(elapsed).toBeGreaterThanOrEqual(1_000) + expect(elapsed).toBeLessThan(5_000) + }) + + // R1-M12: no idempotency key, so a duplicated POST is a duplicated side + // effect. A host that keeps dropping connections must see a small, fixed + // number of POSTs however long the hold window is. + it('re-issues the POST at most once while the host keeps failing (4 s hold)', async () => { + config.upstreamTimeoutMs = 2_000 + config.wakeMaxHoldMs = 4_000 + await startUpstream(req => dropConnection(req)) + + const startedAt = Date.now() + const res = await post(route).expect(503) + + expect(res.body).toMatchObject({ code: 'host_waking', hostRef: 'chatllm' }) + // Witness: the host was reached (POSTs observed) and a wake was requested. + expect(seen.length).toBeGreaterThanOrEqual(1) + expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalled() + expect(seen).toHaveLength(2) + // It answered promptly instead of looping through the whole 4 s window. + expect(Date.now() - startedAt).toBeLessThan(3_000) + }) + + // NEW-fa-2: a retry with less than MIN_ADMISSION_RETRY_TIMEOUT_MS (1 s) of hold + // budget left would time out after the upstream applied it. The 500 ms hold + // budget leaves ~490 ms once the first attempt has failed: inside the + // 100-999 ms band a lowered floor would send a second POST. + it('sends no wake retry when less than the 1 s admission floor of the hold budget remains', async () => { + config.upstreamTimeoutMs = 30_000 + config.wakeMaxHoldMs = 500 + await startUpstream(req => dropConnection(req)) + + const res = await post(route).expect(503) + + expect(res.body).toMatchObject({ code: 'host_waking', hostRef: 'chatllm' }) + // Witness: the first attempt reached the upstream and the wake was requested, + // so the single POST is the first attempt and the retry was withheld. + expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalledTimes(1) + expect(seen.map(entry => `${entry.method} ${entry.url}`)).toEqual([ + `POST ${route.upstreamPath}`, + ]) + }) + + // ADV-HCC-1 / ADV-SEC-3: once the response headers arrived the host has the + // POST (it may already have applied it), so a socket that dies mid-body is a + // 502 for this one request, never a down host to wake and re-POST to. + it('a socket that dies after the response headers answers 502 with exactly one POST', async () => { + await startUpstream(sendHeadersThenDestroy) + + const res = await post(route).expect(502) + + expect(res.body).toEqual({ error: 'Upstream host unavailable' }) + // Witness: the first attempt reached the upstream and got its headers out. + expect(seen.map(entry => `${entry.method} ${entry.url}`)).toEqual([ + `POST ${route.upstreamPath}`, + ]) + expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() + }) + + it('a wake retry whose body dies after the headers answers 502 without a third POST', async () => { + config.wakeMaxHoldMs = 4_000 + await startUpstream((req, res, index) => { + // First attempt: dropped before the headers (host down, wake). Retry: + // headers, then the socket dies mid-body. + if (index === 0) dropConnection(req) + else sendHeadersThenDestroy(req, res) + }) + + const res = await post(route).expect(502) + + expect(res.body).toEqual({ error: 'Upstream host unavailable' }) + // Witness: the pre-headers failure did wake and re-issue once. + expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalledTimes(1) + expect(seen.map(entry => entry.url)).toEqual([route.upstreamPath, route.upstreamPath]) + }) + + // undici's own header timer fires on a connection the host accepted: a slow + // host, not a down one. It must answer 504, not wake and re-issue the POST. + it("undici's headers timeout answers 504 with exactly one POST", async () => { + await startUpstream(() => { + // never answers: the upstream holds the POST + }) + + const res = await withShortUndiciTimeouts(() => post(route).expect(504)) + + expect(res.body).toEqual({ error: 'Gateway Timeout' }) + expect(seen.map(entry => `${entry.method} ${entry.url}`)).toEqual([ + `POST ${route.upstreamPath}`, + ]) + expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() + }) + + it("undici's body timeout after the headers answers 504 with exactly one POST", async () => { + await startUpstream((_req, res) => { + res.writeHead(200, { 'content-type': 'application/json' }) + res.write('{"applied":') // then stalls + }) + + const res = await withShortUndiciTimeouts(() => post(route).expect(504)) + + expect(res.body).toEqual({ error: 'Gateway Timeout' }) + expect(seen.map(entry => `${entry.method} ${entry.url}`)).toEqual([ + `POST ${route.upstreamPath}`, + ]) + expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() + }) +}) + +describe('ADV-HCC-1 task cancel is not re-issued after the response headers', () => { + it('a socket that dies after the response headers answers 502 with exactly one POST', async () => { + serviceMock.forwardCancelToHost.mockImplementation( + forwardCancelToHost as typeof serviceMock.forwardCancelToHost + ) + await startUpstream(sendHeadersThenDestroy) + + const res = await request(makeApp()) + .post('/rpc/hosts/chatllm/tasks/task-1/cancel') + .set('authorization', 'Bearer tok') + .expect(502) + + expect(res.body).toEqual({ error: 'Upstream host unavailable' }) + // Witness: the real forwardCancelToHost reached the upstream once. + expect(serviceMock.forwardCancelToHost).toHaveBeenCalledTimes(1) + expect(seen.map(entry => `${entry.method} ${entry.url}`)).toEqual([ + 'POST /v1/runtime/tasks/task-1/cancel', + ]) + expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() + }) +}) + +describe('R1-M12 /messages keeps the retry-until-deadline loop (it carries a messageId)', () => { + it('re-issues the POST until the host admits the request', async () => { + config.upstreamTimeoutMs = 2_000 + config.wakeMaxHoldMs = 4_000 + await startUpstream((req, res, index) => { + if (index < 3) { + dropConnection(req) + return + } + res.writeHead(200, { 'content-type': 'application/json' }) + res.end(JSON.stringify({ success: true, taskId: 't-1' })) + }) + + await request(makeApp()) + .post('/rpc/hosts/chatllm/messages') + .set('authorization', 'Bearer tok') + .send({ content: 'hello' }) + .expect(200) + + // Three dropped connections then an accepted one: more than the single + // re-issue every other route is limited to. + expect(seen).toHaveLength(4) + expect(seen.every(entry => entry.url.startsWith('/v1/runtime/'))).toBe(true) + }) + + // NEW-fa-2: even the retry-until-deadline route stops re-issuing once less than + // the 1 s admission floor of the hold budget remains (100-999 ms band). + it('sends no retry when less than the 1 s admission floor of the hold budget remains', async () => { + config.upstreamTimeoutMs = 30_000 + config.wakeMaxHoldMs = 500 + await startUpstream(req => dropConnection(req)) + + const res = await request(makeApp()) + .post('/rpc/hosts/chatllm/messages') + .set('authorization', 'Bearer tok') + .send({ content: 'hello' }) + .expect(503) + + expect(res.body).toMatchObject({ code: 'host_waking', hostRef: 'chatllm' }) + // Witness: the first attempt reached the upstream and the wake was requested. + expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalledTimes(1) + expect(seen).toHaveLength(1) + expect(seen[0]!.url.startsWith('/v1/runtime/')).toBe(true) + }) +}) + +describe('artifact download deadlines (NEW-rpx-1)', () => { + const DOWNLOAD_PATH = '/rpc/hosts/chatllm/artifacts/report.bin/download' + const UPSTREAM_DOWNLOAD_PATH = '/v1/runtime/artifacts/report.bin/download' + + function download() { + return request(makeApp()).get(DOWNLOAD_PATH).set('authorization', 'Bearer tok') + } + + it('slow response headers are cut off by upstreamTimeoutMs and answer 504', async () => { + config.upstreamTimeoutMs = 30 + config.artifactDownloadTimeoutMs = 60_000 + await startUpstream((_req, res) => { + // Headers would arrive after 1.5 s; the client must give up long before. + setTimeout(() => { + if (res.destroyed) return + res.writeHead(200, { 'content-type': 'application/octet-stream' }) + res.end('late') + }, 1_500) + }) + + const startedAt = Date.now() + const res = await download().expect(504) + + expect(res.body).toEqual({ error: 'Gateway Timeout' }) + expect(seen.map(entry => entry.url)).toEqual([UPSTREAM_DOWNLOAD_PATH]) + await vi.waitFor(() => expect(seen[0]!.closedEarly).toBe(true)) + expect(Date.now() - startedAt).toBeLessThan(1_200) + }) + + it('a body that stalls after the headers is cut off by artifactDownloadTimeoutMs and answers 504', async () => { + config.upstreamTimeoutMs = 10_000 + config.artifactDownloadTimeoutMs = 50 + let headersFlushed = false + await startUpstream((_req, res) => { + res.writeHead(200, { 'content-type': 'application/octet-stream', 'content-length': '1000' }) + res.write('first-bytes', () => { + headersFlushed = true + }) + // ...and then nothing: the body never completes. + }) + + const startedAt = Date.now() + const res = await download().expect(504) + + expect(res.body).toEqual({ error: 'Gateway Timeout' }) + // Witness: the upstream had already sent its headers and first bytes, so the + // 504 is the body-phase timer, not the 10 s header timer. + expect(seen.map(entry => entry.url)).toEqual([UPSTREAM_DOWNLOAD_PATH]) + expect(headersFlushed).toBe(true) + await vi.waitFor(() => expect(seen[0]!.closedEarly).toBe(true)) + expect(Date.now() - startedAt).toBeLessThan(3_000) + }) + + // NEW-sec-4: on a wake retry the body phase gets min(artifactDownloadTimeoutMs, + // remaining hold budget), so a stalled retry cannot outlive the request. + it('a stalled body on the wake retry is bounded by the remaining hold budget', async () => { + config.upstreamTimeoutMs = 10_000 + config.artifactDownloadTimeoutMs = 60_000 + config.wakeMaxHoldMs = 1_400 + await startUpstream((req, res, index) => { + if (index === 0) { + dropConnection(req) + return + } + res.writeHead(200, { 'content-type': 'application/octet-stream', 'content-length': '1000' }) + res.write('first-bytes') + }) + + const startedAt = Date.now() + await download().expect(504) + const elapsed = Date.now() - startedAt + + // Witness: the wake happened and the retry reached the upstream, which began + // to answer (headers + first bytes) before stalling. + expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalledTimes(1) + expect(seen).toHaveLength(2) + await vi.waitFor(() => expect(seen[1]!.closedEarly).toBe(true)) + // 1.4 s hold budget, not the 60 s download timeout. + expect(elapsed).toBeLessThan(5_000) + }) + + // NEW-tq-3(c): both download timers are released once the response is + // committed, so a completed download leaves no armed timer behind. Sentinel + // delays (7777 header, 8888 body) identify the route's own timers among the + // ones express/undici/supertest create. + it('clears both the header timer and the body timer after a successful download', async () => { + config.upstreamTimeoutMs = 7_777 + config.artifactDownloadTimeoutMs = 8_888 + const realSetTimeout = globalThis.setTimeout + const realClearTimeout = globalThis.clearTimeout + const armed = new Map() + const cleared = new Set() + const setSpy = vi.spyOn(globalThis, 'setTimeout').mockImplementation((( + handler: () => void, + delay?: number, + ...args: unknown[] + ) => { + const timer = realSetTimeout(handler, delay, ...args) + if (delay === 7_777 || delay === 8_888) armed.set(timer, delay) + return timer + }) as typeof setTimeout) + const clearSpy = vi.spyOn(globalThis, 'clearTimeout').mockImplementation((( + timer?: Parameters[0] + ) => { + cleared.add(timer) + return realClearTimeout(timer) + }) as typeof clearTimeout) + await startUpstream((_req, res) => { + res.writeHead(200, { 'content-type': 'application/octet-stream' }) + res.end('artifact-bytes') + }) + + try { + await download().expect(200) + } finally { + setSpy.mockRestore() + clearSpy.mockRestore() + } + + // Witness: the route armed the header timer AND re-armed a body timer. + expect([...armed.values()].sort()).toEqual([7_777, 8_888]) + for (const timer of armed.keys()) expect(cleared.has(timer)).toBe(true) + }) + + it('a healthy download still completes (control for the deadline tests)', async () => { + config.upstreamTimeoutMs = 2_000 + config.artifactDownloadTimeoutMs = 2_000 + await startUpstream((_req, res) => { + res.writeHead(200, { 'content-type': 'application/octet-stream' }) + res.end('artifact-bytes') + }) + + const res = await download().expect(200) + + expect(Buffer.from(res.body).toString()).toBe('artifact-bytes') + expect(seen).toHaveLength(1) + }) +}) + +describe('artifacts list keeps a finite first-attempt deadline', () => { + it('returns 504 when the upstream never answers (read-only, small JSON)', async () => { + config.upstreamTimeoutMs = 60 + await startUpstream(() => { + // never answered + }) + + const res = await request(makeApp()) + .get('/rpc/hosts/chatllm/artifacts') + .set('authorization', 'Bearer tok') + .expect(504) + + expect(res.body).toEqual({ error: 'Gateway Timeout' }) + expect(seen.map(entry => entry.url)).toEqual(['/v1/runtime/artifacts']) + await vi.waitFor(() => expect(seen[0]!.closedEarly).toBe(true)) + expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() + }) +}) + +// R3-L7: the hold deadline is pinned when the request arrives, before the Host +// connection is resolved, so a slow resolution is charged to the same hold +// budget instead of extending it. The wake retry never answers, so the request +// ends when the hold deadline aborts it. +const HELD_ROUTES = [ + { + label: 'send message', + method: 'post', + path: '/rpc/hosts/chatllm/messages', + body: { content: 'hello' }, + upstreamPath: '/v1/runtime/messages', + }, + { + label: 'approve', + method: 'post', + path: '/rpc/hosts/chatllm/approvals/approve', + body: { toolCallId: 'tc-1' }, + upstreamPath: '/v1/runtime/approvals/approve', + }, + { + label: 'deny', + method: 'post', + path: '/rpc/hosts/chatllm/approvals/deny', + body: { toolCallId: 'tc-1' }, + upstreamPath: '/v1/runtime/approvals/deny', + }, + { + label: 'set model', + method: 'post', + path: '/rpc/hosts/chatllm/model', + body: { chatId: 'c1', model: 'claude-haiku-4-5' }, + upstreamPath: '/v1/runtime/model', + }, + { + label: 'cancel task', + method: 'post', + path: '/rpc/hosts/chatllm/tasks/task-1/cancel', + body: {}, + upstreamPath: '/v1/runtime/tasks/task-1/cancel', + }, + { + label: 'list sessions', + method: 'get', + path: '/rpc/hosts/chatllm/sessions', + upstreamPath: '/v1/runtime/sessions', + }, + { + label: 'session transcript', + method: 'get', + path: '/rpc/hosts/chatllm/sessions/agent-1/chat-1/messages', + upstreamPath: '/v1/runtime/sessions/agent-1/chat-1/messages', + }, + { + label: 'context breakdown', + method: 'get', + path: '/rpc/hosts/chatllm/sessions/agent-1/chat-1/context-breakdown', + upstreamPath: '/v1/runtime/sessions/agent-1/chat-1/context-breakdown', + }, + { + label: 'list models', + method: 'get', + path: '/rpc/hosts/chatllm/models', + upstreamPath: '/v1/runtime/models', + }, + { + label: 'task result', + method: 'get', + path: '/rpc/hosts/chatllm/tasks/task-1/result', + upstreamPath: '/v1/runtime/tasks/task-1/result', + }, + { + label: 'list artifacts', + method: 'get', + path: '/rpc/hosts/chatllm/artifacts', + upstreamPath: '/v1/runtime/artifacts', + }, + { + label: 'artifact download', + method: 'get', + path: '/rpc/hosts/chatllm/artifacts/report.bin/download', + upstreamPath: '/v1/runtime/artifacts/report.bin/download', + }, +] as const + +const HOST_RESOLUTION_LATENCY_MS = 500 +const PINNED_HOLD_MS = 1_600 + +describe.each(HELD_ROUTES)('R3-L7 $label pins the hold deadline before host resolution', route => { + it('a slow host resolution is charged to the hold budget', async () => { + config.upstreamTimeoutMs = 30_000 + config.artifactDownloadTimeoutMs = 30_000 + config.wakeMaxHoldMs = PINNED_HOLD_MS + serviceMock.forwardTaskResultFromHost.mockImplementation( + forwardTaskResultFromHost as typeof serviceMock.forwardTaskResultFromHost + ) + serviceMock.forwardCancelToHost.mockImplementation( + forwardCancelToHost as typeof serviceMock.forwardCancelToHost + ) + await startUpstream((req, _res, index) => { + if (index === 0) dropConnection(req) // first attempt: host down + // the wake retry is never answered + }) + const { port } = server!.address() as AddressInfo + serviceMock.resolveHostConnectionForUser.mockImplementation(async () => { + await new Promise(resolve => setTimeout(resolve, HOST_RESOLUTION_LATENCY_MS)) + return { name: 'chatllm', url: `http://127.0.0.1:${port}`, headers: {} } + }) + + const startedAt = Date.now() + const call = + route.method === 'post' + ? request(makeApp()).post(route.path).set('authorization', 'Bearer tok').send(route.body) + : request(makeApp()).get(route.path).set('authorization', 'Bearer tok') + const res = await call.expect(504) + const elapsed = Date.now() - startedAt + + expect(res.body).toEqual({ error: 'Gateway Timeout' }) + // Witness: the slow resolution ran, the first attempt reached the upstream, + // a wake was requested and the retry reached the upstream before the + // deadline aborted it. + expect(serviceMock.resolveHostConnectionForUser).toHaveBeenCalledTimes(1) + expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalledTimes(1) + expect(seen.map(entry => entry.url)).toEqual([route.upstreamPath, route.upstreamPath]) + await vi.waitFor(() => expect(seen[1]!.closedEarly).toBe(true)) + // Pinned at arrival: the whole request fits in the hold budget. Pinned after + // the resolution it would take HOST_RESOLUTION_LATENCY_MS longer. + expect(elapsed).toBeGreaterThanOrEqual(PINNED_HOLD_MS - 100) + expect(elapsed).toBeLessThan(PINNED_HOLD_MS + 400) + }) +}) + +// R3-L6: the first attempt of a mutating call has no client-side timeout (an +// abort after the upstream applied it would answer 504 for a change that took +// effect), but a client that is gone must still release the upstream socket. +const DISCONNECT_ROUTES = [ + ...MUTATING_ROUTES.map(route => ({ ...route, method: 'POST' as const })), + { + label: 'cancel task', + method: 'POST', + path: '/rpc/hosts/chatllm/tasks/task-1/cancel', + body: {}, + upstreamPath: '/v1/runtime/tasks/task-1/cancel', + }, + { + label: 'rename session', + method: 'PATCH', + path: '/rpc/hosts/chatllm/sessions/chatllm/chat-1/name', + body: { title: 'Renamed chat' }, + upstreamPath: '/v1/runtime/sessions/chatllm/chat-1/name', + }, +] as const + +const CLIENT_GONE_RELEASE_MS = 500 + +describe.each(DISCONNECT_ROUTES)( + 'R3-L6 $label releases the upstream call when the client disconnects', + route => { + it('a client that goes away closes the in-flight upstream socket', async () => { + serviceMock.forwardCancelToHost.mockImplementation( + forwardCancelToHost as typeof serviceMock.forwardCancelToHost + ) + await startUpstream(() => { + // never answered: only the client disconnect can end the upstream call + }) + const proxy = makeApp().listen(0, '127.0.0.1') + await new Promise((resolve, reject) => { + proxy.once('listening', resolve) + proxy.once('error', reject) + }) + try { + const { port } = proxy.address() as AddressInfo + const payload = JSON.stringify(route.body) + const clientErrors: string[] = [] + const clientReq = httpRequest({ + host: '127.0.0.1', + port, + method: route.method, + path: route.path, + headers: { + authorization: 'Bearer tok', + 'content-type': 'application/json', + 'content-length': Buffer.byteLength(payload), + }, + }) + // Recorded, not ignored: destroying the request is the event under test. + clientReq.on('error', error => clientErrors.push(error.message)) + clientReq.end(payload) + + // Witness: the POST reached the upstream and is in flight. + await vi.waitFor(() => expect(seen).toHaveLength(1), { timeout: 2_000, interval: 10 }) + expect(seen[0]!.closedEarly).toBe(false) + + clientReq.destroy() + const disconnectedAt = Date.now() + await vi.waitFor(() => expect(seen[0]!.closedEarly).toBe(true), { + timeout: CLIENT_GONE_RELEASE_MS, + interval: 10, + }) + expect(Date.now() - disconnectedAt).toBeLessThan(CLIENT_GONE_RELEASE_MS) + expect(clientErrors).toEqual(['socket hang up']) + expect(seen.map(entry => `${entry.method} ${entry.url}`)).toEqual([ + `${route.method} ${route.upstreamPath}`, + ]) + expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() + } finally { + proxy.closeAllConnections() + await new Promise(resolve => proxy.close(() => resolve())) + } + }) + + it('a client gone before the upstream call starts never sends it', async () => { + serviceMock.forwardCancelToHost.mockImplementation( + forwardCancelToHost as typeof serviceMock.forwardCancelToHost + ) + await startUpstream(() => { + // never answered + }) + const { port: upstreamPort } = server!.address() as AddressInfo + let releaseResolution: () => void = () => { + throw new Error('host resolution was never started') + } + serviceMock.resolveHostConnectionForUser.mockImplementation(async () => { + await new Promise(resolve => { + releaseResolution = resolve + }) + return { name: 'chatllm', url: `http://127.0.0.1:${upstreamPort}`, headers: {} } + }) + const proxy = makeApp().listen(0, '127.0.0.1') + await new Promise((resolve, reject) => { + proxy.once('listening', resolve) + proxy.once('error', reject) + }) + const fetchSpy = vi.spyOn(globalThis, 'fetch') + try { + const { port } = proxy.address() as AddressInfo + const payload = JSON.stringify(route.body) + const clientErrors: string[] = [] + const clientReq = httpRequest({ + host: '127.0.0.1', + port, + method: route.method, + path: route.path, + headers: { + authorization: 'Bearer tok', + 'content-type': 'application/json', + 'content-length': Buffer.byteLength(payload), + }, + }) + // Recorded, not ignored: destroying the request is the event under test. + clientReq.on('error', error => clientErrors.push(error.message)) + clientReq.end(payload) + + // The client leaves while the route is still resolving the host, so the + // response has already closed when the upstream call is prepared. + await vi.waitFor( + () => expect(serviceMock.resolveHostConnectionForUser).toHaveBeenCalledTimes(1), + { timeout: 2_000, interval: 10 } + ) + clientReq.destroy() + await new Promise(resolve => setTimeout(resolve, 100)) + releaseResolution() + + // Liveness witness: the route got past the resolution and started the + // upstream call, on a signal that was already aborted. + const upstreamCall = () => + fetchSpy.mock.calls.find(([input]) => String(input).endsWith(route.upstreamPath)) + await vi.waitFor(() => expect(upstreamCall()).toBeDefined(), { + timeout: 2_000, + interval: 10, + }) + expect(upstreamCall()![1]?.signal?.aborted).toBe(true) + await new Promise(resolve => setTimeout(resolve, 200)) + expect(seen).toEqual([]) + expect(clientErrors).toEqual(['socket hang up']) + expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() + } finally { + fetchSpy.mockRestore() + proxy.closeAllConnections() + await new Promise(resolve => proxy.close(() => resolve())) + } + }) + } +) + +// R3-L4: task cancel is mutating like approve/deny/model, so its first attempt +// has no client-side timeout either: a slow upstream answer is relayed. +describe('R3-L4 task cancel first attempt has no client-side timeout', () => { + it('relays an upstream answer that arrives after upstreamTimeoutMs', async () => { + config.upstreamTimeoutMs = 200 + serviceMock.forwardCancelToHost.mockImplementation( + forwardCancelToHost as typeof serviceMock.forwardCancelToHost + ) + await startUpstream((_req, res) => { + setTimeout(() => { + res.writeHead(200, { 'content-type': 'application/json' }) + res.end(JSON.stringify({ cancelled: true })) + }, 400) + }) + + const res = await request(makeApp()) + .post('/rpc/hosts/chatllm/tasks/task-1/cancel') + .set('authorization', 'Bearer tok') + .expect(200) + + expect(res.body).toEqual({ cancelled: true }) + // Witness: exactly one POST, answered in full by the upstream. + expect(seen).toEqual([ + { method: 'POST', url: '/v1/runtime/tasks/task-1/cancel', closedEarly: false }, + ]) + expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() + }) +}) diff --git a/rpc-proxy/src/__tests__/session-rename-passthrough.test.ts b/rpc-proxy/src/__tests__/session-rename-passthrough.test.ts index c1d28c348e..eaf2e65b29 100644 --- a/rpc-proxy/src/__tests__/session-rename-passthrough.test.ts +++ b/rpc-proxy/src/__tests__/session-rename-passthrough.test.ts @@ -120,7 +120,10 @@ describe('PATCH /rpc/hosts/:hostRef/sessions/:agent/:chatId/name — rename pass }) it('returns 403 when the user cannot access the host', async () => { - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) const fetchMock = vi.fn() globalThis.fetch = fetchMock as unknown as typeof fetch diff --git a/rpc-proxy/src/__tests__/sessions-passthrough.test.ts b/rpc-proxy/src/__tests__/sessions-passthrough.test.ts index 21ce230b38..4438be2eca 100644 --- a/rpc-proxy/src/__tests__/sessions-passthrough.test.ts +++ b/rpc-proxy/src/__tests__/sessions-passthrough.test.ts @@ -162,7 +162,10 @@ describe('GET /rpc/hosts/:hostRef/sessions — passthrough to mcp-host', () => { }) it('returns 403 if the user cannot access the host', async () => { - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) await request(makeApp()) .get('/rpc/hosts/other-host/sessions') .set('authorization', 'Bearer user-token') @@ -422,7 +425,10 @@ describe('GET /rpc/hosts/:hostRef/sessions/:agent/:chatId/context-breakdown — }) it('returns 403 if the user cannot access the host', async () => { - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) await request(makeApp()) .get('/rpc/hosts/other-host/sessions/chatllm/c1/context-breakdown') .set('authorization', 'Bearer user-token') diff --git a/rpc-proxy/src/__tests__/wake-and-hold-idempotent-gets.real-http.test.ts b/rpc-proxy/src/__tests__/wake-and-hold-idempotent-gets.real-http.test.ts new file mode 100644 index 0000000000..2f1b8c5025 --- /dev/null +++ b/rpc-proxy/src/__tests__/wake-and-hold-idempotent-gets.real-http.test.ts @@ -0,0 +1,299 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import express from 'express' +import { type IncomingMessage, type Server, type ServerResponse, createServer } from 'node:http' +import type { AddressInfo } from 'node:net' +import request from 'supertest' +import { config } from '../config.js' +import { apiErrorHandler } from '../errorHandler.js' +import { createRpcRouter } from '../routes/rpc.js' +import { forwardCancelToHost, forwardTaskResultFromHost } from '../services/mcpHostRestService.js' + +// PR #849 R3-M1: a wake-and-hold retry of an idempotent GET keeps re-issuing +// the read until the hold deadline, like `/messages`; the mutating routes keep +// their single re-issue. Every test talks to a REAL http server through the +// real `fetch`; the requests the upstream records are the liveness witness. + +const authTokenMock = vi.hoisted(() => ({ verifyRpcToken: vi.fn() })) +const serviceMock = vi.hoisted(() => ({ + listAllowedServersForUser: vi.fn(), + resolveServerConnectionForUser: vi.fn(), + resolveHostConnectionForUser: vi.fn(), + validateRpcRequest: vi.fn(), + forwardRpcToServer: vi.fn(), + forwardHostMessageToHost: vi.fn(), + forwardHostActivity: vi.fn(), + forwardHostStatus: vi.fn(), + forwardHostHealth: vi.fn(), + forwardTaskResultFromHost: vi.fn(), + forwardCancelToHost: vi.fn(), + UpstreamHostError: class UpstreamHostError extends Error { + constructor( + public readonly status: number, + public readonly bodySnippet: string + ) { + super(`Upstream host returned ${status}: ${bodySnippet}`) + this.name = 'UpstreamHostError' + } + }, +})) +const controlApiMock = vi.hoisted(() => ({ + fetchUserAllowedServersFromControlApi: vi.fn(), + fetchHostConnectionFromControlApi: vi.fn(), + requestHostWakeFromControlApi: vi.fn(), +})) + +vi.mock('../authToken.js', () => authTokenMock) +vi.mock('../services/mcpProxyService.js', () => serviceMock) +vi.mock('../services/controlApiRestService.js', () => controlApiMock) + +function claims() { + return { + sub: 'user-uuid-123', + typ: 'user' as const, + accessScope: 'team' as const, + teamId: 'team-1', + scopes: [ + 'host:approval:write', + 'host:model:write', + 'host:session:read', + 'host:task:read', + 'host:message:invoke', + 'host:wake:write', + ], + hostRefs: ['chatllm'], + jti: 'j1', + iat: 1, + exp: 9999999999, + } +} + +function makeApp() { + const app = express() + app.use(express.json()) + app.use(createRpcRouter()) + app.use(apiErrorHandler) + return app +} + +type SeenRequest = { method: string; url: string } +type Handler = (req: IncomingMessage, res: ServerResponse, index: number) => void + +let server: Server | undefined +let seen: SeenRequest[] = [] + +async function startUpstream(handler: Handler): Promise { + seen = [] + server = createServer((req, res) => { + seen.push({ method: req.method ?? '', url: req.url ?? '' }) + handler(req, res, seen.length - 1) + }) + await new Promise((resolve, reject) => { + server!.once('error', reject) + server!.listen(0, '127.0.0.1', resolve) + }) + const { port } = server.address() as AddressInfo + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + name: 'chatllm', + url: `http://127.0.0.1:${port}`, + headers: {}, + }) +} + +/** Ends the TCP connection without a response: undici reports it as `fetch failed`. */ +function dropConnection(req: IncomingMessage): void { + req.socket.destroy() +} + +function answerJson(res: ServerResponse, status: number, body: unknown): void { + res.writeHead(status, { 'content-type': 'application/json' }) + res.end(JSON.stringify(body)) +} + +/** The first `drops` requests lose their connection; later ones answer 200. */ +function dropThenAnswer(drops: number, body: unknown): Handler { + return (req, res, index) => { + if (index < drops) { + dropConnection(req) + return + } + answerJson(res, 200, body) + } +} + +const original = { + upstreamTimeoutMs: config.upstreamTimeoutMs, + wakeMaxHoldMs: config.wakeMaxHoldMs, +} + +beforeEach(() => { + vi.clearAllMocks() + authTokenMock.verifyRpcToken.mockReturnValue(claims()) + controlApiMock.requestHostWakeFromControlApi.mockResolvedValue({ + kind: 'active', + wakeGeneration: null, + }) + serviceMock.forwardHostHealth.mockResolvedValue({ ok: true }) + serviceMock.forwardTaskResultFromHost.mockImplementation( + forwardTaskResultFromHost as typeof serviceMock.forwardTaskResultFromHost + ) + serviceMock.forwardCancelToHost.mockImplementation( + forwardCancelToHost as typeof serviceMock.forwardCancelToHost + ) + config.upstreamTimeoutMs = 2_000 + config.wakeMaxHoldMs = 4_000 +}) + +afterEach(async () => { + config.upstreamTimeoutMs = original.upstreamTimeoutMs + config.wakeMaxHoldMs = original.wakeMaxHoldMs + if (server) { + server.closeAllConnections() + await new Promise(resolve => server!.close(() => resolve())) + server = undefined + } +}) + +const IDEMPOTENT_GET_ROUTES = [ + { + label: 'list sessions', + path: '/rpc/hosts/chatllm/sessions', + upstreamPath: '/v1/runtime/sessions', + }, + { + label: 'session transcript', + path: '/rpc/hosts/chatllm/sessions/agent-1/chat-1/messages', + upstreamPath: '/v1/runtime/sessions/agent-1/chat-1/messages', + }, + { + label: 'context breakdown', + path: '/rpc/hosts/chatllm/sessions/agent-1/chat-1/context-breakdown', + upstreamPath: '/v1/runtime/sessions/agent-1/chat-1/context-breakdown', + }, + { + label: 'list models', + path: '/rpc/hosts/chatllm/models', + upstreamPath: '/v1/runtime/models', + }, + { + label: 'task result', + path: '/rpc/hosts/chatllm/tasks/task-1/result', + upstreamPath: '/v1/runtime/tasks/task-1/result', + }, + { + label: 'list artifacts', + path: '/rpc/hosts/chatllm/artifacts', + upstreamPath: '/v1/runtime/artifacts', + }, + { + label: 'artifact download', + path: '/rpc/hosts/chatllm/artifacts/report.bin/download', + upstreamPath: '/v1/runtime/artifacts/report.bin/download', + }, +] as const + +describe.each(IDEMPOTENT_GET_ROUTES)('R3-M1 $label retries until the hold deadline', route => { + it('re-issues the read past a second dropped connection and relays the answer', async () => { + const payload = { route: route.label, ok: true } + await startUpstream(dropThenAnswer(2, payload)) + + const res = await request(makeApp()) + .get(route.path) + .set('authorization', 'Bearer tok') + .buffer(true) + .parse((response, callback) => { + let text = '' + response.setEncoding('utf8') + response.on('data', (chunk: string) => (text += chunk)) + response.on('end', () => callback(null, text)) + }) + + expect(res.status).toBe(200) + expect(JSON.parse(res.body as string)).toEqual(payload) + // Witness: two dropped reads and the accepted third one all reached the + // upstream, and exactly one wake was requested for them. + expect(seen).toEqual([ + { method: 'GET', url: route.upstreamPath }, + { method: 'GET', url: route.upstreamPath }, + { method: 'GET', url: route.upstreamPath }, + ]) + expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalledTimes(1) + }) +}) + +// Paired control: the same upstream behavior on a mutating route keeps the +// single re-issue (no idempotency key), so the caller gets host_waking with +// exactly two POSTs and the third never happens. +const MUTATING_ROUTES = [ + { + label: 'approve', + path: '/rpc/hosts/chatllm/approvals/approve', + body: { toolCallId: 'tc-1' }, + upstreamPath: '/v1/runtime/approvals/approve', + }, + { + label: 'deny', + path: '/rpc/hosts/chatllm/approvals/deny', + body: { toolCallId: 'tc-1' }, + upstreamPath: '/v1/runtime/approvals/deny', + }, + { + label: 'set model', + path: '/rpc/hosts/chatllm/model', + body: { chatId: 'c1', model: 'claude-haiku-4-5' }, + upstreamPath: '/v1/runtime/model', + }, + { + label: 'cancel task', + path: '/rpc/hosts/chatllm/tasks/task-1/cancel', + body: {}, + upstreamPath: '/v1/runtime/tasks/task-1/cancel', + }, +] as const + +describe.each(MUTATING_ROUTES)('R3-M1 control: $label keeps a single re-issue', route => { + it('answers host_waking after exactly two POSTs', async () => { + await startUpstream(dropThenAnswer(2, { applied: true })) + + const res = await request(makeApp()) + .post(route.path) + .set('authorization', 'Bearer tok') + .send(route.body) + .expect(503) + + expect(res.body).toMatchObject({ code: 'host_waking', hostRef: 'chatllm' }) + // Witness: the wake was requested and the re-issue reached the upstream. + expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalledTimes(1) + expect(seen).toEqual([ + { method: 'POST', url: route.upstreamPath }, + { method: 'POST', url: route.upstreamPath }, + ]) + }) +}) + +// Lateral finding: forwardTaskResultFromHost threw its UpstreamHostError with an +// empty body, so mcp-host's `host_draining` fence was invisible and a draining +// Host answered 502 without a wake. +describe('task result sees the host_draining fence', () => { + it('wakes and relays the re-issued read after a draining 503', async () => { + await startUpstream((_req, res, index) => { + if (index === 0) { + answerJson(res, 503, { code: 'host_draining' }) + return + } + answerJson(res, 200, { taskId: 'task-1', status: 'completed' }) + }) + + const res = await request(makeApp()) + .get('/rpc/hosts/chatllm/tasks/task-1/result') + .set('authorization', 'Bearer tok') + .expect(200) + + expect(res.body).toEqual({ taskId: 'task-1', status: 'completed' }) + // Witness: the draining answer was seen, one wake followed, then one retry. + expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalledTimes(1) + expect(seen.map(entry => `${entry.method} ${entry.url}`)).toEqual([ + 'GET /v1/runtime/tasks/task-1/result', + 'GET /v1/runtime/tasks/task-1/result', + ]) + }) +}) diff --git a/rpc-proxy/src/__tests__/wake-and-hold-route.test.ts b/rpc-proxy/src/__tests__/wake-and-hold-route.test.ts index bf37f6bf4c..08fea4551e 100644 --- a/rpc-proxy/src/__tests__/wake-and-hold-route.test.ts +++ b/rpc-proxy/src/__tests__/wake-and-hold-route.test.ts @@ -1,7 +1,12 @@ -import { beforeEach, describe, expect, it, vi } from 'vitest' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import express from 'express' +import { createServer } from 'node:http' +import type { AddressInfo } from 'node:net' import request from 'supertest' +import { config } from '../config.js' +import { apiErrorHandler } from '../errorHandler.js' import { createRpcRouter } from '../routes/rpc.js' +import { forwardHostMessageToHost } from '../services/mcpHostRestService.js' const authTokenMock = vi.hoisted(() => ({ verifyRpcToken: vi.fn(), @@ -62,16 +67,14 @@ const HOST_CONNECTION = { url: 'http://chatllm.mcp-host.svc.cluster.local:8080', headers: {}, } +const originalUpstreamTimeoutMs = config.upstreamTimeoutMs +const originalFetch = globalThis.fetch function makeApp() { const app = express() app.use(express.json()) app.use(createRpcRouter()) - app.use( - (err: unknown, _req: express.Request, res: express.Response, _next: express.NextFunction) => { - res.status(500).json({ error: err instanceof Error ? err.message : 'Internal error' }) - } - ) + app.use(apiErrorHandler) return app } @@ -96,6 +99,12 @@ beforeEach(() => { serviceMock.resolveHostConnectionForUser.mockResolvedValue({ ...HOST_CONNECTION }) }) +afterEach(() => { + vi.useRealTimers() + config.upstreamTimeoutMs = originalUpstreamTimeoutMs + globalThis.fetch = originalFetch +}) + describe('POST /rpc/hosts/:hostRef/messages wake-and-hold triggers', () => { it('host-down network error triggers a wake; 200 active leads to one immediate upstream retry', async () => { serviceMock.forwardHostMessageToHost @@ -112,6 +121,29 @@ describe('POST /rpc/hosts/:hostRef/messages wake-and-hold triggers', () => { expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalledTimes(1) expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalledWith('chatllm', 'token') expect(serviceMock.forwardHostMessageToHost).toHaveBeenCalledTimes(2) + expect(serviceMock.forwardHostMessageToHost.mock.calls[0][2]).toEqual({ async: false }) + }) + + it('uses the normal POST timeout before wake-hold retries until the suspended Host admits', async () => { + serviceMock.forwardHostMessageToHost + .mockRejectedValueOnce(hostDownError()) // Initial POST fails quickly on the suspended Host. + .mockRejectedValueOnce(hostDownError()) // Wake was accepted, but admission is not ready yet. + .mockResolvedValueOnce({ success: true, taskId: 't-delayed' }) + controlApiMock.requestHostWakeFromControlApi.mockResolvedValue({ + kind: 'active', + wakeGeneration: null, + }) + + const response = await postMessage(makeApp()).expect(200) + + expect(response.body).toEqual({ success: true, taskId: 't-delayed' }) + expect(serviceMock.forwardHostMessageToHost).toHaveBeenCalledTimes(3) + // The first POST omits timeoutMs so the service applies its normal full upstream timeout. + expect(serviceMock.forwardHostMessageToHost.mock.calls[0][2]).toEqual({ async: false }) + expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalledTimes(1) + expect(serviceMock.forwardHostMessageToHost.mock.calls[0][1]).toBe( + serviceMock.forwardHostMessageToHost.mock.calls.at(-1)![1] + ) }) it('the wake retry re-forwards the SAME stable messageId as the first forward (idempotency identity)', async () => { @@ -220,6 +252,7 @@ describe('POST /rpc/hosts/:hostRef/messages wake-and-hold triggers', () => { expect(response.body).toEqual({ success: true, taskId: 't-2' }) expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalledTimes(1) + expect(serviceMock.forwardHostMessageToHost.mock.calls[0][2]).toEqual({ async: false }) }) it('drain-cancel bounce resolves within the short retry schedule (250ms then 1s)', async () => { @@ -238,13 +271,15 @@ describe('POST /rpc/hosts/:hostRef/messages wake-and-hold triggers', () => { expect(serviceMock.forwardHostMessageToHost).toHaveBeenCalledTimes(3) }) - it('409 not-stateless keeps today behavior exactly: 502 Upstream host unavailable', async () => { + it('409 not-stateless keeps the full upstream timeout and legacy 502 behavior', async () => { serviceMock.forwardHostMessageToHost.mockRejectedValue(hostDownError()) controlApiMock.requestHostWakeFromControlApi.mockResolvedValue({ kind: 'not-stateless' }) const response = await postMessage(makeApp()).expect(502) expect(response.body).toEqual({ error: 'Upstream host unavailable' }) + expect(serviceMock.forwardHostMessageToHost.mock.calls[0][2]).toEqual({ async: false }) + expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalledTimes(1) }) it('404 unknown host maps to a 404 for the caller', async () => { @@ -277,20 +312,29 @@ describe('POST /rpc/hosts/:hostRef/messages wake-and-hold triggers', () => { }) it('host still unreachable after a wake reports active resolves to host_waking, never a hang', async () => { + let wakeStarted!: () => void + const wakeCalled = new Promise(resolve => { + wakeStarted = resolve + }) serviceMock.forwardHostMessageToHost.mockRejectedValue(hostDownError()) - controlApiMock.requestHostWakeFromControlApi.mockResolvedValue({ - kind: 'active', - wakeGeneration: null, + controlApiMock.requestHostWakeFromControlApi.mockImplementation(async () => { + vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout', 'Date'] }) + wakeStarted() + return { kind: 'active', wakeGeneration: null } }) - const response = await postMessage(makeApp()).expect(503) + const responsePromise = postMessage(makeApp()) + .expect(503) + .then(response => response) + await wakeCalled + await vi.advanceTimersByTimeAsync(48_000) + const response = await responsePromise expect(response.body).toMatchObject({ code: 'host_waking', hostRef: 'chatllm' }) - // Trigger + the full short retry schedule were attempted. - expect(serviceMock.forwardHostMessageToHost).toHaveBeenCalledTimes(4) + expect(serviceMock.forwardHostMessageToHost.mock.calls.length).toBeGreaterThan(4) }) - it('a non-availability upstream failure (mcp-host 500) never triggers a wake', async () => { + it('a non-availability upstream failure (mcp-host 500) never enters the wake hold', async () => { serviceMock.forwardHostMessageToHost.mockRejectedValue( new serviceMock.UpstreamHostError(500, '{"error":"boom"}') ) @@ -301,7 +345,7 @@ describe('POST /rpc/hosts/:hostRef/messages wake-and-hold triggers', () => { expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() }) - it('an upstream AbortError keeps the 504 path and never triggers a wake', async () => { + it('an upstream AbortError keeps the 504 path without entering the wake hold', async () => { const abort = new Error('aborted') abort.name = 'AbortError' serviceMock.forwardHostMessageToHost.mockRejectedValue(abort) @@ -311,6 +355,111 @@ describe('POST /rpc/hosts/:hostRef/messages wake-and-hold triggers', () => { expect(response.body).toEqual({ error: 'Gateway Timeout' }) expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() }) + + it('returns 504 when the real host REST service times out a hanging HTTP request', async () => { + config.upstreamTimeoutMs = 25 + const server = createServer(() => { + // Leave the request unanswered so the service timeout must abort it. + }) + await new Promise((resolve, reject) => { + server.once('error', reject) + server.listen(0, '127.0.0.1', resolve) + }) + const { port } = server.address() as AddressInfo + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + ...HOST_CONNECTION, + url: `http://127.0.0.1:${port}`, + }) + serviceMock.forwardHostMessageToHost.mockImplementation( + forwardHostMessageToHost as typeof serviceMock.forwardHostMessageToHost + ) + + try { + await postMessage(makeApp()).expect(504, { error: 'Gateway Timeout' }) + expect(serviceMock.forwardHostMessageToHost).toHaveBeenCalledTimes(1) + expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() + } finally { + server.closeAllConnections() + await new Promise(resolve => server.close(() => resolve())) + } + }) + + it('a wake retry that times out returns 504 instead of 502', async () => { + const timeout = new Error('aborted') + timeout.name = 'AbortError' + serviceMock.forwardHostMessageToHost + .mockRejectedValueOnce(hostDownError()) + .mockRejectedValueOnce(timeout) + controlApiMock.requestHostWakeFromControlApi.mockResolvedValue({ + kind: 'active', + wakeGeneration: null, + }) + + const response = await postMessage(makeApp()).expect(504) + + expect(response.body).toEqual({ error: 'Gateway Timeout' }) + expect(serviceMock.forwardHostMessageToHost).toHaveBeenCalledTimes(2) + expect(serviceMock.forwardHostMessageToHost.mock.calls[1][2]).toEqual({ + async: false, + timeoutMs: expect.any(Number), + }) + }) + + it('the real REST forwarder aborts a hanging wake retry and returns 504', async () => { + config.upstreamTimeoutMs = 30 + let retrySignal: AbortSignal | undefined + const fetchMock = vi + .fn() + .mockRejectedValueOnce(hostDownError()) + .mockImplementationOnce((_url: string, init: RequestInit) => { + retrySignal = init.signal as AbortSignal + return new Promise((_resolve, reject) => { + retrySignal!.addEventListener('abort', () => reject(retrySignal!.reason), { + once: true, + }) + }) + }) + globalThis.fetch = fetchMock as unknown as typeof fetch + serviceMock.forwardHostMessageToHost.mockImplementation( + forwardHostMessageToHost as typeof serviceMock.forwardHostMessageToHost + ) + controlApiMock.requestHostWakeFromControlApi.mockResolvedValue({ + kind: 'active', + wakeGeneration: null, + }) + + const response = await postMessage(makeApp()).expect(504) + + expect(response.body).toEqual({ error: 'Gateway Timeout' }) + expect(fetchMock).toHaveBeenCalledTimes(2) + expect(retrySignal?.aborted).toBe(true) + expect(controlApiMock.requestHostWakeFromControlApi).toHaveBeenCalledTimes(1) + }) + + it('a token without wake scope keeps the original full upstream timeout', async () => { + authTokenMock.verifyRpcToken.mockReturnValue({ + ...VALID_CLAIMS, + scopes: ['host:message:invoke'], + }) + const abort = new Error('aborted') + abort.name = 'AbortError' + serviceMock.forwardHostMessageToHost.mockRejectedValue(abort) + + await postMessage(makeApp()).expect(504) + + expect(serviceMock.forwardHostMessageToHost).toHaveBeenCalledTimes(1) + expect(serviceMock.forwardHostMessageToHost.mock.calls[0][2]).toEqual({ async: false }) + expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() + }) + + it('uses the normal upstream timeout for a healthy POST without requesting a wake', async () => { + serviceMock.forwardHostMessageToHost.mockResolvedValue({ success: true, taskId: 't-stateful' }) + + await postMessage(makeApp()).expect(200) + + expect(serviceMock.forwardHostMessageToHost.mock.calls[0][2]).toEqual({ async: false }) + expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() + }) }) describe('task result and cancel host-down paths', () => { diff --git a/rpc-proxy/src/__tests__/wake-route-coverage.test.ts b/rpc-proxy/src/__tests__/wake-route-coverage.test.ts index b7289c7498..c68ff812eb 100644 --- a/rpc-proxy/src/__tests__/wake-route-coverage.test.ts +++ b/rpc-proxy/src/__tests__/wake-route-coverage.test.ts @@ -2,6 +2,8 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import type { Response as ExpressResponse } from 'express' import express from 'express' import request from 'supertest' +import { config } from '../config.js' +import { apiErrorHandler } from '../errorHandler.js' import { createRpcRouter, respondControlApiHostAccessRejection, @@ -70,11 +72,7 @@ function makeApp() { const app = express() app.use(express.json()) app.use(createRpcRouter()) - app.use( - (err: unknown, _req: express.Request, res: express.Response, _next: express.NextFunction) => { - res.status(500).json({ error: err instanceof Error ? err.message : 'Internal error' }) - } - ) + app.use(apiErrorHandler) return app } @@ -103,6 +101,7 @@ function drainingResponse(): Response { } const originalFetch = globalThis.fetch +const originalUpstreamTimeoutMs = config.upstreamTimeoutMs beforeEach(() => { vi.clearAllMocks() @@ -115,6 +114,79 @@ beforeEach(() => { afterEach(() => { globalThis.fetch = originalFetch + config.upstreamTimeoutMs = originalUpstreamTimeoutMs +}) + +// R2-M3: deny and set-model mutate host state, so their FIRST attempt carries no +// client-side deadline (a timeout would 504 a change the host may have applied). +// Only the read-only artifacts list keeps a finite first-attempt deadline. The +// real-socket versions of these (slow upstream, hung wake retry) are in +// route-upstream-deadlines.real-http.test.ts. +describe('first-attempt upstream deadlines', () => { + it.each([ + { + label: 'deny approval', + scope: 'host:approval:write', + path: '/rpc/hosts/chatllm/approvals/deny', + body: { toolCallId: 'tc-timeout' }, + }, + { + label: 'set model', + scope: 'host:model:write', + path: '/rpc/hosts/chatllm/model', + body: { chatId: 'c1', model: 'claude-haiku-4-5' }, + }, + ])('$label sends its first attempt without a client-side deadline', async row => { + config.upstreamTimeoutMs = 30 + authTokenMock.verifyRpcToken.mockReturnValue(claims([row.scope, 'host:wake:write'])) + // The first attempt carries only the client-disconnect signal (R3-L6): it + // must outlive several upstream timeouts without being aborted. + let firstSignal: AbortSignal | undefined + let abortedWhenAnswered: boolean | undefined + const fetchMock = vi.fn().mockImplementation(async (_url: string, init: RequestInit) => { + firstSignal = init.signal as AbortSignal + await new Promise(resolve => setTimeout(resolve, config.upstreamTimeoutMs * 3)) + abortedWhenAnswered = firstSignal.aborted + return mockFetchResponse(200, JSON.stringify({ ok: true })) + }) + globalThis.fetch = fetchMock as unknown as typeof fetch + + await request(makeApp()) + .post(row.path) + .send(row.body) + .set('authorization', 'Bearer tok') + .expect(200) + + expect(fetchMock).toHaveBeenCalledTimes(1) + expect(firstSignal).toBeInstanceOf(AbortSignal) + expect(abortedWhenAnswered).toBe(false) + expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() + }) + + it('list artifacts returns 504 when the upstream deadline aborts a hung fetch', async () => { + config.upstreamTimeoutMs = 30 + authTokenMock.verifyRpcToken.mockReturnValue(claims(['host:task:read', 'host:wake:write'])) + let upstreamSignal: AbortSignal | undefined + const fetchMock = vi.fn().mockImplementation((_url: string, init: RequestInit) => { + upstreamSignal = init.signal as AbortSignal + return new Promise((_resolve, reject) => { + upstreamSignal!.addEventListener('abort', () => reject(upstreamSignal!.reason), { + once: true, + }) + }) + }) + globalThis.fetch = fetchMock as unknown as typeof fetch + + const res = await request(makeApp()) + .get('/rpc/hosts/chatllm/artifacts') + .set('authorization', 'Bearer tok') + .expect(504) + + expect(res.body).toEqual({ error: 'Gateway Timeout' }) + expect(upstreamSignal?.aborted).toBe(true) + expect(fetchMock).toHaveBeenCalledTimes(1) + expect(controlApiMock.requestHostWakeFromControlApi).not.toHaveBeenCalled() + }) }) describe('§11.4 wake coverage — GET /rpc/hosts/:hostRef/models', () => { @@ -552,8 +624,24 @@ describe('§11.5 route outer-catch responders guard headersSent (one-shot settle it('respondControlApiHostAccessRejection is a no-op once the response is committed', () => { const res = makeRes(true) expect(() => - respondControlApiHostAccessRejection(res as unknown as ExpressResponse, 403) + respondControlApiHostAccessRejection(res as unknown as ExpressResponse, { + status: 403, + code: 'host_access_revoked', + }) ).not.toThrow() expect(res.statusCode).toBe(0) }) + + it('respondControlApiHostAccessRejection writes the 403 with its denial code when uncommitted', () => { + const res = makeRes(false) + respondControlApiHostAccessRejection(res as unknown as ExpressResponse, { + status: 403, + code: 'host_access_revoked', + }) + expect(res.statusCode).toBe(403) + expect(res.body).toEqual({ + error: 'Forbidden: user cannot access this host', + code: 'host_access_revoked', + }) + }) }) diff --git a/rpc-proxy/src/app.ts b/rpc-proxy/src/app.ts index 4c33fbc66e..443a7e6825 100644 --- a/rpc-proxy/src/app.ts +++ b/rpc-proxy/src/app.ts @@ -1,6 +1,7 @@ -import express, { NextFunction, Request, Response } from 'express' +import express from 'express' import cors from 'cors' import { config } from './config.js' +import { apiErrorHandler } from './errorHandler.js' import { createDesktopRouter } from './routes/desktopProxy.js' import { createHealthRouter } from './routes/health.js' import { createMcpOauthRouter } from './routes/mcpOauth.js' @@ -9,21 +10,6 @@ import { createRpcHostActivityStreamRouter } from './routes/rpcHostActivityStrea import { createRpcHostProgressStreamRouter } from './routes/rpcHostProgressStream.js' import { createRpcHostStatusStreamRouter } from './routes/rpcHostStatusStream.js' import { createSandboxUiSessionRouter } from './routes/sandboxUi.js' -import { isUpstreamTimeoutError } from './services/wakeAndHold.js' - -/** - * body-parser flags a request whose declared/streamed body crossed the parser's - * `limit` with `type: 'entity.too.large'`. The terminal error handler below - * would otherwise report it as a 500, which reads like a server fault for what - * is a client-sized payload; mcp-host answers the same condition with 413. - */ -function isEntityTooLargeError(error: unknown): boolean { - return ( - typeof error === 'object' && - error !== null && - (error as { type?: unknown }).type === 'entity.too.large' - ) -} export function createApp() { const app = express() @@ -51,22 +37,7 @@ export function createApp() { res.status(404).json({ error: 'Not Found' }) }) - app.use((err: unknown, _req: Request, res: Response, _next: NextFunction) => { - if (isUpstreamTimeoutError(err)) { - res.status(504).json({ error: 'Gateway Timeout' }) - return - } - - if (isEntityTooLargeError(err)) { - res.status(413).json({ error: 'Payload Too Large' }) - return - } - - res.status(500).json({ - error: 'Internal Server Error', - message: err instanceof Error ? err.message : 'Unknown error', - }) - }) + app.use(apiErrorHandler) return app } diff --git a/rpc-proxy/src/config.ts b/rpc-proxy/src/config.ts index 21ac2c62d9..1a2bc84fe6 100644 --- a/rpc-proxy/src/config.ts +++ b/rpc-proxy/src/config.ts @@ -12,6 +12,7 @@ type Config = { controlApiServiceName: string controlApiCacheTtlMs: number artifactDownloadMaxBytes: number + artifactDownloadTimeoutMs: number streamMaxLifetimeMs: number streamIntervalMs: number streamKeepaliveMs: number @@ -28,7 +29,8 @@ type Config = { /** * Stateless wake-and-hold (Stage 5): a request hitting a down/draining * stateless host is parked while control-api wakes the pod, bounded by - * `wakeMaxHoldMs`. Readiness is re-checked every `wakePollMs` and the wake + * `wakeMaxHoldMs` (values above MAX_REQUEST_HOLD_MS are clamped at load). + * Readiness is re-checked every `wakePollMs` and the wake * is re-triggered every `wakeRetriggerMs` (covers a dropped HCC watch * event). See src/services/wakeAndHold.ts. */ @@ -135,15 +137,43 @@ export function parseArtifactDownloadMaxBytes(rawMb: string): number { return bytes } +/** Largest delay setTimeout/AbortSignal.timeout accept; above it they fire immediately. */ +export const MAX_TIMER_DELAY_MS = 2_147_483_647 + export function parsePositiveIntMs(name: string, raw: string): number { const trimmed = raw.trim() const n = Number(trimmed) if (!Number.isInteger(n) || n < 1) { throw new Error(`${name} contains invalid value "${trimmed}" (must be a positive integer)`) } + if (n > MAX_TIMER_DELAY_MS) { + throw new Error( + `${name} contains invalid value "${trimmed}" (must be at most ${MAX_TIMER_DELAY_MS} ms)` + ) + } return n } +/** + * Upper bound of one held request: it leaves room for a final upstream attempt + * before Desktop's own 60s deadline. The hold budget is capped here, at config + * load, so the configured value is the effective one. + */ +export const MAX_REQUEST_HOLD_MS = 48_000 + +/** + * An operator-set value above the cap is clamped and warned about at boot. + * The built-in default equals the cap, so it never reaches the warning. + */ +export function parseWakeMaxHoldMs(raw: string): number { + const configuredMs = parsePositiveIntMs('RPC_PROXY_WAKE_MAX_HOLD_MS', raw) + if (configuredMs <= MAX_REQUEST_HOLD_MS) return configuredMs + console.warn( + `[RPC_PROXY] RPC_PROXY_WAKE_MAX_HOLD_MS=${configuredMs} exceeds the request hold cap; clamped to ${MAX_REQUEST_HOLD_MS}` + ) + return MAX_REQUEST_HOLD_MS +} + const DEV_RPC_JWT_PUBLIC_KEY = normalizePem(`-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArCIYGHehMPpGKePxaKQa rDX5yrzifU5i4fzpI3EtkKSU6s5ug7EkKxc2DdMekoqXe9vr7qKyVwiilUIusXLX @@ -164,7 +194,10 @@ export const config: Config = { ), jwtIssuer: requiredOrDevDefault('RPC_PROXY_JWT_ISSUER', 'control-api'), jwtAudience: requiredOrDevDefault('RPC_PROXY_JWT_AUDIENCE', 'rpc-proxy'), - upstreamTimeoutMs: Number(process.env.RPC_PROXY_UPSTREAM_TIMEOUT_MS || 8000), + upstreamTimeoutMs: parsePositiveIntMs( + 'RPC_PROXY_UPSTREAM_TIMEOUT_MS', + process.env.RPC_PROXY_UPSTREAM_TIMEOUT_MS || '8000' + ), maxTokenLength: Number(process.env.RPC_PROXY_MAX_TOKEN_LENGTH || 4096), allowedMethodPattern: new RegExp( process.env.RPC_PROXY_ALLOWED_METHOD_PATTERN || '^[a-zA-Z0-9_./:-]{1,120}$' @@ -183,6 +216,10 @@ export const config: Config = { artifactDownloadMaxBytes: parseArtifactDownloadMaxBytes( process.env.RPC_PROXY_ARTIFACT_DOWNLOAD_MAX_MB || '50' ), + artifactDownloadTimeoutMs: parsePositiveIntMs( + 'RPC_PROXY_ARTIFACT_DOWNLOAD_TIMEOUT_MS', + process.env.RPC_PROXY_ARTIFACT_DOWNLOAD_TIMEOUT_MS || '60000' + ), streamMaxLifetimeMs: Number(process.env.RPC_PROXY_STREAM_MAX_LIFETIME_MS || 600000), streamIntervalMs: Number(process.env.RPC_PROXY_STREAM_INTERVAL_MS || 3000), streamKeepaliveMs: Number(process.env.RPC_PROXY_STREAM_KEEPALIVE_MS || 15000), @@ -202,10 +239,7 @@ export const config: Config = { activityStreamMaxPerUserHost: Number( process.env.RPC_PROXY_ACTIVITY_STREAM_MAX_PER_USER_HOST || 1 ), - wakeMaxHoldMs: parsePositiveIntMs( - 'RPC_PROXY_WAKE_MAX_HOLD_MS', - process.env.RPC_PROXY_WAKE_MAX_HOLD_MS || '90000' - ), + wakeMaxHoldMs: parseWakeMaxHoldMs(process.env.RPC_PROXY_WAKE_MAX_HOLD_MS || '48000'), wakePollMs: parsePositiveIntMs( 'RPC_PROXY_WAKE_POLL_MS', process.env.RPC_PROXY_WAKE_POLL_MS || '2000' diff --git a/rpc-proxy/src/errorHandler.ts b/rpc-proxy/src/errorHandler.ts new file mode 100644 index 0000000000..9d588dc422 --- /dev/null +++ b/rpc-proxy/src/errorHandler.ts @@ -0,0 +1,51 @@ +import type { NextFunction, Request, Response } from 'express' +import { UpstreamBodyReadError } from './services/upstreamBody.js' +import { isUpstreamTimeoutError } from './services/wakeAndHold.js' + +/** + * body-parser flags a request whose declared/streamed body crossed the parser's + * `limit` with `type: 'entity.too.large'`. The terminal error handler below + * would otherwise report it as a 500, which reads like a server fault for what + * is a client-sized payload; mcp-host answers the same condition with 413. + */ +function isEntityTooLargeError(error: unknown): boolean { + return ( + typeof error === 'object' && + error !== null && + (error as { type?: unknown }).type === 'entity.too.large' + ) +} + +/** + * The app's terminal Express error handler. It lives in its own module so route + * tests mount this very function instead of a copy of its 504 mapping: a test + * app that re-implements the mapping keeps passing after production changes it. + */ +export function apiErrorHandler( + err: unknown, + _req: Request, + res: Response, + _next: NextFunction +): void { + if (isUpstreamTimeoutError(err)) { + res.status(504).json({ error: 'Gateway Timeout' }) + return + } + + // The upstream answered with headers and its body then failed: a gateway + // failure on a request the host received, never an rpc-proxy fault (500). + if (err instanceof UpstreamBodyReadError) { + res.status(502).json({ error: 'Upstream host unavailable' }) + return + } + + if (isEntityTooLargeError(err)) { + res.status(413).json({ error: 'Payload Too Large' }) + return + } + + res.status(500).json({ + error: 'Internal Server Error', + message: err instanceof Error ? err.message : 'Unknown error', + }) +} diff --git a/rpc-proxy/src/routes/rpc.ts b/rpc-proxy/src/routes/rpc.ts index 9181da0590..99efa5c80f 100644 --- a/rpc-proxy/src/routes/rpc.ts +++ b/rpc-proxy/src/routes/rpc.ts @@ -17,6 +17,12 @@ import { fetchUserConnectorsFromControlApi, requestHostWakeFromControlApi, } from '../services/controlApiRestService.js' +import { + type HostAccessDenialCode, + isHostAccessDenied, + respondHostAccessDenied, + withoutReservedHostAccessCode, +} from '../services/hostAccessDenial.js' import { type HostRuntimeMessageRequest, forwardCancelToHost, @@ -31,6 +37,7 @@ import { resolveServerConnectionForUser, validateRpcRequest, } from '../services/mcpProxyService.js' +import { readMutatingResponseBody } from '../services/upstreamBody.js' import { isUpstreamTimeoutError, isWakeEligibleHostError, @@ -109,6 +116,23 @@ function guardedNext(res: ExpressResponse, next: NextFunction, error: unknown): next(error) } +/** + * The single writer of a relayed mcp-host response body. Every passthrough + * route answers through here so an upstream 403 can never carry the reserved + * Host-access `code` that only respondHostAccessDenied may set. `contentType` + * is set only by the routes that relay the upstream content type. + */ +function sendUpstreamBody( + res: ExpressResponse, + status: number, + body: string, + contentType?: string +): void { + res.status(status) + if (contentType !== undefined) res.type(contentType) + res.send(withoutReservedHostAccessCode(status, body)) +} + /** * A 503 carrying mcp-host's `host_draining` fence is a wake-eligible upstream * error, not a client-visible failure: convert it so the finite routes below @@ -123,35 +147,71 @@ function sessionDrainingFence(response: Response, body: string): Error | null { }) } -function controlApiHostAccessRejectionStatus(error: unknown): 401 | 403 | 409 | null { +export type ControlApiHostAccessRejection = + | { status: 401 | 409 } + | { status: 403; code: HostAccessDenialCode } + +function controlApiHostAccessRejection(error: unknown): ControlApiHostAccessRejection | null { if (!(error instanceof Error) || error.name !== 'ControlApiHostAccessRejectedError') { return null } - const status = (error as Error & { status?: unknown }).status - return status === 401 || status === 403 || status === 409 ? status : null + const { status, denialCode } = error as Error & { status?: unknown; denialCode?: unknown } + if (status === 401 || status === 409) return { status } + if ( + status === 403 && + (denialCode === 'host_access_revoked' || denialCode === 'host_access_denied') + ) { + return { status, code: denialCode } + } + return null } export function respondControlApiHostAccessRejection( res: ExpressResponse, - status: 401 | 403 | 409 + rejection: ControlApiHostAccessRejection ): void { if (res.headersSent) { console.warn( - `[RPC_PROXY] suppressing duplicate terminal response (host-access-rejection ${status})` + `[RPC_PROXY] suppressing duplicate terminal response (host-access-rejection ${rejection.status})` ) return } - if (status === 401) { + if (rejection.status === 401) { res.status(401).json({ error: 'Unauthorized' }) return } - if (status === 403) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (rejection.status === 403) { + respondHostAccessDenied(res, rejection.code) return } res.status(409).json({ error: 'Direct run attribution conflict' }) } +/** + * Abort signal for a MUTATING upstream call (approve, deny, set-model, task + * cancel). The first attempt has no client-side timeout by design: aborting a + * request the upstream may already have applied would answer 504 for a change + * that took effect, and the user's retry would then hit "No pending approval". + * Only a wake-and-hold retry passes `timeoutMs`, bounded by the hold deadline; + * session rename, which has no wake retry, passes its plain upstream timeout. + * A client that disconnects before the response is written aborts the call on + * every attempt, so a gone client never pins the upstream socket. + */ +function mutatingCallSignal(timeoutMs: number | undefined, res: ExpressResponse): AbortSignal { + const clientGone = new AbortController() + // A client can leave while the host is still being resolved; its 'close' + // has already fired by now, so a listener alone would never see it. + if (res.closed && !res.writableFinished) { + clientGone.abort(new Error('client disconnected')) + } + res.once('close', () => { + if (!res.writableFinished) clientGone.abort(new Error('client disconnected')) + }) + return timeoutMs === undefined + ? clientGone.signal + : AbortSignal.any([clientGone.signal, AbortSignal.timeout(timeoutMs)]) +} + class ProxyArtifactTooLargeError extends Error { constructor(readonly maxBytes: number) { super(`Artifact too large to download: max ${maxBytes} bytes`) @@ -404,6 +464,7 @@ export function createRpcRouter(): Router { : {}), } + const wakeDeadlineMs = Date.now() + config.wakeMaxHoldMs const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, ...(traceContext.sessionId @@ -416,8 +477,8 @@ export function createRpcRouter(): Router { } : {}), }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } if (host.attributionBindingStatus === 'unavailable') { @@ -439,6 +500,11 @@ export function createRpcRouter(): Router { ) let upstreamResponse: Record | null = null try { + // Always use the configured upstream timeout for the first POST. + // A separate statelessness pre-check would double-charge the + // rate-limited wake endpoint and risk aborting a healthy send. + // The wake path below already checks statelessness when a + // connection-level failure triggers it. upstreamResponse = await forwardHostMessageToHost(host, forwardedBody, { async: isAsync, }) @@ -458,9 +524,14 @@ export function createRpcRouter(): Router { host, claims: auth, rpcAccessToken, - attemptUpstream: async () => { + deadlineMs: wakeDeadlineMs, + // The per-request messageId makes a re-presented POST a + // duplicate_delivery replay, so retrying until the deadline is safe. + retryUntilDeadline: true, + attemptUpstream: async timeoutMs => { const retried = await forwardHostMessageToHost(host, forwardedBody, { async: isAsync, + timeoutMs, }) res.status(200).json(retried) }, @@ -477,9 +548,9 @@ export function createRpcRouter(): Router { // the wake path (which would re-forward an already-delivered message). res.status(200).json(upstreamResponse) } catch (error) { - const rejectedStatus = controlApiHostAccessRejectionStatus(error) - if (rejectedStatus) { - respondControlApiHostAccessRejection(res, rejectedStatus) + const rejection = controlApiHostAccessRejection(error) + if (rejection) { + respondControlApiHostAccessRejection(res, rejection) return } console.warn( @@ -515,8 +586,8 @@ export function createRpcRouter(): Router { const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } @@ -596,11 +667,12 @@ export function createRpcRouter(): Router { res.status(400).json({ error: 'hostRef is required' }) return } + const wakeDeadlineMs = Date.now() + config.wakeMaxHoldMs const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } const baseUrl = host.url.replace(/\/+$/, '') @@ -617,16 +689,17 @@ export function createRpcRouter(): Router { // host:approval:write; wake capability rides on the token. A suspended // (network-down) or draining Host triggers a wake-and-hold instead of a // bare next(error)/passthrough. - const attempt = async () => { + const attempt = async (timeoutMs?: number) => { const response = await fetch(`${baseUrl}/v1/runtime/approvals/approve`, { method: 'POST', headers: { 'content-type': 'application/json', ...host.headers }, body: JSON.stringify(upstreamBody), + signal: mutatingCallSignal(timeoutMs, res), }) - const body = await response.text() + const body = await readMutatingResponseBody(response) const draining = sessionDrainingFence(response, body) if (draining) throw draining - res.status(response.status).send(body) + sendUpstreamBody(res, response.status, body) } try { await attempt() @@ -638,6 +711,7 @@ export function createRpcRouter(): Router { host, claims: auth, rpcAccessToken, + deadlineMs: wakeDeadlineMs, attemptUpstream: attempt, respondLegacy: legacyError => respondUpstreamUnavailable(res, legacyError), }) @@ -663,11 +737,12 @@ export function createRpcRouter(): Router { res.status(400).json({ error: 'hostRef is required' }) return } + const wakeDeadlineMs = Date.now() + config.wakeMaxHoldMs const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } const baseUrl = host.url.replace(/\/+$/, '') @@ -680,16 +755,17 @@ export function createRpcRouter(): Router { requestId: parsed.toolCallId || parsed.requestId, } // Wake-eligible finite operation (§11.4): scope stays host:approval:write. - const attempt = async () => { + const attempt = async (timeoutMs?: number) => { const response = await fetch(`${baseUrl}/v1/runtime/approvals/deny`, { method: 'POST', headers: { 'content-type': 'application/json', ...host.headers }, body: JSON.stringify(upstreamBody), + signal: mutatingCallSignal(timeoutMs, res), }) - const body = await response.text() + const body = await readMutatingResponseBody(response) const draining = sessionDrainingFence(response, body) if (draining) throw draining - res.status(response.status).send(body) + sendUpstreamBody(res, response.status, body) } try { await attempt() @@ -701,6 +777,7 @@ export function createRpcRouter(): Router { host, claims: auth, rpcAccessToken, + deadlineMs: wakeDeadlineMs, attemptUpstream: attempt, respondLegacy: legacyError => respondUpstreamUnavailable(res, legacyError), }) @@ -744,11 +821,12 @@ export function createRpcRouter(): Router { res.status(400).json({ error: 'Invalid session pagination query' }) return } + const wakeDeadlineMs = Date.now() + config.wakeMaxHoldMs const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } const baseUrl = host.url.replace(/\/+$/, '') @@ -763,19 +841,21 @@ export function createRpcRouter(): Router { // host:session:read; wake capability rides on the token. A suspended // (network-down) or draining Host triggers a wake-and-hold instead of a // bare next(error)/passthrough. - const forwardSessionList = async () => { + const forwardSessionList = async (timeoutMs = config.upstreamTimeoutMs) => { const response = await fetch(upstreamUrl, { method: 'GET', headers: { ...host.headers }, - signal: AbortSignal.timeout(config.upstreamTimeoutMs), + signal: AbortSignal.timeout(timeoutMs), }) const body = await response.text() const draining = sessionDrainingFence(response, body) if (draining) throw draining - res - .status(response.status) - .type(response.headers.get('content-type') || 'application/json') - .send(body) + sendUpstreamBody( + res, + response.status, + body, + response.headers.get('content-type') || 'application/json' + ) } try { await forwardSessionList() @@ -793,6 +873,9 @@ export function createRpcRouter(): Router { host, claims: auth, rpcAccessToken, + deadlineMs: wakeDeadlineMs, + // idempotent GET: a re-issued read cannot duplicate a side effect + retryUntilDeadline: true, attemptUpstream: forwardSessionList, respondLegacy: legacyError => respondUpstreamUnavailable(res, legacyError), }) @@ -842,11 +925,12 @@ export function createRpcRouter(): Router { res.status(400).json({ error: 'Invalid session messages pagination query' }) return } + const wakeDeadlineMs = Date.now() + config.wakeMaxHoldMs const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } const baseUrl = host.url.replace(/\/+$/, '') @@ -864,19 +948,21 @@ export function createRpcRouter(): Router { `[RPC_PROXY] user=${auth.sub} host=${hostRef} method=get-session-messages agent=${agent} chatId=${chatId}` ) // Wake-eligible finite operation (§11.4): scope stays host:session:read. - const forwardTranscript = async () => { + const forwardTranscript = async (timeoutMs = config.upstreamTimeoutMs) => { const response = await fetch(upstreamUrl, { method: 'GET', headers: { ...host.headers }, - signal: AbortSignal.timeout(config.upstreamTimeoutMs), + signal: AbortSignal.timeout(timeoutMs), }) const body = await response.text() const draining = sessionDrainingFence(response, body) if (draining) throw draining - res - .status(response.status) - .type(response.headers.get('content-type') || 'application/json') - .send(body) + sendUpstreamBody( + res, + response.status, + body, + response.headers.get('content-type') || 'application/json' + ) } try { await forwardTranscript() @@ -892,6 +978,9 @@ export function createRpcRouter(): Router { host, claims: auth, rpcAccessToken, + deadlineMs: wakeDeadlineMs, + // idempotent GET: a re-issued read cannot duplicate a side effect + retryUntilDeadline: true, attemptUpstream: forwardTranscript, respondLegacy: legacyError => respondUpstreamUnavailable(res, legacyError), }) @@ -925,11 +1014,12 @@ export function createRpcRouter(): Router { res.status(400).json({ error: 'Invalid hostRef, agent, or chatId' }) return } + const wakeDeadlineMs = Date.now() + config.wakeMaxHoldMs const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } const baseUrl = host.url.replace(/\/+$/, '') @@ -937,22 +1027,24 @@ export function createRpcRouter(): Router { `[RPC_PROXY] user=${auth.sub} host=${hostRef} method=get-context-breakdown agent=${agent} chatId=${chatId}` ) // Wake-eligible finite operation (§11.4): scope stays host:session:read. - const attempt = async () => { + const attempt = async (timeoutMs = config.upstreamTimeoutMs) => { const response = await fetch( `${baseUrl}/v1/runtime/sessions/${encodeURIComponent(agent)}/${encodeURIComponent(chatId)}/context-breakdown`, { method: 'GET', headers: { ...host.headers }, - signal: AbortSignal.timeout(config.upstreamTimeoutMs), + signal: AbortSignal.timeout(timeoutMs), } ) const body = await response.text() const draining = sessionDrainingFence(response, body) if (draining) throw draining - res - .status(response.status) - .type(response.headers.get('content-type') || 'application/json') - .send(body) + sendUpstreamBody( + res, + response.status, + body, + response.headers.get('content-type') || 'application/json' + ) } try { await attempt() @@ -964,6 +1056,9 @@ export function createRpcRouter(): Router { host, claims: auth, rpcAccessToken, + deadlineMs: wakeDeadlineMs, + // idempotent GET: a re-issued read cannot duplicate a side effect + retryUntilDeadline: true, attemptUpstream: attempt, respondLegacy: legacyError => respondUpstreamUnavailable(res, legacyError), }) @@ -1009,8 +1104,8 @@ export function createRpcRouter(): Router { const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } const baseUrl = host.url.replace(/\/+$/, '') @@ -1038,14 +1133,18 @@ export function createRpcRouter(): Router { method: 'PATCH', headers: { 'content-type': 'application/json', ...host.headers }, body: JSON.stringify(req.body), - signal: AbortSignal.timeout(config.upstreamTimeoutMs), + // No wake retry re-issues a rename, so the timeout stays; a client + // that leaves still releases the upstream call. + signal: mutatingCallSignal(config.upstreamTimeoutMs, res), } ) const body = await response.text() - res - .status(response.status) - .type(response.headers.get('content-type') || 'application/json') - .send(body) + sendUpstreamBody( + res, + response.status, + body, + response.headers.get('content-type') || 'application/json' + ) } catch (error) { respondUpstreamUnavailable(res, error) } @@ -1072,11 +1171,12 @@ export function createRpcRouter(): Router { res.status(400).json({ error: 'hostRef is required' }) return } + const wakeDeadlineMs = Date.now() + config.wakeMaxHoldMs const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } const baseUrl = host.url.replace(/\/+$/, '') @@ -1086,19 +1186,21 @@ export function createRpcRouter(): Router { : `${baseUrl}/v1/runtime/models` console.info(`[RPC_PROXY] user=${auth.sub} host=${hostRef} method=list-models`) // Wake-eligible finite operation (§11.4): scope stays host:session:read. - const attempt = async () => { + const attempt = async (timeoutMs = config.upstreamTimeoutMs) => { const response = await fetch(upstreamUrl, { method: 'GET', headers: { ...host.headers }, - signal: AbortSignal.timeout(config.upstreamTimeoutMs), + signal: AbortSignal.timeout(timeoutMs), }) const body = await response.text() const draining = sessionDrainingFence(response, body) if (draining) throw draining - res - .status(response.status) - .type(response.headers.get('content-type') || 'application/json') - .send(body) + sendUpstreamBody( + res, + response.status, + body, + response.headers.get('content-type') || 'application/json' + ) } try { await attempt() @@ -1110,6 +1212,9 @@ export function createRpcRouter(): Router { host, claims: auth, rpcAccessToken, + deadlineMs: wakeDeadlineMs, + // idempotent GET: a re-issued read cannot duplicate a side effect + retryUntilDeadline: true, attemptUpstream: attempt, respondLegacy: legacyError => respondUpstreamUnavailable(res, legacyError), }) @@ -1137,11 +1242,12 @@ export function createRpcRouter(): Router { res.status(400).json({ error: 'hostRef is required' }) return } + const wakeDeadlineMs = Date.now() + config.wakeMaxHoldMs const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } const body = req.body as Record @@ -1152,19 +1258,22 @@ export function createRpcRouter(): Router { const baseUrl = host.url.replace(/\/+$/, '') console.info(`[RPC_PROXY] user=${auth.sub} host=${hostRef} method=set-model`) // Wake-eligible finite operation (§11.4): scope stays host:model:write. - const attempt = async () => { + const attempt = async (timeoutMs?: number) => { const response = await fetch(`${baseUrl}/v1/runtime/model`, { method: 'POST', headers: { 'content-type': 'application/json', ...host.headers }, body: JSON.stringify(body), + signal: mutatingCallSignal(timeoutMs, res), }) - const upstreamBody = await response.text() + const upstreamBody = await readMutatingResponseBody(response) const draining = sessionDrainingFence(response, upstreamBody) if (draining) throw draining - res - .status(response.status) - .type(response.headers.get('content-type') || 'application/json') - .send(upstreamBody) + sendUpstreamBody( + res, + response.status, + upstreamBody, + response.headers.get('content-type') || 'application/json' + ) } try { await attempt() @@ -1176,6 +1285,7 @@ export function createRpcRouter(): Router { host, claims: auth, rpcAccessToken, + deadlineMs: wakeDeadlineMs, attemptUpstream: attempt, respondLegacy: legacyError => respondUpstreamUnavailable(res, legacyError), }) @@ -1205,10 +1315,11 @@ export function createRpcRouter(): Router { return } + const wakeDeadlineMs = Date.now() + config.wakeMaxHoldMs const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { + if (isHostAccessDenied(host)) { res.status(404).json({ error: 'Host not found or not accessible' }) return } @@ -1216,8 +1327,8 @@ export function createRpcRouter(): Router { console.info( `[RPC_PROXY] user=${auth.sub} host=${hostRef} method=get-task-result taskId=${taskId}` ) - const attemptTaskResult = async () => { - const result = await forwardTaskResultFromHost(host, taskId) + const attemptTaskResult = async (timeoutMs = config.upstreamTimeoutMs) => { + const result = await forwardTaskResultFromHost(host, taskId, timeoutMs) if (!result) { res.status(404).json({ error: 'Task result not found' }) return @@ -1237,6 +1348,9 @@ export function createRpcRouter(): Router { host, claims: auth, rpcAccessToken, + deadlineMs: wakeDeadlineMs, + // idempotent GET: a re-issued read cannot duplicate a side effect + retryUntilDeadline: true, attemptUpstream: attemptTaskResult, respondLegacy: legacyError => respondUpstreamUnavailable(res, legacyError), }) @@ -1272,10 +1386,11 @@ export function createRpcRouter(): Router { return } + const wakeDeadlineMs = Date.now() + config.wakeMaxHoldMs const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { + if (isHostAccessDenied(host)) { res.status(404).json({ error: 'Host not found or not accessible' }) return } @@ -1284,13 +1399,20 @@ export function createRpcRouter(): Router { `[RPC_PROXY] user=${auth.sub} host=${hostRef} method=cancel-task taskId=${taskId}` ) - const attemptCancel = async () => { - const result = await forwardCancelToHost(host, taskId, auth.sub) + const attemptCancel = async (timeoutMs?: number) => { + const result = await forwardCancelToHost( + host, + taskId, + auth.sub, + mutatingCallSignal(timeoutMs, res) + ) if (result.body) { - res - .status(result.status) - .type(result.contentType || 'application/json') - .send(result.body) + sendUpstreamBody( + res, + result.status, + result.body, + result.contentType || 'application/json' + ) } else { res.status(result.status).end() } @@ -1308,6 +1430,7 @@ export function createRpcRouter(): Router { host, claims: auth, rpcAccessToken, + deadlineMs: wakeDeadlineMs, attemptUpstream: attemptCancel, respondLegacy: legacyError => respondUpstreamUnavailable(res, legacyError), }) @@ -1342,23 +1465,29 @@ export function createRpcRouter(): Router { res.status(400).json({ error: 'Invalid host reference' }) return } + const wakeDeadlineMs = Date.now() + config.wakeMaxHoldMs const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } const baseUrl = host.url.replace(/\/+$/, '') // Wake-eligible finite operation (§11.4): scope stays host:task:read. - const attempt = async () => { + // Unlike approve/deny/model this is a read-only list of small JSON: a + // timeout cannot leave a half-applied change, and without one a hung + // host would pin this socket. It therefore keeps a finite deadline on + // the first attempt as well. + const attempt = async (timeoutMs = config.upstreamTimeoutMs) => { const response = await fetch(`${baseUrl}/v1/runtime/artifacts`, { headers: { ...host.headers }, + signal: AbortSignal.timeout(timeoutMs), }) const body = await response.text() const draining = sessionDrainingFence(response, body) if (draining) throw draining - res.status(response.status).send(body) + sendUpstreamBody(res, response.status, body) } try { await attempt() @@ -1370,6 +1499,9 @@ export function createRpcRouter(): Router { host, claims: auth, rpcAccessToken, + deadlineMs: wakeDeadlineMs, + // idempotent GET: a re-issued read cannot duplicate a side effect + retryUntilDeadline: true, attemptUpstream: attempt, respondLegacy: legacyError => respondUpstreamUnavailable(res, legacyError), }) @@ -1409,48 +1541,75 @@ export function createRpcRouter(): Router { res.status(400).json({ error: 'Invalid filename' }) return } + const wakeDeadlineMs = Date.now() + config.wakeMaxHoldMs const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } const baseUrl = host.url.replace(/\/+$/, '') // Wake-eligible finite operation (§11.4): scope stays host:task:read. // The success path only commits (`res.send`) at the very end, so a // wake retry before that point is safe from duplicate delivery. - const attemptDownload = async () => { - const response = await fetch( - `${baseUrl}/v1/runtime/artifacts/${encodeURIComponent(filename)}/download`, - { - headers: { ...host.headers }, - } - ) - if (!response.ok) { - const errBody = await response.text() - const draining = sessionDrainingFence(response, errBody) - if (draining) throw draining - res.status(response.status).send(errBody) - return - } - const contentType = response.headers.get('content-type') || 'application/octet-stream' - const safeFilename = filename.replace(/[^a-zA-Z0-9_.-]/g, '_') - const redaction = response.headers.get('x-clerum-redaction') - let buffer: Buffer + const attemptDownload = async ( + timeoutMs = config.upstreamTimeoutMs, + holdDeadlineMs?: number + ) => { + const controller = new AbortController() + let timeout = setTimeout(() => controller.abort(), timeoutMs) try { - buffer = await readArtifactResponseBuffer(response) - } catch (error) { - if (error instanceof ProxyArtifactTooLargeError) { - res.status(413).json({ error: 'Artifact too large to download' }) + const response = await fetch( + `${baseUrl}/v1/runtime/artifacts/${encodeURIComponent(filename)}/download`, + { + headers: { ...host.headers }, + signal: controller.signal, + } + ) + // The upstream timeout protects connection and header latency. Once + // headers arrive, allow the separately bounded artifact transfer to + // finish; the small JSON-operation deadline must not truncate a + // valid large download while its body is streaming. On a wake retry + // the transfer is also bounded by what is left of the hold budget, + // so a stalled body cannot outlive the request deadline. + clearTimeout(timeout) + timeout = setTimeout( + () => controller.abort(), + holdDeadlineMs === undefined + ? config.artifactDownloadTimeoutMs + : Math.min( + config.artifactDownloadTimeoutMs, + Math.max(0, holdDeadlineMs - Date.now()) + ) + ) + if (!response.ok) { + const errBody = await response.text() + const draining = sessionDrainingFence(response, errBody) + if (draining) throw draining + sendUpstreamBody(res, response.status, errBody) return } - throw error + const contentType = response.headers.get('content-type') || 'application/octet-stream' + const safeFilename = filename.replace(/[^a-zA-Z0-9_.-]/g, '_') + const redaction = response.headers.get('x-clerum-redaction') + let buffer: Buffer + try { + buffer = await readArtifactResponseBuffer(response) + } catch (error) { + if (error instanceof ProxyArtifactTooLargeError) { + res.status(413).json({ error: 'Artifact too large to download' }) + return + } + throw error + } + res.setHeader('Content-Type', contentType) + res.setHeader('Content-Disposition', `attachment; filename="${safeFilename}"`) + if (redaction) res.setHeader('X-Clerum-Redaction', redaction) + res.send(buffer) + } finally { + clearTimeout(timeout) } - res.setHeader('Content-Type', contentType) - res.setHeader('Content-Disposition', `attachment; filename="${safeFilename}"`) - if (redaction) res.setHeader('X-Clerum-Redaction', redaction) - res.send(buffer) } try { await attemptDownload() @@ -1462,6 +1621,9 @@ export function createRpcRouter(): Router { host, claims: auth, rpcAccessToken, + deadlineMs: wakeDeadlineMs, + // idempotent GET: a re-issued read cannot duplicate a side effect + retryUntilDeadline: true, attemptUpstream: attemptDownload, respondLegacy: legacyError => respondUpstreamUnavailable(res, legacyError), }) @@ -1489,8 +1651,8 @@ export function createRpcRouter(): Router { const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } const limit = Number(req.query.limit || 50) @@ -1527,8 +1689,8 @@ export function createRpcRouter(): Router { const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } @@ -1568,8 +1730,8 @@ export function createRpcRouter(): Router { const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } diff --git a/rpc-proxy/src/routes/rpcHostActivityStream.ts b/rpc-proxy/src/routes/rpcHostActivityStream.ts index ce1264b059..abbb25f6c8 100644 --- a/rpc-proxy/src/routes/rpcHostActivityStream.ts +++ b/rpc-proxy/src/routes/rpcHostActivityStream.ts @@ -6,6 +6,7 @@ import { requireRpcAuth, requireScope, } from '../middleware/auth.js' +import { isHostAccessDenied, respondHostAccessDenied } from '../services/hostAccessDenial.js' import { resolveHostConnectionForUser } from '../services/mcpProxyService.js' const activeStreamCountsByUser = new Map() @@ -99,8 +100,8 @@ export function createRpcHostActivityStreamRouter(): Router { const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } diff --git a/rpc-proxy/src/routes/rpcHostProgressStream.ts b/rpc-proxy/src/routes/rpcHostProgressStream.ts index 6a22da640f..d0d5ca1505 100644 --- a/rpc-proxy/src/routes/rpcHostProgressStream.ts +++ b/rpc-proxy/src/routes/rpcHostProgressStream.ts @@ -6,6 +6,7 @@ import { requireRpcAuth, requireScope, } from '../middleware/auth.js' +import { isHostAccessDenied, respondHostAccessDenied } from '../services/hostAccessDenial.js' import { resolveHostConnectionForUser } from '../services/mcpProxyService.js' function isWildcardOrInvalidHostRef(hostRef: string): boolean { @@ -72,8 +73,8 @@ export function createRpcHostProgressStreamRouter(): Router { const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } diff --git a/rpc-proxy/src/routes/rpcHostStatusStream.ts b/rpc-proxy/src/routes/rpcHostStatusStream.ts index d3fd705442..3352425420 100644 --- a/rpc-proxy/src/routes/rpcHostStatusStream.ts +++ b/rpc-proxy/src/routes/rpcHostStatusStream.ts @@ -6,6 +6,7 @@ import { requireRpcAuth, requireScope, } from '../middleware/auth.js' +import { isHostAccessDenied, respondHostAccessDenied } from '../services/hostAccessDenial.js' import { UpstreamHostError, forwardHostStatus, @@ -95,8 +96,8 @@ export function createRpcHostStatusStreamRouter(): Router { const host = await resolveHostConnectionForUser(auth.sub, hostRef, rpcAccessToken, { teamId: auth.teamId, }) - if (!host) { - res.status(403).json({ error: 'Forbidden: user cannot access this host' }) + if (isHostAccessDenied(host)) { + respondHostAccessDenied(res, host) return } diff --git a/rpc-proxy/src/services/__tests__/controlApiRestService.hostAccessDenialBody.test.ts b/rpc-proxy/src/services/__tests__/controlApiRestService.hostAccessDenialBody.test.ts new file mode 100644 index 0000000000..3173ce1a19 --- /dev/null +++ b/rpc-proxy/src/services/__tests__/controlApiRestService.hostAccessDenialBody.test.ts @@ -0,0 +1,75 @@ +import { describe, expect, it } from 'vitest' +import { + ControlApiHostAccessRejectedError, + fetchHostConnectionFromControlApi, +} from '../controlApiRestService.js' + +// control-api's 403 for a Host-access denial carries its reason in a header and +// a fixed body the proxy never parses. The body must still be read to the end: +// an unread response body keeps the upstream connection checked out until the +// garbage collector finds it. The stand-in Response streams its body in chunks +// and reports when the last one was consumed, so "drained" is observed on the +// stream itself, not inferred from the mapped code. + +const REASON_HEADER = 'x-host-access-denial-reason' +const CHUNKS = 3 + +function streamedDenial(reason: string): { + response: Response + state: { pulls: number; drained: boolean } +} { + const state = { pulls: 0, drained: false } + const encoder = new TextEncoder() + const body = new ReadableStream({ + pull(controller) { + state.pulls += 1 + controller.enqueue(encoder.encode('{"error":"Forbidden"}')) + if (state.pulls >= CHUNKS) { + controller.close() + state.drained = true + } + }, + }) + return { + response: new Response(body, { status: 403, headers: { [REASON_HEADER]: reason } }), + state, + } +} + +const BINDING = { runId: 'run-1', sessionId: 'session-1', origin: 'direct_chat' } as const + +describe('fetchHostConnectionFromControlApi 403 body', () => { + it('reads the whole 403 body of a host lookup before returning the denial code', async () => { + const { response, state } = streamedDenial('team_membership_missing') + + const resolved = await fetchHostConnectionFromControlApi('user-1', 'host-1', 'token', { + fetchImpl: async () => response, + }) + + // Liveness witness: the reason header was read and mapped, so the 403 branch ran. + expect(resolved).toEqual({ denied: true, code: 'host_access_revoked' }) + expect(response.bodyUsed).toBe(true) + expect(state.drained).toBe(true) + expect(state.pulls).toBe(CHUNKS) + }) + + it('reads the whole 403 body of a direct-run lookup before throwing the rejection', async () => { + const { response, state } = streamedDenial('host_disabled') + + const error = await fetchHostConnectionFromControlApi('user-1', 'host-1', 'token', { + directRunBinding: BINDING, + fetchImpl: async () => response, + }).then( + () => { + throw new Error('expected the direct-run lookup to reject') + }, + (caught: unknown) => caught + ) + + expect(error).toBeInstanceOf(ControlApiHostAccessRejectedError) + expect((error as ControlApiHostAccessRejectedError).denialCode).toBe('host_access_denied') + expect(response.bodyUsed).toBe(true) + expect(state.drained).toBe(true) + expect(state.pulls).toBe(CHUNKS) + }) +}) diff --git a/rpc-proxy/src/services/__tests__/mcpHostRestService.test.ts b/rpc-proxy/src/services/__tests__/mcpHostRestService.test.ts index 795f85994d..d984099bc4 100644 --- a/rpc-proxy/src/services/__tests__/mcpHostRestService.test.ts +++ b/rpc-proxy/src/services/__tests__/mcpHostRestService.test.ts @@ -1,5 +1,196 @@ -import { describe, expect, it } from 'vitest' -import { __test__normalizeHostStatusPayload as normalize } from '../mcpHostRestService.js' +import { describe, expect, it, vi } from 'vitest' +import { config } from '../../config.js' +import { + forwardCancelToHost, + forwardHostActivity, + forwardHostHealth, + forwardHostMessageToHost, + forwardHostStatus, + forwardTaskResultFromHost, + __test__normalizeHostStatusPayload as normalize, +} from '../mcpHostRestService.js' +import { isUpstreamTimeoutError } from '../wakeAndHold.js' + +describe('host message availability probe', () => { + it('aborts at the requested short timeout with the standard abort reason', async () => { + vi.useFakeTimers() + const fetchMock = vi.spyOn(globalThis, 'fetch').mockImplementation( + (_input, init) => + new Promise((_resolve, reject) => { + init?.signal?.addEventListener('abort', () => reject(init.signal?.reason)) + }) + ) + try { + const pending = forwardHostMessageToHost( + { name: 'chatllm', url: 'http://chatllm:8080', headers: {} }, + { content: 'hello', hostRef: 'chatllm' }, + { timeoutMs: 750 } + ) + const rejection = expect(pending).rejects.toMatchObject({ + name: 'AbortError', + }) + await vi.advanceTimersByTimeAsync(750) + await rejection + expect(fetchMock).toHaveBeenCalledTimes(1) + } finally { + fetchMock.mockRestore() + vi.useRealTimers() + } + }) +}) + +// Every upstream forwarder aborts its own fetch when the deadline fires. What +// the caller sees is whatever `signal.reason` is: the app's 504 mapping and the +// wake coordinator both classify by error NAME (AbortError / TimeoutError), so +// an abort that carries a custom reason would be reported as a 500 / 502. +describe('upstream forwarders abort with the standard classified reason', () => { + const HOST = { name: 'chatllm', url: 'http://chatllm:8080', headers: {} } + + const PRODUCERS: Array<{ + label: string + timeoutMs: number + call: () => Promise + }> = [ + { + label: 'forwardHostMessageToHost', + timeoutMs: 750, + call: () => + forwardHostMessageToHost( + HOST, + { content: 'hello', hostRef: 'chatllm' }, + { timeoutMs: 750 } + ), + }, + { + label: 'forwardTaskResultFromHost', + timeoutMs: 640, + call: () => forwardTaskResultFromHost(HOST, 'task-1', 640), + }, + { + label: 'forwardHostStatus', + timeoutMs: config.upstreamTimeoutMs, + call: () => forwardHostStatus(HOST), + }, + { + label: 'forwardHostHealth', + timeoutMs: config.upstreamTimeoutMs, + call: () => forwardHostHealth(HOST), + }, + { + label: 'forwardHostActivity', + timeoutMs: config.upstreamTimeoutMs, + call: () => forwardHostActivity(HOST, 10), + }, + ] + + it.each(PRODUCERS)( + '$label rejects with an AbortError at its deadline, not earlier', + async ({ timeoutMs, call }) => { + vi.useFakeTimers() + let signal: AbortSignal | undefined + const fetchMock = vi.spyOn(globalThis, 'fetch').mockImplementation( + (_input, init) => + new Promise((_resolve, reject) => { + signal = init?.signal as AbortSignal + signal.addEventListener('abort', () => reject(signal!.reason)) + }) + ) + try { + const pending = call() + const rejection = pending.then( + () => { + throw new Error('expected the forwarder to reject at its deadline') + }, + (error: unknown) => error + ) + + // Liveness witness: the request is in flight and still un-aborted just + // before the deadline, so the rejection below is the deadline timer. + await vi.advanceTimersByTimeAsync(timeoutMs - 1) + expect(fetchMock).toHaveBeenCalledTimes(1) + expect(signal?.aborted).toBe(false) + + await vi.advanceTimersByTimeAsync(1) + const error = await rejection + + expect(signal?.aborted).toBe(true) + expect(error).toBeInstanceOf(Error) + expect((error as Error).name).toBe('AbortError') + expect(isUpstreamTimeoutError(error)).toBe(true) + } finally { + fetchMock.mockRestore() + vi.useRealTimers() + } + } + ) +}) + +// R3-L4 / R3-L6: task cancel is a mutating POST with no idempotency key, so it +// owns no deadline. The route passes one signal that carries the wake-hold +// deadline (retry only) and the client's disconnect. +describe('forwardCancelToHost aborts only through the caller signal', () => { + const HOST = { name: 'chatllm', url: 'http://chatllm:8080', headers: {} } + + it('stays in flight past the upstream timeout and rejects with the caller abort reason', async () => { + vi.useFakeTimers() + let signal: AbortSignal | undefined + const fetchMock = vi.spyOn(globalThis, 'fetch').mockImplementation( + (_input, init) => + new Promise((_resolve, reject) => { + signal = init?.signal as AbortSignal + signal.addEventListener('abort', () => reject(signal!.reason)) + }) + ) + try { + const controller = new AbortController() + const pending = forwardCancelToHost(HOST, 'task-1', 'user-1', controller.signal) + const rejection = pending.then( + () => { + throw new Error('expected the forwarder to reject when the caller aborts') + }, + (error: unknown) => error + ) + + // Liveness witness: the POST is in flight on the caller's own signal, and + // four upstream timeouts later nothing has aborted it. + await vi.advanceTimersByTimeAsync(config.upstreamTimeoutMs * 4) + expect(fetchMock).toHaveBeenCalledTimes(1) + expect(signal).toBe(controller.signal) + expect(signal?.aborted).toBe(false) + + const reason = new DOMException('The operation was aborted.', 'AbortError') + controller.abort(reason) + const error = await rejection + + expect(error).toBe(reason) + expect(isUpstreamTimeoutError(error)).toBe(true) + } finally { + fetchMock.mockRestore() + vi.useRealTimers() + } + }) + + // mcp-host checks cancel ownership against this userId, so a missing caller + // subject must fail here instead of being replaced by a stand-in identity. + it('forwards the caller subject and refuses an empty one before any request', async () => { + const fetchMock = vi + .spyOn(globalThis, 'fetch') + .mockImplementation(async () => new Response(null, { status: 204 })) + try { + await forwardCancelToHost(HOST, 'task-1', 'user-1', new AbortController().signal) + // Control and liveness witness: a real subject reaches the upstream body. + expect(fetchMock).toHaveBeenCalledTimes(1) + expect(JSON.parse(String(fetchMock.mock.calls[0]![1]?.body))).toEqual({ userId: 'user-1' }) + + await expect( + forwardCancelToHost(HOST, 'task-1', '', new AbortController().signal) + ).rejects.toThrow('task cancel requires the caller subject') + expect(fetchMock).toHaveBeenCalledTimes(1) + } finally { + fetchMock.mockRestore() + } + }) +}) const baseUpstream = { agent: { diff --git a/rpc-proxy/src/services/__tests__/wakeAndHold.test.ts b/rpc-proxy/src/services/__tests__/wakeAndHold.test.ts index 0f5c887c2f..6739003fd9 100644 --- a/rpc-proxy/src/services/__tests__/wakeAndHold.test.ts +++ b/rpc-proxy/src/services/__tests__/wakeAndHold.test.ts @@ -1,7 +1,9 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import type { Response as ExpressResponse } from 'express' +import { config } from '../../config.js' import type { ResolvedServerConnection, RpcAccessClaims, RpcScope } from '../../types.js' import type { HostWakeApiResponse } from '../controlApiRestService.js' +import { UpstreamBodyReadError } from '../upstreamBody.js' import { WakeAndHoldCoordinator, type WakeCoordinatorDeps, @@ -425,6 +427,20 @@ describe('host error classification', () => { }) expect(isHostDownNetworkError(upstream)).toBe(false) }) + + // ADV-HCC-1: the wrapper's message embeds the body failure's text, which can + // itself read like a down host ("read ECONNRESET", "socket hang up"). The + // headers arrived, so it is still never a down-host signal. + it('never classifies a body failure after the headers as a down host', () => { + const bodyFailure = new Error('read ECONNRESET') + // Witness: the same failure raised before fetch() resolved IS a down host. + expect(isHostDownNetworkError(bodyFailure)).toBe(true) + + expect(isHostDownNetworkError(new UpstreamBodyReadError(bodyFailure))).toBe(false) + expect(isHostDownNetworkError(new UpstreamBodyReadError(new Error('socket hang up')))).toBe( + false + ) + }) }) describe('post-resolution re-forward hardening (respondWithWakeAndHold)', () => { @@ -474,11 +490,12 @@ describe('post-resolution re-forward hardening (respondWithWakeAndHold)', () => function respondOptions( coordinator: WakeAndHoldCoordinator, res: FakeRes, - attemptUpstream: () => Promise, + attemptUpstream: (timeoutMs: number, deadlineMs: number) => Promise, overrides?: { hostRef?: string host?: ResolvedServerConnection respondLegacy?: (error: unknown) => void + retryUntilDeadline?: boolean } ) { const hostRef = overrides?.hostRef ?? 'chatllm' @@ -489,6 +506,7 @@ describe('post-resolution re-forward hardening (respondWithWakeAndHold)', () => claims: holdClaims(hostRef, Date.now() + 3_600_000), rpcAccessToken: 'rpc-token', attemptUpstream, + retryUntilDeadline: overrides?.retryUntilDeadline, respondLegacy: overrides?.respondLegacy ?? ((_error: unknown) => { @@ -559,7 +577,7 @@ describe('post-resolution re-forward hardening (respondWithWakeAndHold)', () => expect(requestWake.mock.calls.length).toBe(wakeCalls) }) - it('(c) an unresolved proceed still walks the short retry schedule exactly as today', async () => { + it('(c) an unresolved proceed retries availability until admission succeeds', async () => { const { coordinator, requestWake } = makeCoordinator() requestWake.mockResolvedValue({ kind: 'active', wakeGeneration: null }) @@ -571,15 +589,193 @@ describe('post-resolution re-forward hardening (respondWithWakeAndHold)', () => .mockImplementationOnce(async () => { res.status(200).json({ success: true, taskId: 't-3' }) }) - const pending = respondWithWakeAndHold(respondOptions(coordinator, res, attemptUpstream)) + const pending = respondWithWakeAndHold( + respondOptions(coordinator, res, attemptUpstream, { retryUntilDeadline: true }) + ) - await vi.advanceTimersByTimeAsync(1_250) // 0ms + 250ms + 1000ms schedule + await vi.advanceTimersByTimeAsync(750) // 0ms + 250ms + 750ms attempts await pending expect(attemptUpstream).toHaveBeenCalledTimes(3) // pre-resolution retries are the feature expect(res.statusCode).toBe(200) expect(coordinator.trackedCoordinationCount()).toBe(0) }) + it('active wake response keeps trying until late upstream admission succeeds within the hold', async () => { + const { coordinator, requestWake, probeReady } = makeCoordinator() + requestWake.mockResolvedValue({ kind: 'active', wakeGeneration: null }) + + const res = makeRes() + let upstreamAdmits = false + const attemptUpstream = vi.fn(async () => { + if (!upstreamAdmits) throw upstreamDrainingError() + res.status(200).json({ success: true, taskId: 't-late-admission' }) + }) + const pending = respondWithWakeAndHold( + respondOptions(coordinator, res, attemptUpstream, { retryUntilDeadline: true }) + ) + + await vi.advanceTimersByTimeAsync(1_250) + expect(attemptUpstream).toHaveBeenCalledTimes(3) + expect(res.headersSent).toBe(false) + expect(probeReady).not.toHaveBeenCalled() + expect(coordinator.trackedCoordinationCount()).toBe(0) + + upstreamAdmits = true + await vi.advanceTimersByTimeAsync(500) + await pending + expect(attemptUpstream).toHaveBeenCalledTimes(4) + expect(res.statusCode).toBe(200) + await vi.advanceTimersByTimeAsync(MAX_HOLD_MS * 3) + expect(attemptUpstream).toHaveBeenCalledTimes(4) + }) + + it('returns host_waking at the absolute deadline without a later admission attempt', async () => { + const { coordinator, requestWake } = makeCoordinator() + requestWake.mockResolvedValue({ kind: 'active', wakeGeneration: null }) + const res = makeRes() + const attemptUpstream = vi.fn().mockRejectedValue(upstreamDrainingError()) + const pending = respondWithWakeAndHold({ + ...respondOptions(coordinator, res, attemptUpstream, { retryUntilDeadline: true }), + deadlineMs: Date.now() + 2_500, + }) + + // Attempts at 0, 250 and 750 ms; at 1750 ms only 750 ms remain, below the + // admission floor, so no fourth attempt is made. + await vi.advanceTimersByTimeAsync(2_500) + await pending + expect(attemptUpstream).toHaveBeenCalledTimes(3) + expect(res.statusCode).toBe(503) + expect(res.body).toMatchObject({ code: 'host_waking', hostRef: 'chatllm' }) + await vi.advanceTimersByTimeAsync(MAX_HOLD_MS) + expect(attemptUpstream).toHaveBeenCalledTimes(3) + }) + + it('passes each admission retry only the remaining hold budget', async () => { + const { coordinator, requestWake } = makeCoordinator() + requestWake.mockResolvedValue({ kind: 'active', wakeGeneration: null }) + const res = makeRes() + const attemptUpstream = vi.fn(async (_timeoutMs: number) => { + throw upstreamDrainingError() + }) + const pending = respondWithWakeAndHold({ + ...respondOptions(coordinator, res, attemptUpstream, { retryUntilDeadline: true }), + deadlineMs: Date.now() + 1_500, + }) + + await vi.advanceTimersByTimeAsync(1_500) + await pending + + expect(attemptUpstream.mock.calls.map(([timeoutMs]) => timeoutMs)).toEqual([1_500, 1_250]) + expect(res.statusCode).toBe(503) + }) + + it('caps an admission retry timeout at the configured upstream timeout', async () => { + const { coordinator, requestWake } = makeCoordinator() + requestWake.mockResolvedValue({ kind: 'active', wakeGeneration: null }) + const res = makeRes() + const attemptUpstream = vi.fn(async (_timeoutMs: number) => { + res.status(200).json({ success: true, taskId: 't-capped' }) + }) + const deadlineMs = Date.now() + config.upstreamTimeoutMs + 5_000 + const pending = respondWithWakeAndHold({ + ...respondOptions(coordinator, res, attemptUpstream), + deadlineMs, + }) + + await pending + + expect(attemptUpstream).toHaveBeenCalledWith(config.upstreamTimeoutMs, deadlineMs) + expect(res.statusCode).toBe(200) + }) + + // R1-M13: the retry floor sat at 100 ms, below the upstream's admission + // latency, so a retry with a sliver of budget timed out AFTER the upstream + // applied the request (applied-once + 504). Below the floor the caller gets a + // retryable 503 instead. + it('skips an admission retry when less than the 1 s floor remains', async () => { + const { coordinator, requestWake } = makeCoordinator() + requestWake.mockResolvedValue({ kind: 'active', wakeGeneration: null }) + const res = makeRes() + const attemptUpstream = vi.fn(async (_timeoutMs: number) => {}) + const pending = respondWithWakeAndHold({ + ...respondOptions(coordinator, res, attemptUpstream), + deadlineMs: Date.now() + 999, + }) + + await pending + + expect(attemptUpstream).not.toHaveBeenCalled() + expect(res.statusCode).toBe(503) + expect(res.body).toMatchObject({ code: 'host_waking' }) + }) + + it('lets a deliberately short upstream timeout retry below the 1 s floor', async () => { + const { coordinator, requestWake } = makeCoordinator() + requestWake.mockResolvedValue({ kind: 'active', wakeGeneration: null }) + const res = makeRes() + const attemptUpstream = vi.fn(async (_timeoutMs: number) => { + res.status(200).json({ ok: true }) + }) + const previous = config.upstreamTimeoutMs + config.upstreamTimeoutMs = 300 + try { + const pending = respondWithWakeAndHold({ + ...respondOptions(coordinator, res, attemptUpstream), + deadlineMs: Date.now() + 400, + }) + await pending + } finally { + config.upstreamTimeoutMs = previous + } + + expect(attemptUpstream).toHaveBeenCalledTimes(1) + expect(attemptUpstream.mock.calls[0]![0]).toBe(300) + expect(res.statusCode).toBe(200) + }) + + // R1-M12: without an idempotency key every extra POST is a duplicate side + // effect, so a route that does not opt in gets ONE retry after the wake. + it('re-issues a non-idempotent request at most once, then answers host_waking', async () => { + const { coordinator, requestWake } = makeCoordinator() + requestWake.mockResolvedValue({ kind: 'active', wakeGeneration: null }) + const res = makeRes() + const attemptUpstream = vi.fn(async (_timeoutMs: number) => { + throw upstreamDrainingError() + }) + const pending = respondWithWakeAndHold(respondOptions(coordinator, res, attemptUpstream)) + + await vi.advanceTimersByTimeAsync(MAX_HOLD_MS) + await pending + + expect(attemptUpstream).toHaveBeenCalledTimes(1) + expect(res.statusCode).toBe(503) + expect(res.body).toMatchObject({ code: 'host_waking', hostRef: 'chatllm' }) + await vi.advanceTimersByTimeAsync(MAX_HOLD_MS * 3) + expect(attemptUpstream).toHaveBeenCalledTimes(1) + }) + + it('opting in with retryUntilDeadline keeps re-issuing until the host admits the request', async () => { + const { coordinator, requestWake } = makeCoordinator() + requestWake.mockResolvedValue({ kind: 'active', wakeGeneration: null }) + const res = makeRes() + const attemptUpstream = vi + .fn() + .mockRejectedValueOnce(upstreamDrainingError()) + .mockRejectedValueOnce(upstreamDrainingError()) + .mockImplementationOnce(async () => { + res.status(200).json({ ok: true }) + }) + const pending = respondWithWakeAndHold( + respondOptions(coordinator, res, attemptUpstream, { retryUntilDeadline: true }) + ) + + await vi.advanceTimersByTimeAsync(750) + await pending + + expect(attemptUpstream).toHaveBeenCalledTimes(3) + expect(res.statusCode).toBe(200) + }) + it('(d) two concurrent holds for different hosts do not cross-cancel each other', async () => { const { coordinator, requestWake, probeReady } = makeCoordinator() requestWake.mockResolvedValue({ kind: 'wake-requested', wakeGeneration: 1 }) @@ -674,6 +870,31 @@ describe('post-resolution re-forward hardening (respondWithWakeAndHold)', () => expect(coordinator.trackedCoordinationCount()).toBe(0) }) + it('suppresses a hold outcome after another path commits the response', async () => { + const { coordinator, requestWake } = makeCoordinator() + let finishWake!: (value: HostWakeApiResponse) => void + requestWake.mockImplementation( + () => + new Promise(resolve => { + finishWake = resolve + }) + ) + const res = makeRes() + const attemptUpstream = vi.fn(async () => {}) + const respondLegacy = vi.fn() + const pending = respondWithWakeAndHold( + respondOptions(coordinator, res, attemptUpstream, { respondLegacy }) + ) + + res.status(200).json({ success: true }) + finishWake({ kind: 'active', wakeGeneration: null }) + await pending + expect(attemptUpstream).not.toHaveBeenCalled() + expect(respondLegacy).not.toHaveBeenCalled() + expect(res.statusCode).toBe(200) + expect(coordinator.trackedCoordinationCount()).toBe(0) + }) + it('a readiness probe landing after resolution no-ops loudly instead of re-settling', async () => { const { coordinator, requestWake, probeReady } = makeCoordinator() requestWake diff --git a/rpc-proxy/src/services/controlApiRestService.ts b/rpc-proxy/src/services/controlApiRestService.ts index 65d772ac3d..961d798f48 100644 --- a/rpc-proxy/src/services/controlApiRestService.ts +++ b/rpc-proxy/src/services/controlApiRestService.ts @@ -1,5 +1,11 @@ import { config } from '../config.js' import { ResolvedServerConnection } from '../types.js' +import { + type HostAccessDenial, + type HostAccessDenialCode, + hostAccessDenialCodeForReason, + hostAccessDenied, +} from './hostAccessDenial.js' export type UserAllowedServers = { userId: string @@ -42,9 +48,19 @@ export type DirectRunBindingRequest = { } export class ControlApiHostAccessRejectedError extends Error { - constructor(readonly status: number) { + /** + * Set for a 403 only: whether control-api's denial reason proves the access + * was removed from the user. Absent for 401/409, which are not authorization + * denials of the Host. + */ + readonly denialCode: HostAccessDenialCode | null + constructor( + readonly status: number, + denialCode: HostAccessDenialCode | null = null + ) { super(`Control API rejected host access (${status})`) this.name = 'ControlApiHostAccessRejectedError' + this.denialCode = denialCode } } @@ -207,7 +223,7 @@ export async function fetchHostConnectionFromControlApi( directRunBinding?: DirectRunBindingRequest fetchImpl?: typeof fetch } = {} -): Promise { +): Promise { const directRunBinding = options.directRunBinding const response = await (options.fetchImpl ?? fetch)( `${controlApiBaseUrl()}/rpc/access/users/${encodeURIComponent(userId)}/mcp-hosts/${encodeURIComponent(hostRef)}`, @@ -222,10 +238,17 @@ export async function fetchHostConnectionFromControlApi( } ) - if (!directRunBinding && (response.status === 403 || response.status === 404)) { - return null + if (!directRunBinding && response.status === 403) { + return hostAccessDenied(await readHostAccessDenialCode(response)) } - if (response.status === 401 || response.status === 403 || response.status === 409) { + if (!directRunBinding && response.status === 404) { + await drainBody(response) + return hostAccessDenied('host_access_denied') + } + if (response.status === 403) { + throw new ControlApiHostAccessRejectedError(403, await readHostAccessDenialCode(response)) + } + if (response.status === 401 || response.status === 409) { await drainBody(response) throw new ControlApiHostAccessRejectedError(response.status) } @@ -236,12 +259,12 @@ export async function fetchHostConnectionFromControlApi( const parsed = (await response.json()) as Partial if (parsed.userId !== userId) { - return null + return hostAccessDenied('host_access_denied') } const url = typeof parsed.url === 'string' ? parsed.url.trim() : '' const resolvedHostRef = typeof parsed.hostRef === 'string' ? parsed.hostRef.trim() : '' if (!url || !resolvedHostRef || resolvedHostRef !== hostRef) { - return null + return hostAccessDenied('host_access_denied') } const attributionBindingStatus = directRunBinding ? parsed.bindingStatus : undefined if ( @@ -277,6 +300,18 @@ export type HostWakeApiResponse = | { kind: 'rate-limited'; retryAfterSeconds: number } | { kind: 'auth'; status: number } +/** + * control-api reports the denial reason in the `x-host-access-denial-reason` + * response header; its 403 body is the fixed `{"error":"Forbidden"}` and is never + * parsed. An absent or unknown header yields the non-revoking code: an + * unreadable reason can never prove the access was removed. + */ +async function readHostAccessDenialCode(response: Response): Promise { + const reason = response.headers.get('x-host-access-denial-reason') + await drainBody(response) + return hostAccessDenialCodeForReason(reason) +} + async function drainBody(response: Response): Promise { try { await response.arrayBuffer() diff --git a/rpc-proxy/src/services/hostAccessDenial.ts b/rpc-proxy/src/services/hostAccessDenial.ts new file mode 100644 index 0000000000..8166349eac --- /dev/null +++ b/rpc-proxy/src/services/hostAccessDenial.ts @@ -0,0 +1,88 @@ +import type { Response as ExpressResponse } from 'express' + +/** + * rpc-proxy's OWN authorization denial for a Host (never a passthrough of an + * mcp-host 403 body). Status stays 403 and `error` keeps its text; `code` is the + * machine field Desktop uses to tell "access was removed from this user" from + * "denied for a reason that does not prove revocation". + */ +export const HOST_ACCESS_DENIED_ERROR = 'Forbidden: user cannot access this host' + +export type HostAccessDenialCode = 'host_access_revoked' | 'host_access_denied' + +export type HostAccessDenial = { denied: true; code: HostAccessDenialCode } + +/** + * control-api denial reasons (control-api `RpcHostAccessDenialReason`) that + * prove the access was actually removed from the user. Every other reason + * (`host_disabled`, `host_missing`, `host_claim_missing`, `subject_mismatch`, + * an unknown or absent reason) is a denial that does not prove revocation. + * `subject_mismatch` in particular: rpc-proxy calls + * `/rpc/access/users/:userId/...` with the token's own `sub` as `userId` + * (control-api `routes/rpc-access/users.ts`), so a mismatch is a request-shape + * error, never a grant that was removed. + */ +const REVOKED_CONTROL_API_REASONS: ReadonlySet = new Set([ + 'team_membership_missing', + 'directory_grant_missing', +]) + +export function hostAccessDenialCodeForReason(reason: unknown): HostAccessDenialCode { + return typeof reason === 'string' && REVOKED_CONTROL_API_REASONS.has(reason) + ? 'host_access_revoked' + : 'host_access_denied' +} + +export function hostAccessDenied(code: HostAccessDenialCode): HostAccessDenial { + return { denied: true, code } +} + +/** Narrows `resolveHostConnectionForUser`'s result: true means "respond 403". */ +export function isHostAccessDenied( + resolved: T | HostAccessDenial +): resolved is HostAccessDenial { + return (resolved as Partial).denied === true +} + +const RESERVED_HOST_ACCESS_CODES: ReadonlySet = new Set([ + 'host_access_revoked', + 'host_access_denied', +]) + +/** + * The body rpc-proxy may relay for an upstream (mcp-host) response. Desktop + * trusts the `code` of a 403 JSON body as rpc-proxy's own authorization verdict, + * so a passthrough 403 must never carry a reserved Host-access code: an mcp-host + * could otherwise forge a "confirmed revocation" that hides the agent. Such a + * `code` field is dropped; the status and every other field are kept. A non-403 + * status, a non-JSON body and a JSON value that is not an object pass unchanged + * (Desktop reads the code only from a parsed JSON object of a 403). + */ +export function withoutReservedHostAccessCode(status: number, body: string): string { + if (status !== 403) return body + let parsed: unknown + try { + parsed = JSON.parse(body) + } catch { + // Not JSON: Desktop cannot read a `code` from it, so it is relayed as is. + return body + } + if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) return body + const { code, ...rest } = parsed as Record + if (!RESERVED_HOST_ACCESS_CODES.has(code)) return body + console.warn( + `[RPC_PROXY] dropped reserved host-access code from an upstream 403 body code=${String(code)}` + ) + return JSON.stringify(rest) +} + +/** The single writer of rpc-proxy's own Host authorization 403. */ +export function respondHostAccessDenied( + res: ExpressResponse, + code: HostAccessDenialCode | HostAccessDenial +): void { + res.status(403).json({ + error: HOST_ACCESS_DENIED_ERROR, + code: typeof code === 'string' ? code : code.code, + }) +} diff --git a/rpc-proxy/src/services/mcpHostRestService.ts b/rpc-proxy/src/services/mcpHostRestService.ts index 0f68856555..d4b6623f97 100644 --- a/rpc-proxy/src/services/mcpHostRestService.ts +++ b/rpc-proxy/src/services/mcpHostRestService.ts @@ -1,6 +1,7 @@ import { config } from '../config.js' import type { TraceContextV1 } from '../traceContext.js' import { ResolvedServerConnection } from '../types.js' +import { readMutatingResponseBody } from './upstreamBody.js' /** * Per-MCP-server health row as emitted by mcp-host and forwarded by rpc-proxy. @@ -135,7 +136,7 @@ export type HostRuntimeMessageRequest = { export async function forwardHostMessageToHost( host: ResolvedServerConnection, message: HostRuntimeMessageRequest, - options?: { async?: boolean } + options?: { async?: boolean; timeoutMs?: number } ): Promise> { const payload = { ...message, @@ -144,7 +145,9 @@ export async function forwardHostMessageToHost( const baseUrl = host.url.replace(/\/+$/, '') const query = options?.async ? '?async=true' : '' const abortController = new AbortController() - const timeout = setTimeout(() => abortController.abort(), config.upstreamTimeoutMs) + const timeout = setTimeout(() => { + abortController.abort() + }, options?.timeoutMs ?? config.upstreamTimeoutMs) try { const response = await fetch(`${baseUrl}/v1/runtime/messages${query}`, { method: 'POST', @@ -199,11 +202,12 @@ export async function forwardHostMessageToHost( export async function forwardTaskResultFromHost( host: ResolvedServerConnection, - taskId: string + taskId: string, + timeoutMs = config.upstreamTimeoutMs ): Promise | null> { const baseUrl = host.url.replace(/\/+$/, '') const abortController = new AbortController() - const timeout = setTimeout(() => abortController.abort(), config.upstreamTimeoutMs) + const timeout = setTimeout(() => abortController.abort(), timeoutMs) try { const response = await fetch( `${baseUrl}/v1/runtime/tasks/${encodeURIComponent(taskId)}/result`, @@ -217,8 +221,10 @@ export async function forwardTaskResultFromHost( } ) if (response.status === 404) return null - if (!response.ok) throw new UpstreamHostError(response.status, '') const rawBody = await response.text() + // The body carries mcp-host's `host_draining` fence, which is what makes a + // draining 503 wake-eligible (isHostDrainingError reads bodySnippet). + if (!response.ok) throw new UpstreamHostError(response.status, rawBody.slice(0, 300)) if (!rawBody.trim()) return null try { const parsed = JSON.parse(rawBody) as unknown @@ -239,38 +245,38 @@ export type CancelUpstreamResult = { contentType: string | null } +/** + * Task cancel is a mutating POST without an idempotency key, so it owns no + * deadline: the caller's `signal` carries the client disconnect and, on a + * wake-and-hold retry only, the hold deadline (rpc.ts `mutatingCallSignal`). + */ export async function forwardCancelToHost( host: ResolvedServerConnection, taskId: string, - userId?: string + userId: string, + signal: AbortSignal ): Promise { + // mcp-host applies its ownership check to this userId, so it must be the + // caller's own subject; a stand-in identity would be checked instead. + if (!userId) throw new Error('task cancel requires the caller subject') const baseUrl = host.url.replace(/\/+$/, '') - const abortController = new AbortController() - const timeout = setTimeout(() => abortController.abort(), config.upstreamTimeoutMs) - // Mirrors the approve/deny pattern: always forward userId so mcp-host can - // apply the ownership check. Falls back to 'desktop-app' when absent. - const upstreamBody = { userId: userId || 'desktop-app' } - try { - const response = await fetch( - `${baseUrl}/v1/runtime/tasks/${encodeURIComponent(taskId)}/cancel`, - { - method: 'POST', - headers: { - 'content-type': 'application/json', - ...host.headers, - }, - body: JSON.stringify(upstreamBody), - signal: abortController.signal, - } - ) - const body = await response.text() - return { - status: response.status, - body, - contentType: response.headers.get('content-type'), - } - } finally { - clearTimeout(timeout) + const upstreamBody = { userId } + const response = await fetch(`${baseUrl}/v1/runtime/tasks/${encodeURIComponent(taskId)}/cancel`, { + method: 'POST', + headers: { + 'content-type': 'application/json', + ...host.headers, + }, + body: JSON.stringify(upstreamBody), + signal, + }) + // Cancel is a mutating POST without an idempotency key: a body failure + // after the headers must not re-issue it through the wake path. + const body = await readMutatingResponseBody(response) + return { + status: response.status, + body, + contentType: response.headers.get('content-type'), } } diff --git a/rpc-proxy/src/services/mcpProxyService.ts b/rpc-proxy/src/services/mcpProxyService.ts index 43551cffb8..7cf8595b1a 100644 --- a/rpc-proxy/src/services/mcpProxyService.ts +++ b/rpc-proxy/src/services/mcpProxyService.ts @@ -6,6 +6,7 @@ import { fetchHostConnectionFromControlApi, fetchUserAllowedServersFromControlApi, } from './controlApiRestService.js' +import { type HostAccessDenial, isHostAccessDenied } from './hostAccessDenial.js' export { validateRpcRequest, forwardRpcToServer } from './mcpRpcService.js' export { @@ -83,11 +84,11 @@ export async function resolveHostConnectionForUser( requestId?: string directRunBinding?: DirectRunBindingRequest } -): Promise { +): Promise { const host = await fetchHostConnectionFromControlApi(userId, hostRef, rpcAccessToken, { directRunBinding: edgeContext?.directRunBinding, }) - if (!host) return null + if (isHostAccessDenied(host)) return host const headers: Record = { ...host.headers, diff --git a/rpc-proxy/src/services/upstreamBody.ts b/rpc-proxy/src/services/upstreamBody.ts new file mode 100644 index 0000000000..23efe098c4 --- /dev/null +++ b/rpc-proxy/src/services/upstreamBody.ts @@ -0,0 +1,32 @@ +/** + * A failure while reading an upstream response body AFTER its headers arrived. + * The upstream has already received the request (and for a mutating call may + * have applied it), so this is never a down-host signal: the wake-and-hold + * classifier excludes it by name and a mutating POST is not re-issued. The + * original failure is kept as `cause` (a timeout there still maps to 504). + */ +export class UpstreamBodyReadError extends Error { + constructor(cause: unknown) { + super( + `Upstream response body read failed after headers: ${ + cause instanceof Error ? cause.message : String(cause) + }`, + { cause } + ) + this.name = 'UpstreamBodyReadError' + } +} + +/** + * Reads the body of a mutating upstream call (approve, deny, set-model, + * cancel). Only an error raised before `fetch()` resolved may trigger a wake + * and a re-issued POST; a socket that dies mid-body surfaces as + * {@link UpstreamBodyReadError} instead. + */ +export async function readMutatingResponseBody(response: Response): Promise { + try { + return await response.text() + } catch (error) { + throw new UpstreamBodyReadError(error) + } +} diff --git a/rpc-proxy/src/services/wakeAndHold.ts b/rpc-proxy/src/services/wakeAndHold.ts index d6d81c8446..914dde3dbb 100644 --- a/rpc-proxy/src/services/wakeAndHold.ts +++ b/rpc-proxy/src/services/wakeAndHold.ts @@ -4,6 +4,11 @@ import type { ResolvedServerConnection, RpcAccessClaims } from '../types.js' import { type HostWakeApiResponse, requestHostWakeFromControlApi } from './controlApiRestService.js' import { forwardHostHealth } from './mcpHostRestService.js' +/** Strip control characters and newlines from user-derived hostRef before log interpolation. */ +function sanitizeHostRefForLog(hostRef: string): string { + return hostRef.replace(/[\r\n\t\x00-\x1f\x7f]/g, '') +} + /** * Stateless wake-and-hold (Stage 5, Issue #791 §11). * @@ -39,8 +44,15 @@ const ENTRY_TTL_MARGIN_MS = 5_000 */ const MAX_TRACKED_WAKE_COORDINATIONS = 1_000 const WAKE_SCOPE = 'host:wake:write' as const -/** Short upstream retry schedule after a wake reports the host is up. */ -const PROCEED_RETRY_DELAYS_MS = [0, 250, 1_000] +const ADMISSION_RETRY_DELAYS_MS = [250, 500, 1_000] +/** + * An admission retry with less budget than this cannot outlast the upstream's + * own admission latency: it would time out AFTER the upstream applied the + * request, turning an applied-once operation into a 504. Below it the request + * answers a retryable 503 host_waking instead. Bounded by `upstreamTimeoutMs` + * so a deliberately short upstream timeout still gets a retry. + */ +const MIN_ADMISSION_RETRY_TIMEOUT_MS = 1_000 export type WakeHoldOutcome = /** Host is (or just became) reachable — re-issue the upstream request now. */ @@ -110,6 +122,8 @@ export type WakeHoldParams = { claims: RpcAccessClaims /** Raw bearer forwarded to the wake plane when this caller is wake-capable. */ rpcAccessToken: string + /** Optional request deadline, shared with upstream admission retries. */ + deadlineMs?: number } /** @@ -181,9 +195,23 @@ export class WakeAndHoldCoordinator { lastKnownState: 'unknown', } } - const holdBudgetMs = Math.min(this.deps.maxHoldMs, tokenBudgetMs) + const holdBudgetMs = Math.min( + this.deps.maxHoldMs, + tokenBudgetMs, + params.deadlineMs === undefined ? Infinity : Math.max(0, params.deadlineMs - now) + ) const deadlineReason = - holdBudgetMs < this.deps.maxHoldMs ? 'token-expiring' : 'max-hold-exceeded' + tokenBudgetMs <= Math.min(this.deps.maxHoldMs, (params.deadlineMs ?? Infinity) - now) + ? 'token-expiring' + : 'max-hold-exceeded' + if (holdBudgetMs <= 0) { + return { + kind: 'waking', + retryAfterMs: this.deps.pollMs, + reason: deadlineReason, + lastKnownState: 'unknown', + } + } const key = wakeCoordinationKey(params.claims, params.hostRef) const wakeCapable = isWakeCapable(params.claims, params.hostRef, now) @@ -330,6 +358,14 @@ export class WakeAndHoldCoordinator { // bounded by the per-waiter deadlines and the entry TTL. return } + await this.processWakeResponse(entry, response, options) + } + + private async processWakeResponse( + entry: HostWakeEntry, + response: HostWakeApiResponse, + options: { initial: boolean } + ): Promise { if (entry.settled) { console.debug( `[RPC_PROXY] late wake-hold artifact ignored (already resolved) host=${entry.hostRef} artifact=wake-response kind=${response.kind}` @@ -491,20 +527,42 @@ export function isHostDrainingError(error: unknown): boolean { return upstream.status === 503 && String(upstream.bodySnippet || '').includes('host_draining') } +/** + * undici's own header/body timers (300 s defaults). They fire on a connection + * the host accepted, so they prove a slow host, not a down one: a timeout, and + * never a reason to wake and re-issue the request. + */ +const UNDICI_TIMEOUT_CODES: ReadonlySet = new Set([ + 'UND_ERR_HEADERS_TIMEOUT', + 'UND_ERR_BODY_TIMEOUT', +]) + /** Fetch uses AbortError for controller aborts and TimeoutError for - * AbortSignal.timeout(). Both represent the same sanitized 504 boundary. */ + * AbortSignal.timeout(); undici's own header/body timers reject with their + * code on `cause`. All represent the same sanitized 504 boundary. An + * UpstreamBodyReadError is a timeout when the body read it wraps was one. */ export function isUpstreamTimeoutError(error: unknown): boolean { - return error instanceof Error && (error.name === 'AbortError' || error.name === 'TimeoutError') + if (!(error instanceof Error)) return false + if (error.name === 'AbortError' || error.name === 'TimeoutError') return true + if (error.name === 'UpstreamBodyReadError') return isUpstreamTimeoutError(error.cause) + return UNDICI_TIMEOUT_CODES.has((error as Error & { cause?: { code?: unknown } }).cause?.code) } /** * True for a network-level fetch failure against the upstream host (no HTTP - * response at all — suspended pod, no endpoints). Excludes AbortError (today's - * 504 path) and UpstreamHostError (the host answered). + * response at all — suspended pod, no endpoints). Excludes timeouts (today's + * 504 path), UpstreamHostError (the host answered) and UpstreamBodyReadError + * (the headers arrived, so the host received the request). */ export function isHostDownNetworkError(error: unknown): boolean { if (!(error instanceof Error)) return false - if (isUpstreamTimeoutError(error) || error.name === 'UpstreamHostError') return false + if ( + isUpstreamTimeoutError(error) || + error.name === 'UpstreamHostError' || + error.name === 'UpstreamBodyReadError' + ) { + return false + } const cause = (error as Error & { cause?: { code?: unknown } }).cause const details = `${error.message} ${String(cause?.code || '')}`.toLowerCase() return ( @@ -582,17 +640,36 @@ export type RespondWithWakeAndHoldOptions = { claims: RpcAccessClaims /** Raw bearer forwarded to the wake plane when the caller is wake-capable. */ rpcAccessToken: string - /** Re-issues the original upstream request and writes the success response. */ - attemptUpstream: () => Promise + /** + * Re-issues the original upstream request with a deadline-bounded timeout. + * The route's FIRST attempt runs without a client-side timeout for mutating + * calls; `timeoutMs` only ever applies to these hold retries. `deadlineMs` + * is the absolute request deadline, for phases after the headers (a body + * transfer) that `timeoutMs` alone does not bound. + */ + attemptUpstream: (timeoutMs: number, deadlineMs: number) => Promise /** Writes today's error response (502/504) — the pre-wake behavior. */ respondLegacy: (error: unknown) => void coordinator?: WakeAndHoldCoordinator + /** Absolute deadline captured before a route's initial availability probe. */ + deadlineMs?: number + /** + * Keep re-issuing the request (with backoff) until it is accepted or the + * deadline expires. Only safe when a re-issued request cannot duplicate a + * side effect: the idempotent GET reads (sessions, transcript, context + * breakdown, models, task result, artifacts list and download, which commits + * only at `res.send`), and `/messages`, whose per-request `messageId` + * mcp-host's admission sink dedupes. Approve/deny/model/cancel have no + * idempotency key, so they re-issue at most ONCE after the hold: each extra + * POST risks a duplicate side effect. Defaults to false. + */ + retryUntilDeadline?: boolean } /** * Drives one held request through the wake flow and always writes a response: - * - proceed → short upstream retry schedule (draining bounces clear in ~1-2s), - * then structured host_waking if the pod is still not answering. + * - proceed → retry availability failures until upstream accepts or the + * single request deadline expires. * - legacy → today's 502 path (409 not-stateless, wake plane failure, or a * non-wake-capable caller with no wake authorization to ride). * - not-found→ 404. @@ -602,12 +679,12 @@ export async function respondWithWakeAndHold( options: RespondWithWakeAndHoldOptions ): Promise { const coordinator = options.coordinator ?? hostWakeCoordinator + const safeHostRef = sanitizeHostRefForLog(options.hostRef) + const deadlineMs = options.deadlineMs ?? Date.now() + config.wakeMaxHoldMs if (options.res.headersSent) { // A response is already committed for this request: parking it could only // ever produce a duplicate upstream delivery. Refuse loudly. - console.warn( - `[RPC_PROXY] wake-hold refused: response already committed host=${options.hostRef}` - ) + console.warn(`[RPC_PROXY] wake-hold refused: response already committed host=${safeHostRef}`) return } const outcome = await coordinator.hold({ @@ -615,23 +692,37 @@ export async function respondWithWakeAndHold( host: options.host, claims: options.claims, rpcAccessToken: options.rpcAccessToken, + deadlineMs, }) + if (options.res.headersSent) { + console.debug( + `[RPC_PROXY] late wake-hold artifact ignored (already resolved) host=${safeHostRef} artifact=hold-outcome` + ) + return + } + switch (outcome.kind) { case 'proceed': { - for (const delayMs of PROCEED_RETRY_DELAYS_MS) { - if (delayMs > 0) await sleep(delayMs) + let availabilityFailures = 0 + let attempts = 0 + const maxAttempts = options.retryUntilDeadline ? Infinity : 1 + const minAttemptTimeoutMs = Math.min(MIN_ADMISSION_RETRY_TIMEOUT_MS, config.upstreamTimeoutMs) + while (Date.now() < deadlineMs) { if (options.res.headersSent) { // Resolution latch: the response was committed while this retry // schedule was pending. Re-forwarding now would duplicate a message // the upstream already accepted. console.debug( - `[RPC_PROXY] late wake-hold artifact ignored (already resolved) host=${options.hostRef} artifact=proceed-retry` + `[RPC_PROXY] late wake-hold artifact ignored (already resolved) host=${safeHostRef} artifact=proceed-retry` ) return } try { - await options.attemptUpstream() + const remainingMs = deadlineMs - Date.now() + if (remainingMs < minAttemptTimeoutMs) break + attempts++ + await options.attemptUpstream(Math.min(config.upstreamTimeoutMs, remainingMs), deadlineMs) return } catch (error) { if (options.res.headersSent) { @@ -639,17 +730,28 @@ export async function respondWithWakeAndHold( // success write threw): the request is resolved. A retry here is // the duplicate-delivery bug — never re-forward, fail loudly. console.warn( - `[RPC_PROXY] wake-hold post-response failure suppressed (already resolved) host=${options.hostRef} error=${ + `[RPC_PROXY] wake-hold post-response failure suppressed (already resolved) host=${safeHostRef} error=${ error instanceof Error ? error.message : String(error) }` ) return } - if (isWakeEligibleHostError(error)) continue + if (isWakeEligibleHostError(error)) { + // Single-retry routes stop here: the host is still unreachable + // after the wake, so the caller gets the retryable 503 below. + if (attempts >= maxAttempts) break + const remainingMs = deadlineMs - Date.now() + const delayMs = + ADMISSION_RETRY_DELAYS_MS[ + Math.min(availabilityFailures++, ADMISSION_RETRY_DELAYS_MS.length - 1) + ] + if (remainingMs > 0) await sleep(Math.min(delayMs, remainingMs)) + continue + } // The host answered with a non-availability failure — exactly // today's behavior for an up-but-erroring host. console.warn( - `[RPC_PROXY] wake-hold upstream retry failed host=${options.hostRef} error=${ + `[RPC_PROXY] wake-hold upstream retry failed host=${safeHostRef} error=${ error instanceof Error ? error.message : String(error) }` ) @@ -659,12 +761,12 @@ export async function respondWithWakeAndHold( } if (options.res.headersSent) { console.debug( - `[RPC_PROXY] late wake-hold artifact ignored (already resolved) host=${options.hostRef} artifact=host-waking-response` + `[RPC_PROXY] late wake-hold artifact ignored (already resolved) host=${safeHostRef} artifact=host-waking-response` ) return } console.warn( - `[RPC_PROXY] wake-hold host still unreachable after wake host=${options.hostRef} lastKnownState=${outcome.lastKnownState}` + `[RPC_PROXY] wake-hold admission ended without an accepted attempt host=${safeHostRef} attempts=${attempts} lastKnownState=${outcome.lastKnownState}` ) respondHostWaking(options.res, { hostRef: options.hostRef, @@ -675,7 +777,7 @@ export async function respondWithWakeAndHold( } case 'legacy': console.warn( - `[RPC_PROXY] wake-hold falling back to legacy error path host=${options.hostRef} reason=${outcome.reason}` + `[RPC_PROXY] wake-hold falling back to legacy error path host=${safeHostRef} reason=${outcome.reason}` ) options.respondLegacy(new Error(`Upstream host unavailable (${outcome.reason})`)) return @@ -684,7 +786,7 @@ export async function respondWithWakeAndHold( return case 'waking': console.info( - `[RPC_PROXY] wake-hold responding host_waking host=${options.hostRef} reason=${outcome.reason} lastKnownState=${outcome.lastKnownState}` + `[RPC_PROXY] wake-hold responding host_waking host=${safeHostRef} reason=${outcome.reason} lastKnownState=${outcome.lastKnownState}` ) respondHostWaking(options.res, { hostRef: options.hostRef, diff --git a/rpc-proxy/test/routes.rpc.test.ts b/rpc-proxy/test/routes.rpc.test.ts index 5b9131c8f6..3527e6a7fb 100644 --- a/rpc-proxy/test/routes.rpc.test.ts +++ b/rpc-proxy/test/routes.rpc.test.ts @@ -21,7 +21,10 @@ const serviceMock = vi.hoisted(() => ({ const controlApiMock = vi.hoisted(() => ({ ControlApiHostAccessRejectedError: class ControlApiHostAccessRejectedError extends Error { - constructor(public readonly status: number) { + constructor( + public readonly status: number, + public readonly denialCode: string | null = null + ) { super(`Control API rejected host access (${status})`) this.name = 'ControlApiHostAccessRejectedError' } @@ -305,12 +308,21 @@ describe('routes/rpc', () => { }) it.each([ - [401, { error: 'Unauthorized' }], - [403, { error: 'Forbidden: user cannot access this host' }], - [409, { error: 'Direct run attribution conflict' }], + [401, null, { error: 'Unauthorized' }], + [ + 403, + 'host_access_revoked', + { error: 'Forbidden: user cannot access this host', code: 'host_access_revoked' }, + ], + [ + 403, + 'host_access_denied', + { error: 'Forbidden: user cannot access this host', code: 'host_access_denied' }, + ], + [409, null, { error: 'Direct run attribution conflict' }], ] as const)( - 'preserves a Control API %s from atomic host resolution and never forwards', - async (status, expectedBody) => { + 'preserves a Control API %s (%s) from atomic host resolution and never forwards', + async (status, denialCode, expectedBody) => { authTokenMock.verifyRpcToken.mockReturnValue({ sub: 'user-1', typ: 'user', @@ -323,7 +335,7 @@ describe('routes/rpc', () => { exp: 9999999999, }) serviceMock.resolveHostConnectionForUser.mockRejectedValue( - new controlApiMock.ControlApiHostAccessRejectedError(status) + new controlApiMock.ControlApiHostAccessRejectedError(status, denialCode) ) await request(makeApp()) @@ -662,7 +674,10 @@ describe('routes/rpc', () => { iat: 1, exp: 9999999999, }) - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) const app = makeApp() await request(app) .get('/rpc/hosts/not-allowed/status') @@ -682,7 +697,10 @@ describe('routes/rpc', () => { iat: 1, exp: 9999999999, }) - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) const app = makeApp() await request(app) .get('/rpc/hosts/not-allowed/health') @@ -702,7 +720,10 @@ describe('routes/rpc', () => { iat: 1, exp: 9999999999, }) - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) const app = makeApp() await request(app) .post('/rpc/hosts/not-allowed/messages') diff --git a/rpc-proxy/test/routes.rpcHostActivityStream.test.ts b/rpc-proxy/test/routes.rpcHostActivityStream.test.ts index 347b38f155..5ef118f243 100644 --- a/rpc-proxy/test/routes.rpcHostActivityStream.test.ts +++ b/rpc-proxy/test/routes.rpcHostActivityStream.test.ts @@ -76,7 +76,10 @@ describe('routes/rpcHostActivityStream', () => { iat: 1, exp: 9999999999, }) - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) const app = makeApp() await request(app) .get('/rpc/hosts/nope/activity/stream') diff --git a/rpc-proxy/test/routes.rpcHostStatusStream.test.ts b/rpc-proxy/test/routes.rpcHostStatusStream.test.ts index ee0daf3626..00d7c88dac 100644 --- a/rpc-proxy/test/routes.rpcHostStatusStream.test.ts +++ b/rpc-proxy/test/routes.rpcHostStatusStream.test.ts @@ -92,7 +92,10 @@ describe('routes/rpcHostStatusStream', () => { iat: 1, exp: 9999999999, }) - serviceMock.resolveHostConnectionForUser.mockResolvedValue(null) + serviceMock.resolveHostConnectionForUser.mockResolvedValue({ + denied: true, + code: 'host_access_denied', + }) const app = makeApp() await request(app) .get('/rpc/hosts/nope/status/stream') diff --git a/rpc-proxy/test/services.controlApiHostAccess.test.ts b/rpc-proxy/test/services.controlApiHostAccess.test.ts index 26e612a56a..0d2e1ac093 100644 --- a/rpc-proxy/test/services.controlApiHostAccess.test.ts +++ b/rpc-proxy/test/services.controlApiHostAccess.test.ts @@ -61,4 +61,150 @@ describe('control-api canonical host access client', () => { await expect(rejection).rejects.toBeInstanceOf(ControlApiHostAccessRejectedError) await expect(rejection).rejects.toMatchObject({ status }) }) + + const jsonResponse = (status: number, body: unknown, headers: Record = {}) => + vi.fn( + async () => + new Response(JSON.stringify(body), { + status, + headers: { 'content-type': 'application/json', ...headers }, + }) + ) + + // control-api reports the denial reason in this header; its 403 body is always + // exactly {"error":"Forbidden"}. Pinned literally so the test fails if the two + // services ever disagree on the name. + const REASON_HEADER = 'x-host-access-denial-reason' + const FORBIDDEN_BODY = { error: 'Forbidden' } + + // R1-M6: the denial reason (read from the response header, never the body) + // decides whether Desktop may treat the loss as a revocation. Only the two + // reasons that prove removed access map to `host_access_revoked`; + // `subject_mismatch` is a request-shape error (R3-L8). + it.each([ + ['team_membership_missing', 'host_access_revoked'], + ['directory_grant_missing', 'host_access_revoked'], + ['subject_mismatch', 'host_access_denied'], + ['host_disabled', 'host_access_denied'], + ['host_missing', 'host_access_denied'], + ['host_claim_missing', 'host_access_denied'], + ['unheard_of_reason', 'host_access_denied'], + ])('a non-direct-run 403 with reason %s resolves to the denial %s', async (reason, code) => { + const fetchImpl = jsonResponse(403, FORBIDDEN_BODY, { [REASON_HEADER]: reason }) + + await expect( + fetchHostConnectionFromControlApi('user-1', 'host-a', 'signed-rpc-token', { + fetchImpl: fetchImpl as typeof fetch, + }) + ).resolves.toEqual({ denied: true, code }) + expect(fetchImpl).toHaveBeenCalledOnce() + }) + + it('a non-direct-run 403 without the reason header resolves to host_access_denied', async () => { + const fetchImpl = jsonResponse(403, FORBIDDEN_BODY) + + await expect( + fetchHostConnectionFromControlApi('user-1', 'host-a', 'signed-rpc-token', { + fetchImpl: fetchImpl as typeof fetch, + }) + ).resolves.toEqual({ denied: true, code: 'host_access_denied' }) + expect(fetchImpl).toHaveBeenCalledOnce() + }) + + it('a non-direct-run 403 with an unknown header value resolves to host_access_denied', async () => { + const fetchImpl = jsonResponse(403, FORBIDDEN_BODY, { [REASON_HEADER]: 'a_reason_added_later' }) + + await expect( + fetchHostConnectionFromControlApi('user-1', 'host-a', 'signed-rpc-token', { + fetchImpl: fetchImpl as typeof fetch, + }) + ).resolves.toEqual({ denied: true, code: 'host_access_denied' }) + expect(fetchImpl).toHaveBeenCalledOnce() + }) + + it('a non-direct-run 403 with a non-JSON body still honours the header', async () => { + // A revoking reason: a denied-anyway reason would pass even if the header + // were ignored. + const fetchImpl = vi.fn( + async () => + new Response('not json', { + status: 403, + headers: { [REASON_HEADER]: 'team_membership_missing' }, + }) + ) + + await expect( + fetchHostConnectionFromControlApi('user-1', 'host-a', 'signed-rpc-token', { + fetchImpl: fetchImpl as typeof fetch, + }) + ).resolves.toEqual({ denied: true, code: 'host_access_revoked' }) + }) + + it('a reason present only in the body (legacy shape) is ignored', async () => { + // Proves the body is never read: the same reason maps to host_access_revoked + // when it arrives as the header (see the table above) but not from the body. + const fetchImpl = jsonResponse(403, { error: 'Forbidden', reason: 'team_membership_missing' }) + + await expect( + fetchHostConnectionFromControlApi('user-1', 'host-a', 'signed-rpc-token', { + fetchImpl: fetchImpl as typeof fetch, + }) + ).resolves.toEqual({ denied: true, code: 'host_access_denied' }) + expect(fetchImpl).toHaveBeenCalledOnce() + }) + + it('a non-direct-run 404 resolves to host_access_denied', async () => { + const fetchImpl = jsonResponse(404, { error: 'not found' }) + + await expect( + fetchHostConnectionFromControlApi('user-1', 'host-a', 'signed-rpc-token', { + fetchImpl: fetchImpl as typeof fetch, + }) + ).resolves.toEqual({ denied: true, code: 'host_access_denied' }) + expect(fetchImpl).toHaveBeenCalledOnce() + }) + + it.each([ + ['userId mismatch', { userId: 'someone-else', hostRef: 'host-a', url: 'http://h:8080' }], + ['hostRef mismatch', { userId: 'user-1', hostRef: 'host-b', url: 'http://h:8080' }], + ['empty url', { userId: 'user-1', hostRef: 'host-a', url: ' ' }], + ])('a 200 with %s resolves to host_access_denied', async (_label, body) => { + const fetchImpl = jsonResponse(200, body) + + await expect( + fetchHostConnectionFromControlApi('user-1', 'host-a', 'signed-rpc-token', { + fetchImpl: fetchImpl as typeof fetch, + }) + ).resolves.toEqual({ denied: true, code: 'host_access_denied' }) + }) + + it.each([ + ['team_membership_missing', 'host_access_revoked'], + ['host_disabled', 'host_access_denied'], + [undefined, 'host_access_denied'], + ])('a direct-run 403 (reason %s) throws with denialCode %s', async (reason, code) => { + const fetchImpl = jsonResponse(403, FORBIDDEN_BODY, reason ? { [REASON_HEADER]: reason } : {}) + + await expect( + fetchHostConnectionFromControlApi('user-1', 'host-a', 'signed-rpc-token', { + directRunBinding: BINDING, + fetchImpl: fetchImpl as typeof fetch, + }) + ).rejects.toMatchObject({ status: 403, denialCode: code }) + }) + + it.each([401, 409] as const)('a %s carries no denial code', async status => { + const fetchImpl = jsonResponse( + status, + { error: 'x', reason: 'team_membership_missing' }, + { [REASON_HEADER]: 'team_membership_missing' } + ) + + await expect( + fetchHostConnectionFromControlApi('user-1', 'host-a', 'signed-rpc-token', { + directRunBinding: BINDING, + fetchImpl: fetchImpl as typeof fetch, + }) + ).rejects.toMatchObject({ status, denialCode: null }) + }) }) diff --git a/rpc-proxy/test/services.mcpProxyService.test.ts b/rpc-proxy/test/services.mcpProxyService.test.ts index daf1d21643..678fada2be 100644 --- a/rpc-proxy/test/services.mcpProxyService.test.ts +++ b/rpc-proxy/test/services.mcpProxyService.test.ts @@ -242,12 +242,12 @@ describe('services/mcpProxyService', () => { }) }) - it('returns null on forbidden host resolution', async () => { + it('resolves a forbidden host to a non-revoking denial when control-api gives no reason', async () => { vi.spyOn(globalThis, 'fetch').mockResolvedValueOnce( mkResponse(403, JSON.stringify({ error: 'Forbidden' })) ) const host = await resolveHostConnectionForUser('user-1', 'agent2', 'rpc-token') - expect(host).toBeNull() + expect(host).toEqual({ denied: true, code: 'host_access_denied' }) }) it('forwards host message and returns REST response body', async () => { diff --git a/scripts/dev/autoresearch-benchmark.sh b/scripts/dev/autoresearch-benchmark.sh new file mode 100755 index 0000000000..a2701b14cc --- /dev/null +++ b/scripts/dev/autoresearch-benchmark.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +set -euo pipefail + +if [[ "${1:-}" != "--" || $# -lt 2 ]]; then + printf 'Usage: autoresearch-benchmark.sh -- BENCHMARK [ARG...]\n' >&2 + exit 2 +fi + +SCRIPT_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)" +REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || true)" +if [[ -z "${REPO_ROOT}" ]]; then + printf 'ERROR: autoresearch benchmark must run inside a Git checkout\n' >&2 + exit 2 +fi +cd "${REPO_ROOT}" + +# The measurement intake is deliberately run before the candidate benchmark. +# It refuses stale/missing origin/dev ancestry, detached default measurements, +# and merge conflicts without fetching or changing repository state. +if bash "${SCRIPT_ROOT}/scripts/dev/repo-intake-packet.sh" --measure -- true; then + printf 'AUTORESEARCH_GATE branch_freshness=pass\n' +else + status=$? + printf 'AUTORESEARCH_GATE branch_freshness=fail status=%s\n' "${status}" >&2 + exit "${status}" +fi + +shift +exec "$@" diff --git a/scripts/dev/autoresearch-checks.sh b/scripts/dev/autoresearch-checks.sh new file mode 100755 index 0000000000..28ff8f15d5 --- /dev/null +++ b/scripts/dev/autoresearch-checks.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +set -euo pipefail + +REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || true)" +if [[ -z "${REPO_ROOT}" ]]; then + printf 'ERROR: autoresearch checks must run inside a Git checkout\n' >&2 + exit 2 +fi +cd "${REPO_ROOT}" + +# AutoResearch maps a non-zero checks command to its tests_pass hard gate. +# Keep the parity result explicit in the captured packet output as well. +if make test-service-matrix; then + printf 'AUTORESEARCH_GATE service_matrix_parity=pass\n' +else + status=$? + printf 'AUTORESEARCH_GATE service_matrix_parity=fail status=%s\n' "${status}" >&2 + exit "${status}" +fi diff --git a/scripts/e2e/_lib/hcc-watch-api-proxy.mjs b/scripts/e2e/_lib/hcc-watch-api-proxy.mjs index 31eca8cfb3..b69444f2da 100644 --- a/scripts/e2e/_lib/hcc-watch-api-proxy.mjs +++ b/scripts/e2e/_lib/hcc-watch-api-proxy.mjs @@ -4,6 +4,11 @@ import https from 'node:https' import { createSecureContext } from 'node:tls' import { createBookmarkObservation } from './hcc-watch-bookmarks.mjs' +// Individual fixture commands remain one-minute bounded; a single continuous +// cache-loss observation has a two-minute ceiling even when renewed. +const MAX_CHANNEL_HOLD_WINDOW_MS = 60_000 +const MAX_CHANNEL_HOLD_BUDGET_MS = 120_000 + export function proxyUpstreamErrorRecord(error) { const codes = [ 'ECONNREFUSED', @@ -46,7 +51,15 @@ export function validateCommand(command, allowedPaths) { } else if (command.action === 'release') { if (typeof command.pauseId !== 'string' || !/^[a-zA-Z0-9-]{1,80}$/.test(command.pauseId)) throw new Error('invalid_pause_id') - } else if (command.action !== 'observe-bookmarks') throw new Error('invalid_control_action') + } else if (command.action === 'hold-channel') { + if ( + !Number.isInteger(command.durationMs) || + command.durationMs < 1000 || + command.durationMs > MAX_CHANNEL_HOLD_WINDOW_MS + ) + throw new Error('invalid_channel_hold_duration') + } else if (command.action !== 'release-channel' && command.action !== 'observe-bookmarks') + throw new Error('invalid_control_action') return command } @@ -60,12 +73,15 @@ export function createProxy({ controlDir, periodMs, minAgeMs, + channelPath, + churnEnabled = true, }) { // CA bytes configure TLS trust only; they never enter the request or destination. const upstreamSecureContext = createSecureContext({ ca: upstreamCa }) const streams = new Set() const bookmarks = createBookmarkObservation() let pause = null + let channelHold = null let commandId = null let commandContent = null const writeRecord = (name, fields) => { @@ -83,6 +99,20 @@ export function createProxy({ old.resume?.() acknowledge({ id: old.id, state: reason }) } + const finishChannelHold = reason => { + if (!channelHold) return + const held = channelHold + channelHold = null + clearTimeout(held.timer) + writeRecord('channel-hold', { + id: held.id, + state: reason, + cut: held.cut, + rejected: held.rejected, + renewals: held.renewals, + deadlineAtMs: held.deadlineAtMs, + }) + } const server = https.createServer({ key, cert }, (request, response) => { const url = new URL(request.url, 'https://fixture.invalid') if (!request.url.startsWith('/') || request.url.startsWith('//')) { @@ -90,6 +120,12 @@ export function createProxy({ return } const watch = url.searchParams.get('watch') === 'true' + if (channelHold && request.method === 'GET' && url.pathname === channelPath) { + channelHold.rejected++ + response.writeHead(503, { 'content-type': 'application/json' }) + response.end(JSON.stringify({ kind: 'Status', apiVersion: 'v1', status: 'Failure', reason: 'ServiceUnavailable', code: 503 })) + return + } const kind = url.pathname.endsWith('/mcpservers') ? 'McpServer' : url.pathname.endsWith('/contexts') @@ -188,6 +224,72 @@ export function createProxy({ throw new Error('pause_not_held') finishPause('released') acknowledge({ id: command.id, state: 'released' }) + } else if (command.action === 'hold-channel') { + if (!channelPath) throw new Error('channel_hold_unavailable') + if (channelHold) { + const now = Date.now() + const budgetDeadlineMs = channelHold.startedAtMs + MAX_CHANNEL_HOLD_BUDGET_MS + const deadlineAtMs = Math.min(now + command.durationMs, budgetDeadlineMs) + if (deadlineAtMs <= now) throw new Error('channel_hold_budget_exhausted') + clearTimeout(channelHold.timer) + channelHold.id = command.id + channelHold.deadlineAtMs = deadlineAtMs + channelHold.renewals++ + channelHold.timer = setTimeout( + () => finishChannelHold('expired'), + deadlineAtMs - now, + ) + writeRecord('channel-hold', { + id: channelHold.id, + state: 'held', + cut: channelHold.cut, + rejected: channelHold.rejected, + renewals: channelHold.renewals, + deadlineAtMs, + }) + acknowledge({ + id: command.id, + state: 'held', + count: channelHold.cut, + renewed: true, + renewals: channelHold.renewals, + deadlineAtMs, + }) + } else { + let count = 0 + const now = Date.now() + const deadlineAtMs = now + command.durationMs + channelHold = { + id: command.id, + startedAtMs: now, + deadlineAtMs, + cut: 0, + rejected: 0, + renewals: 0, + timer: setTimeout(() => finishChannelHold('expired'), command.durationMs), + } + for (const stream of streams) { + if (stream.watch && stream.request.method === 'GET' && + new URL(stream.request.url, 'https://fixture.invalid').pathname === channelPath) { + stream.close() + count++ + } + } + channelHold.cut = count + writeRecord('channel-hold', { + id: command.id, + state: 'held', + cut: count, + rejected: 0, + renewals: 0, + deadlineAtMs, + }) + acknowledge({ id: command.id, state: 'held', count, renewals: 0, deadlineAtMs }) + } + } else if (command.action === 'release-channel') { + if (!channelHold) throw new Error('channel_hold_not_active') + finishChannelHold('released') + acknowledge({ id: command.id, state: 'released' }) } else if (command.action === 'observe-bookmarks') { writeRecord('bookmarks', bookmarks.finish()) acknowledge({ id: command.id, state: 'observed' }) @@ -210,15 +312,16 @@ export function createProxy({ acknowledge({ id: observedId, state: 'rejected' }) } }, 100) - const churn = setInterval(() => { + const churn = churnEnabled ? setInterval(() => { if (fs.existsSync(`${controlDir}/paused`)) return for (const stream of streams) if (stream.watch && Date.now() - stream.born >= minAgeMs) stream.close() - }, periodMs) + }, periodMs) : null const close = () => { clearInterval(poll) - clearInterval(churn) + if (churn) clearInterval(churn) finishPause('shutdown') + finishChannelHold('shutdown') for (const stream of streams) stream.close() server.close() } @@ -235,6 +338,8 @@ if (import.meta.main) { controlDir: '/churn-ctl', periodMs: Number(process.env.CHURN_PERIOD_MS), minAgeMs: Number(process.env.CHURN_MIN_AGE_MS), + channelPath: process.env.CHANNEL_PATH, + churnEnabled: process.env.CHURN_DISABLED !== '1', }) proxy.server.listen(8443, '0.0.0.0') process.on('SIGTERM', proxy.close) diff --git a/scripts/e2e/_lib/hcc-watch-pr-a.sh b/scripts/e2e/_lib/hcc-watch-pr-a.sh index 0b24d121bf..8cb517078a 100755 --- a/scripts/e2e/_lib/hcc-watch-pr-a.sh +++ b/scripts/e2e/_lib/hcc-watch-pr-a.sh @@ -15,7 +15,7 @@ hcc_pr_a_command() { pod="$(hcc_pr_a_proxy_pod)" || return 1 HCC_PR_A_COMMAND_ID="${RUN_ID}-$(date +%s)-${RANDOM}" command="$(jq -cn --arg id "$HCC_PR_A_COMMAND_ID" --arg action "$action" --arg value "$value" \ - '{id:$id,action:$action} + (if $action=="arm" then {path:$value,method:"GET",durationMs:25000} elif $action=="cut" then {kind:$value} elif $action=="release" then {pauseId:$value} else {} end)')" + '{id:$id,action:$action} + (if $action=="arm" then {path:$value,method:"GET",durationMs:25000} elif $action=="cut" then {kind:$value} elif $action=="release" then {pauseId:$value} elif $action=="hold-channel" then {durationMs:($value|tonumber)} else {} end)')" printf '%s' "$command" | kctl exec -i "pod/$pod" -n "$HCC_NS" -c proxy -- node -e \ 'const fs=require("fs");const value=fs.readFileSync(0);fs.writeFileSync("/churn-ctl/next.json",value,{mode:384});fs.renameSync("/churn-ctl/next.json","/churn-ctl/command.json")' } @@ -24,7 +24,7 @@ hcc_pr_a_ack() { local state=$1 pod pod="$(hcc_pr_a_proxy_pod)" || return 1 kctl exec "pod/$pod" -n "$HCC_NS" -c proxy -- node -e \ - 'const fs=require("fs");try{const a=JSON.parse(fs.readFileSync("/churn-ctl/ack.json"));if(a.id!==process.argv[1]||a.state!==process.argv[2]||(a.state==="cut"&&a.count<1))process.exit(1)}catch{process.exit(1)}' \ + 'const fs=require("fs");try{const a=JSON.parse(fs.readFileSync("/churn-ctl/ack.json"));if(a.id!==process.argv[1]||a.state!==process.argv[2]||((a.state==="cut"||a.state==="held")&&a.count<1))process.exit(1)}catch{process.exit(1)}' \ "$HCC_PR_A_COMMAND_ID" "$state" >/dev/null 2>&1 } @@ -386,15 +386,19 @@ hcc_pr_a_enable_proxy() { # Generated private material stays in this anonymous pipe, never an artifact. node "$HCC_PR_A_LIB/hcc-watch-tls-manifest.mjs" "$PROXY_NAME" "$HCC_NS" "$RUN_ID" \ "$HCC_PR_A_LIB/hcc-watch-api-proxy.mjs" | kctl create -f - >/dev/null 2>&1 || die 'PR A TLS fixture creation failed' - patch="$(jq -cn --arg name "$PROXY_NAME" --arg paths "$paths" '{spec:{template:{spec:{ + patch="$(jq -cn --arg name "$PROXY_NAME" --arg paths "$paths" --arg channel_ns "${CHANNEL_NS:-channels}" \ + --argjson selective "${HCC_PR_A_SELECTIVE_CHANNEL:-0}" '{spec:{template:{spec:{ securityContext:{fsGroup:1000}, containers:[{name:"proxy",command:["node","/fixture/proxy.mjs"],args:[], - env:[{name:"PAUSE_PATHS",value:$paths}],volumeMounts:[ + env:([{name:"PAUSE_PATHS",value:$paths},{name:"CHANNEL_PATH",value:("/apis/clerum.io/v1alpha1/namespaces/"+$channel_ns+"/communicationchannels")}] + + (if $selective==1 then [{name:"CHURN_DISABLED",value:"1"}] else [] end)),volumeMounts:[ {name:"fixture-code",mountPath:"/fixture",readOnly:true}, {name:"fixture-tls",mountPath:"/fixture-tls",readOnly:true}, - {name:"upstream-ca",mountPath:"/upstream-ca",readOnly:true}]}], + {name:"upstream-ca",mountPath:"/upstream-ca",readOnly:true}, + {name:"churn-control",mountPath:"/churn-ctl"}]}], volumes:[{name:"fixture-code",configMap:{name:$name}}, {name:"fixture-tls","secret":{secretName:$name,defaultMode:288}}, - {name:"upstream-ca",configMap:{name:"kube-root-ca.crt"}}]}}}}')" + {name:"upstream-ca",configMap:{name:"kube-root-ca.crt"}}, + {name:"churn-control",emptyDir:{}}]}}}}')" kctl patch deployment "$PROXY_NAME" -n "$HCC_NS" --type=strategic -p "$patch" >/dev/null kctl rollout status deployment "$PROXY_NAME" -n "$HCC_NS" --timeout=60s >/dev/null || die 'PR A TLS proxy not Ready' } diff --git a/scripts/e2e/e2e-hcc-stateless-cache-containment.sh b/scripts/e2e/e2e-hcc-stateless-cache-containment.sh new file mode 100755 index 0000000000..d850b32367 --- /dev/null +++ b/scripts/e2e/e2e-hcc-stateless-cache-containment.sh @@ -0,0 +1,285 @@ +#!/usr/bin/env bash +# Branch-owned live gate for #396. The TLS fixture cuts only the +# CommunicationChannel watch/list while forwarding all other Kubernetes API +# requests. A known conversation is created through rpc-proxy before the cut. +set -euo pipefail +umask 077 +# shellcheck disable=SC2034 # Shared fixture helpers consume these globals. + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" +# The branch profile is seeded from the canonical repository credentials. +# Resolve them in-process when the caller has not supplied this journey's +# explicit password; never print the resolved value in gate evidence. +# shellcheck source=scripts/e2e/load-dotenv.sh +source "${REPO_ROOT}/scripts/e2e/load-dotenv.sh" +dotenv_load_canonical_root "${REPO_ROOT}" +# shellcheck source=scripts/e2e/admin-credentials.sh +source "${REPO_ROOT}/scripts/e2e/admin-credentials.sh" +# shellcheck source=scripts/e2e/e2e-lib.sh +source "${SCRIPT_DIR}/e2e-lib.sh" +# shellcheck source=scripts/e2e/_lib/hcc-watch-recovery-fixture.sh +source "${SCRIPT_DIR}/_lib/hcc-watch-recovery-fixture.sh" +# shellcheck source=scripts/e2e/_lib/hcc-watch-recovery-assertions.sh +source "${SCRIPT_DIR}/_lib/hcc-watch-recovery-assertions.sh" +# shellcheck source=scripts/e2e/_lib/hcc-watch-recovery-logs.sh +source "${SCRIPT_DIR}/_lib/hcc-watch-recovery-logs.sh" +# shellcheck source=scripts/e2e/_lib/hcc-watch-recovery-lock.sh +source "${SCRIPT_DIR}/_lib/hcc-watch-recovery-lock.sh" +# shellcheck source=scripts/e2e/_lib/hcc-watch-pr-a.sh +source "${SCRIPT_DIR}/_lib/hcc-watch-pr-a.sh" + +die() { fail "$*"; exit 1; } +wait_until() { + local timeout=$1 description=$2 elapsed=0 + shift 2 + while [ "$elapsed" -lt "$timeout" ]; do + "$@" && return 0 + sleep 1 + elapsed=$((elapsed + 1)) + done + echo "Timed out waiting for ${description}" >&2 + return 1 +} + +if [ -z "$E2E_KUBECONTEXT" ] || ! is_branch_scoped_e2e_context "$E2E_KUBECONTEXT"; then + die 'select an explicit branch-scoped Minikube context' +fi +require_safe_kube_context +[ "${E2E_HCC_WATCH_FAULT_INJECTION:-0}" = 1 ] || die 'set E2E_HCC_WATCH_FAULT_INJECTION=1' +[ -n "${E2E_BRANCH_PROFILE_ENV:-}" ] && [ -r "$E2E_BRANCH_PROFILE_ENV" ] || + die 'provide E2E_BRANCH_PROFILE_ENV from the owned branch profile' +[ -n "${E2E_PROFILE_PORTS_ENV:-}" ] && [ -r "$E2E_PROFILE_PORTS_ENV" ] || + die 'provide E2E_PROFILE_PORTS_ENV from the owned branch profile' +profile_port_value() { + awk -F= -v key="$1" '$1 == key { sub(/^[^=]*=/, ""); print; exit }' "$E2E_PROFILE_PORTS_ENV" +} +CONTROL_API_BASE_URL="$(profile_port_value CONTROL_API_URL)" +CONTROL_UI_BASE_URL="$(profile_port_value CONTROL_UI_URL)" +EXTERNAL_REST_API_BASE_URL="$(profile_port_value EXTERNAL_REST_API_URL)" +RPC_PROXY_BASE_URL="$(profile_port_value RPC_PROXY_URL)" +export CONTROL_API_BASE_URL CONTROL_UI_BASE_URL EXTERNAL_REST_API_BASE_URL RPC_PROXY_BASE_URL +require_branch_profile_urls "$E2E_KUBECONTEXT" "$E2E_PROFILE_PORTS_ENV" || + die 'profile URLs do not match the owned ports' +kctl get nodes -o json | jq -e --arg c "$E2E_KUBECONTEXT" \ + 'any(.items[]; .metadata.labels["minikube.k8s.io/name"]==$c)' >/dev/null || + die 'target is not the selected Minikube profile' + +HCC_NS=control-plane HCC_DEPLOY=host-context-controller +HOST_NS=mcp-host MCP_NS=mcp-server CHANNEL_NS=channels +HOST_REF="${E2E_STATELESS_HOST_REF:-chatllm-stateless}" +RUN_ID="$(date +%s)-$$" +PROXY_NAME="$(truncate_rfc1123 "e2e-hcc-channel-${RUN_ID}")" +PROBE_NAME="$(truncate_rfc1123 "${PROXY_NAME}-negative-probe")" +PROXY_EGRESS_NP="$(truncate_rfc1123 "${PROXY_NAME}-api")" +HCC_PROXY_NP="$(truncate_rfc1123 "${PROXY_NAME}-from-hcc")" +PROBE_EGRESS_NP="$(truncate_rfc1123 "${PROXY_NAME}-from-probe")" +THREAD_ID="e2e-hcc-cache-${RUN_ID}" +MARKER="hcc-cache-${RUN_ID}" +START_TIME="$(date -u +"%Y-%m-%dT%H:%M:%SZ")" +HCC_LOG_BUFFER="$(mktemp "${TMPDIR:-/tmp}/hcc-channel-log.XXXXXX")" +HCC_LOG_STREAM_PID='' +HCC_GATE_LOCK_ACQUIRED=0 HCC_GATE_LOCK_NAME='' HCC_GATE_LOCK_UID='' +HCC_GATE_FINALIZATION_FAILURE='' +HCC_PATCHED=0 PROXY_CREATED=0 PROBE_CREATED=0 HCC_PR_A_TLS_CREATED=0 +HCC_UID='' HCC_RESTARTS='' +ORIGINAL_REPLICAS='' +HCC_PR_A_CHANNEL_HOLD_ID='' + +cleanup() { + local status=$? cleanup_failed=0 restore_ok=1 + set +e + stop_hcc_recovery_log_stream + if [ "$HCC_PATCHED" = 1 ]; then + if hcc_pr_a_restore && kctl rollout status deployment "$HCC_DEPLOY" -n "$HCC_NS" --timeout=180s >/dev/null; then + HCC_PATCHED=0 + else + restore_ok=0 + fi + fi + if [ "$restore_ok" = 1 ]; then + [ "$PROXY_CREATED" = 0 ] || delete_hcc_proxy_fixture || cleanup_failed=1 + if [ "$HCC_PR_A_TLS_CREATED" = 1 ]; then + kctl delete configmap "$PROXY_NAME" -n "$HCC_NS" --ignore-not-found --wait=true --timeout=60s >/dev/null 2>&1 || cleanup_failed=1 + kctl delete secret "$PROXY_NAME" -n "$HCC_NS" --ignore-not-found --wait=true --timeout=60s >/dev/null 2>&1 || cleanup_failed=1 + fi + else + cleanup_failed=1 + echo "HCC restoration failed; retained proxy ${HCC_NS}/${PROXY_NAME} on ${E2E_KUBECONTEXT}" >&2 + fi + finalize_hcc_watch_gate_lock "$cleanup_failed" "$restore_ok" || cleanup_failed=1 + rm -f "$HCC_LOG_BUFFER" "${HCC_PR_A_CONFIG_SNAPSHOT:-}" + [ "$cleanup_failed" = 0 ] || status=1 + print_results + exit "$status" +} +trap cleanup EXIT + +header 'Stateless Host containment during CommunicationChannel cache uncertainty' +require_branch_owned_hcc_gate +acquire_hcc_watch_gate_lock || die 'another HCC watch gate owns this profile' +ORIGINAL_REPLICAS="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.spec.replicas}')" +[ "$ORIGINAL_REPLICAS" = 1 ] || die 'HCC must have exactly one replica' +[ "$(kctl get host "$HOST_REF" -n "$HOST_NS" -o jsonpath='{.spec.lifecycle.stateless}')" = true ] || + die 'selected Host must request stateless lifecycle' + +HCC_IMAGE="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.spec.template.spec.containers[?(@.name=="host-context-controller")].image}')" +[ -n "$HCC_IMAGE" ] || die 'could not resolve the running HCC image' +K8S_API_SERVICE_HOST="$(kctl exec deployment/"$HCC_DEPLOY" -n "$HCC_NS" -c host-context-controller -- printenv KUBERNETES_SERVICE_HOST)" +node -e 'process.exit(require("node:net").isIP(process.argv[1])===4?0:1)' "$K8S_API_SERVICE_HOST" || + die 'Kubernetes API service address is not IPv4' +K8S_API_CIDR="${K8S_API_SERVICE_HOST}/32" +EXT_BASE="${EXTERNAL_REST_API_BASE_URL%/}" +RPC_BASE="${RPC_PROXY_BASE_URL%/}" +DEV_EMAIL="${E2E_DEV_LOGIN_EMAIL:-test@clerum.io}" +DEV_PASSWORD="${E2E_USER_PASSWORD:-}" +if [ -z "$DEV_PASSWORD" ]; then + DEV_PASSWORD="$(e2e_resolve_admin_password "$REPO_ROOT" || true)" +fi +[ -n "$DEV_PASSWORD" ] || die 'known-session proof requires E2E_USER_PASSWORD or a canonical admin password' +hcc_pr_a_preflight +create_hcc_api_proxy +HCC_PR_A_SELECTIVE_CHANNEL=1 hcc_pr_a_enable_proxy +E2E_HCC_PR_A=1 verify_hcc_proxy_network_policy +HCC_PATCHED=1 +kctl patch deployment "$HCC_DEPLOY" -n "$HCC_NS" --type=strategic -p "$(hcc_pr_a_redirect_patch)" >/dev/null +kctl rollout status deployment "$HCC_DEPLOY" -n "$HCC_NS" --timeout=180s >/dev/null || + die 'HCC did not become Ready through selective TLS proxy' +read -r HCC_UID HCC_RESTARTS <<<"$(wait_for_hcc_identity 30)" +start_hcc_recovery_log_stream + +curl -fsS -m 10 "${EXT_BASE}/health" >/dev/null || die 'external-rest-api unavailable' +curl -fsS -m 10 "${RPC_BASE}/health" >/dev/null || die 'rpc-proxy unavailable' +login="$(curl -fsS -m 20 -X POST "${EXT_BASE}/api/v1/auth/password-login" \ + -H 'Content-Type: application/json' -d "$(jq -cn --arg e "$DEV_EMAIL" --arg p "$DEV_PASSWORD" '{email:$e,password:$p}')")" || + die 'E2E user login failed' +SESSION_TOKEN="$(jq -er '.token' <<<"$login")" || die 'login returned no token' +mint_rpc_token() { + local body + # The existing messages route uses wakeAndHold when the stateless Host is + # suspended. Keep wake capability on this user token so the baseline message + # can take that supported Desktop-to-rpc-proxy path when it is needed. + body="$(curl -fsS -m 20 -X POST "${EXT_BASE}/api/v1/rpc/token" \ + -H "Authorization: Bearer ${SESSION_TOKEN}" -H 'Content-Type: application/json' \ + -d "$(jq -cn --arg host "$HOST_REF" '{hostRefs:[$host],scopes:["host:message:invoke","host:session:read","host:status:read","host:health:read","host:wake:write"]}')")" || return 1 + RPC_TOKEN="$(jq -er '.token' <<<"$body")" +} +mint_rpc_token || die 'could not mint authorized RPC token' +message="$(jq -cn --arg content "Remember ${MARKER}." --arg thread "$THREAD_ID" '{content:$content,threadId:$thread}')" +baseline_accepted=0 +for attempt in 1 2 3; do + mint_rpc_token || die 'could not refresh authorized RPC token' + response="$(curl -sS -m 120 -w '\n%{http_code}' -X POST "${RPC_BASE}/api/v1/rpc/hosts/${HOST_REF}/messages" \ + -H "Authorization: Bearer ${RPC_TOKEN}" -H 'Content-Type: application/json' -d "$message")" || + die 'baseline message through rpc-proxy failed' + if [ "${response##*$'\n'}" = 200 ]; then + baseline_accepted=1 + break + fi + if [ "$attempt" = 3 ] || [ "${response##*$'\n'}" != 503 ] || + ! jq -e '.code=="host_waking"' <<<"${response%$'\n'*}" >/dev/null; then + die 'baseline message was not accepted' + fi + wait_until 90 'Host wake after retryable response' host_runtime_is_always_on "$HOST_REF" || + die 'Host did not wake after retryable response' +done +[ "$baseline_accepted" = 1 ] || die 'baseline message was not accepted' + +session_visible() { + local response status body + mint_rpc_token || return 1 + response="$(curl -sS -m 30 -w '\n%{http_code}' \ + -H "Authorization: Bearer ${RPC_TOKEN}" \ + "${RPC_BASE}/api/v1/rpc/hosts/${HOST_REF}/sessions/${HOST_REF}/${THREAD_ID}/messages")" || return 1 + status="${response##*$'\n'}" body="${response%$'\n'*}" + [ "$status" = 200 ] && jq -e --arg marker "$MARKER" \ + 'any(.turns[]?; .user_input|contains($marker))' <<<"$body" >/dev/null +} +session_listed() { + local response status body + mint_rpc_token || return 1 + response="$(curl -sS -m 30 -w '\n%{http_code}' \ + -H "Authorization: Bearer ${RPC_TOKEN}" \ + "${RPC_BASE}/api/v1/rpc/hosts/${HOST_REF}/sessions?agent=${HOST_REF}&limit=100")" || return 1 + status="${response##*$'\n'}" body="${response%$'\n'*}" + [ "$status" = 200 ] && jq -e --arg thread "$THREAD_ID" --arg agent "$HOST_REF" \ + 'any(.items[]?; .chatId==$thread and .agent==$agent)' <<<"$body" >/dev/null +} +wait_until 30 'baseline session transcript' session_visible || die 'known session was not readable' +wait_until 30 'baseline session list' session_listed || die 'known session was not listed' +wait_until 90 'baseline active Host' host_runtime_is_always_on "$HOST_REF" || + die 'Host was not active before fault injection' +baseline="$(kctl get deployment "$HOST_REF" -n "$HOST_NS" -o json)" +baseline_template="$(jq -Sc '.spec.template' <<<"$baseline")" +baseline_pod="$(kctl get pods -n "$HOST_NS" -l "app=${HOST_REF}" -o json | + jq -er '[.items[]|select(.metadata.deletionTimestamp==null)|select(any(.status.conditions[]?;.type=="Ready" and .status=="True"))]|if length==1 then .[0].metadata.uid else error("pod identity ambiguous") end')" +jq -e '[.spec.template.spec.containers[].env[]? | select(.name=="CLERUM_SESSION_DB_DIR" and .value=="/var/lib/clerum/state")]|length==1' <<<"$baseline" >/dev/null || + die 'baseline SQLite directory is not the stateless path' + +cache_held_active() { + local status deployment pod + status="$(kctl get host "$HOST_REF" -n "$HOST_NS" -o json)" || return 1 + jq -e '[.status.conditions[]?|select(.type=="StatelessEnableRejected")][0] | + .status=="False" and .reason=="CommunicationChannelCacheUnsynced"' <<<"$status" >/dev/null || return 1 + host_runtime_is_always_on "$HOST_REF" || return 1 + deployment="$(kctl get deployment "$HOST_REF" -n "$HOST_NS" -o json)" || return 1 + [ "$(jq -Sc '.spec.template' <<<"$deployment")" = "$baseline_template" ] || return 1 + pod="$(kctl get pods -n "$HOST_NS" -l "app=${HOST_REF}" -o json | + jq -er '[.items[]|select(.metadata.deletionTimestamp==null)|select(any(.status.conditions[]?;.type=="Ready" and .status=="True"))]|if length==1 then .[0].metadata.uid else error("pod identity ambiguous") end')" || return 1 + [ "$pod" = "$baseline_pod" ] +} + +channel_hold_is_active() { + local expected_id=$1 pod + pod="$(hcc_pr_a_proxy_pod)" || return 1 + kctl exec "pod/$pod" -n "$HCC_NS" -c proxy -- node -e \ + 'const fs=require("fs");try{const hold=JSON.parse(fs.readFileSync("/churn-ctl/channel-hold.json"));if(hold.id!==process.argv[1]||hold.state!=="held"||!Number.isFinite(hold.deadlineAtMs)||hold.deadlineAtMs<=Date.now())process.exit(1)}catch{process.exit(1)}' \ + "$expected_id" >/dev/null 2>&1 +} + +renew_channel_hold() { + local cycle=$1 + hcc_pr_a_command hold-channel 60000 || die "cycle ${cycle}: channel-hold renewal command failed" + HCC_PR_A_CHANNEL_HOLD_ID="$HCC_PR_A_COMMAND_ID" + wait_until 5 'selective hold renewal acknowledgement' hcc_pr_a_ack held || + die "cycle ${cycle}: channel-hold renewal was not acknowledged" + wait_until 5 'live selective hold after renewal' channel_hold_is_active "$HCC_PR_A_CHANNEL_HOLD_ID" || + die "cycle ${cycle}: renewed channel hold is not active" +} + +for cycle in 1 2; do + before="$(log_count 'CommunicationChannel watch ended;')" + hcc_pr_a_command hold-channel 60000 || die "cycle ${cycle}: fault command failed" + HCC_PR_A_CHANNEL_HOLD_ID="$HCC_PR_A_COMMAND_ID" + wait_until 5 'selective hold acknowledgement' hcc_pr_a_ack held || + die "cycle ${cycle}: no live channel watch was cut" + wait_until 5 'live selective hold' channel_hold_is_active "$HCC_PR_A_CHANNEL_HOLD_ID" || + die "cycle ${cycle}: channel hold is not active" + wait_log_count 'CommunicationChannel watch ended;' "$((before + 1))" 20 || + die "cycle ${cycle}: HCC did not lose channel authority" + wait_until 30 'reconciliation while channel cache is unsynced' cache_held_active || + die "cycle ${cycle}: active stateless Host did not converge during watch loss" + channel_hold_is_active "$HCC_PR_A_CHANNEL_HOLD_ID" || + die "cycle ${cycle}: channel hold expired during cache reconciliation" + renew_channel_hold "$cycle" + channel_hold_is_active "$HCC_PR_A_CHANNEL_HOLD_ID" || + die "cycle ${cycle}: channel hold expired before session transcript read" + session_visible || die "cycle ${cycle}: known session unavailable through rpc-proxy" + channel_hold_is_active "$HCC_PR_A_CHANNEL_HOLD_ID" || + die "cycle ${cycle}: channel hold expired during session transcript read" + renew_channel_hold "$cycle" + channel_hold_is_active "$HCC_PR_A_CHANNEL_HOLD_ID" || + die "cycle ${cycle}: channel hold expired before session list read" + session_listed || die "cycle ${cycle}: known session absent from authorized list" + channel_hold_is_active "$HCC_PR_A_CHANNEL_HOLD_ID" || + die "cycle ${cycle}: channel hold expired during session list read" + cache_held_active || die "cycle ${cycle}: template, SQLite path or pod identity changed" + hcc_pr_a_command release-channel || die "cycle ${cycle}: release command failed" + wait_until 5 'selective release acknowledgement' hcc_pr_a_ack released || + die "cycle ${cycle}: release was not acknowledged" + wait_until 30 'CommunicationChannel watch recovery' wait_host_mode "$HOST_REF" accepted 1 || + die "cycle ${cycle}: channel authority did not recover" + ok "cycle ${cycle}: reconciled active stateless Host, stable template/pod and readable session" +done +assert_hcc_identity +ok 'two selective watch-loss cycles completed without HCC restart' diff --git a/scripts/e2e/e2e-stateless-cron-block.sh b/scripts/e2e/e2e-stateless-cron-block.sh index f13a66b3c6..d3a260d9cf 100755 --- a/scripts/e2e/e2e-stateless-cron-block.sh +++ b/scripts/e2e/e2e-stateless-cron-block.sh @@ -5,6 +5,8 @@ # because tool selection is model-driven in the live chat path. This T2 gate # proves the runtime lifecycle contract with deterministic heartbeat evidence: # activeCronSchedules blocks suspension, and clearing it releases suspension. +# It accelerates HCC cadences only with E2E_ALLOW_STATELESS_CADENCE_ACCELERATION=1 +# and fails cleanup if the saved values are not restored and read back exactly. # # Registered gate: e2e-stateless-cron-block (run directly; not wired to Make). set -euo pipefail @@ -184,6 +186,10 @@ probe_pod_cron_policy_artifacts() { } save_and_set_hcc_cadences() { + if [ "${E2E_ALLOW_STATELESS_CADENCE_ACCELERATION:-0}" != 1 ]; then + fail "refusing to accelerate stateless lifecycle cadences; set E2E_ALLOW_STATELESS_CADENCE_ACCELERATION=1 on an owned profile" + return 1 + fi header "PRECONDITION (labeled setup) -- HCC test cadences (idle=${TEST_IDLE_MINUTES}min, drain=${TEST_DRAIN_GRACE_MS}ms)" local keys=(CONTEXT_MAPPER_STATELESS_IDLE_MINUTES CONTEXT_MAPPER_STATELESS_IDLE_FLOOR_MINUTES \ CONTEXT_MAPPER_STATELESS_DRAIN_GRACE_MS CONTEXT_MAPPER_HEARTBEAT_POLL_MS) @@ -208,14 +214,32 @@ save_and_set_hcc_cadences() { restore_hcc_cadences() { [ -n "$HCC_ENV_SAVED" ] || return 0 log "restoring HCC cadence env" - local args=() line k v + local args=() line k v actual while IFS= read -r line; do [ -z "$line" ] && continue k="${line%%=*}"; v="${line#*=}" if [ -n "$v" ]; then args+=("${k}=${v}"); else args+=("${k}-"); fi done <<< "$HCC_ENV_SAVED" - [ ${#args[@]} -gt 0 ] && kctl set env "deployment/${HCC_DEPLOY}" -n "$HCC_NS" "${args[@]}" >/dev/null 2>&1 || true - kctl rollout status "deployment/${HCC_DEPLOY}" -n "$HCC_NS" --timeout=120s >/dev/null 2>&1 || true + if [ ${#args[@]} -eq 0 ]; then return 0; fi + if ! kctl set env "deployment/${HCC_DEPLOY}" -n "$HCC_NS" "${args[@]}" >/dev/null 2>&1; then + fail "failed to restore HCC cadence env" + return 1 + fi + if ! kctl rollout status "deployment/${HCC_DEPLOY}" -n "$HCC_NS" --timeout=120s >/dev/null 2>&1; then + fail "HCC rollout did not settle after cadence restore" + return 1 + fi + while IFS= read -r line; do + [ -z "$line" ] && continue + k="${line%%=*}"; v="${line#*=}" + actual="$(kctl get "deployment/${HCC_DEPLOY}" -n "$HCC_NS" \ + -o jsonpath="{.spec.template.spec.containers[0].env[?(@.name=='${k}')].value}" 2>/dev/null || true)" + if [ "$actual" != "$v" ]; then + fail "HCC cadence restore readback mismatch for ${k}: expected '${v}', got '${actual}'" + return 1 + fi + done <<< "$HCC_ENV_SAVED" + HCC_ENV_SAVED="" } restore_cron_allow_env() { @@ -255,7 +279,7 @@ cleanup() { local rc=$? stop_active_cron_pin "$CRON_POD_UID" >/dev/null 2>&1 || true restore_cron_allow_env - restore_hcc_cadences + restore_hcc_cadences || rc=1 print_results exit "$rc" } diff --git a/scripts/e2e/e2e-stateless-suspend-wake.sh b/scripts/e2e/e2e-stateless-suspend-wake.sh index 937543a298..d6cfa29808 100755 --- a/scripts/e2e/e2e-stateless-suspend-wake.sh +++ b/scripts/e2e/e2e-stateless-suspend-wake.sh @@ -53,6 +53,8 @@ # CONTEXT_MAPPER_STATELESS_IDLE_MINUTES = 1 (minimum legal T_idle; tracker rejects 0) # CONTEXT_MAPPER_STATELESS_DRAIN_GRACE_MS = 20000 # CONTEXT_MAPPER_HEARTBEAT_POLL_MS = 5000 +# Acceleration is opt-in with E2E_ALLOW_STATELESS_CADENCE_ACCELERATION=1, and +# cleanup fails loud if kubectl set env, rollout, or exact readback fails. # # Prereqs (each a HARD FAIL with the concrete reason): # - Host CR chatllm-stateless with spec.lifecycle.stateless=true + pod Ready @@ -61,7 +63,8 @@ # - host image present on the node (pod already Ready proves this) # # Usage: -# KUBECONTEXT=clerum-test bash scripts/e2e/e2e-stateless-suspend-wake.sh +# E2E_ALLOW_STATELESS_CADENCE_ACCELERATION=1 \ +# KUBECONTEXT=clerum-test bash scripts/e2e/e2e-stateless-suspend-wake.sh # E2E_SKIP_DESKTOP_PHASE=1 ... # opt out of the Playwright phase (exit 3) # ====================================================================== set -euo pipefail @@ -128,17 +131,31 @@ STATELESS_DISABLED=0 restore_hcc_env() { [ -n "$HCC_ENV_SAVED" ] || return 0 - local args=() key val + local args=() key val actual while IFS='=' read -r key val; do [ -n "$key" ] || continue if [ -n "$val" ]; then args+=("${key}=${val}"); else args+=("${key}-"); fi done <<< "$HCC_ENV_SAVED" [ ${#args[@]} -gt 0 ] || return 0 log "Restoring HCC cadence env on deployment/${HCC_DEPLOY}" - kctl set env "deployment/${HCC_DEPLOY}" -n "$HCC_NS" "${args[@]}" >/dev/null 2>&1 || \ - warn "failed to restore HCC env (manual check advised)" - kctl rollout status "deployment/${HCC_DEPLOY}" -n "$HCC_NS" --timeout=180s >/dev/null 2>&1 || \ + if ! kctl set env "deployment/${HCC_DEPLOY}" -n "$HCC_NS" "${args[@]}" >/dev/null 2>&1; then + warn "failed to restore HCC env (manual check required)" + return 1 + fi + if ! kctl rollout status "deployment/${HCC_DEPLOY}" -n "$HCC_NS" --timeout=180s >/dev/null 2>&1; then warn "HCC rollout did not settle after env restore" + return 1 + fi + while IFS='=' read -r key val; do + [ -n "$key" ] || continue + actual="$(kctl get "deployment/${HCC_DEPLOY}" -n "$HCC_NS" \ + -o jsonpath="{.spec.template.spec.containers[0].env[?(@.name=='${key}')].value}" 2>/dev/null || true)" + if [ "$actual" != "$val" ]; then + warn "HCC restore readback mismatch for ${key}: expected '${val}', got '${actual}'" + return 1 + fi + done <<< "$HCC_ENV_SAVED" + HCC_ENV_SAVED="" } cleanup_on_exit() { @@ -172,7 +189,7 @@ cleanup_on_exit() { --ignore-not-found --grace-period=0 --force >/dev/null 2>&1 fi if [ -n "$CORDONED_NODE" ]; then kctl uncordon "$CORDONED_NODE" >/dev/null 2>&1; fi - restore_hcc_env + restore_hcc_env || status=1 exit "$status" } trap cleanup_on_exit EXIT @@ -554,6 +571,10 @@ force_idle_and_suspend() { } save_and_set_hcc_cadences() { + if [ "${E2E_ALLOW_STATELESS_CADENCE_ACCELERATION:-0}" != 1 ]; then + fail "refusing to accelerate stateless lifecycle cadences; set E2E_ALLOW_STATELESS_CADENCE_ACCELERATION=1 on an owned profile" + return 1 + fi header "Setting HCC test cadences (idle-immediate, drain=${TEST_DRAIN_GRACE_MS}ms, poll=${TEST_POLL_MS}ms)" local keys=(CONTEXT_MAPPER_STATELESS_IDLE_MINUTES CONTEXT_MAPPER_STATELESS_IDLE_FLOOR_MINUTES \ CONTEXT_MAPPER_STATELESS_DRAIN_GRACE_MS CONTEXT_MAPPER_HEARTBEAT_POLL_MS) diff --git a/scripts/e2e/e2e-stateless-wake-recovery.sh b/scripts/e2e/e2e-stateless-wake-recovery.sh index 4f8dea47a7..759dd8749c 100755 --- a/scripts/e2e/e2e-stateless-wake-recovery.sh +++ b/scripts/e2e/e2e-stateless-wake-recovery.sh @@ -81,6 +81,11 @@ # measured_total_ms, infra_overhead_ms}]}} # R4 additionally emits prewarm_to_ready_ms:[ms] -- the per-cycle overlap # series (informational, no budget). +# R2 additionally emits cold_resume_baseline:[{cycle, first_post_status, +# admission_attempts, first_200_ms, status_flipped_to_replicas_patched_ms}] +# -- characterization-only admission/scale data for later before/after +# comparison. It is informational, has no budget, and never replaces the +# existing ms gate metric. # Each measured turn (baseline, R1, R2, R4) is attributed with the serving # pod's [TurnTiming] phase line (queue_wait/session_load/prompt_assembly/ # llm_wall/llm_calls/tool_loop/tools_called/input_chars_approx) and @@ -119,6 +124,9 @@ # CONTEXT_MAPPER_STATELESS_IDLE_MINUTES = 1 # CONTEXT_MAPPER_STATELESS_DRAIN_GRACE_MS = 20000 # CONTEXT_MAPPER_HEARTBEAT_POLL_MS = 5000 +# Acceleration is opt-in with E2E_ALLOW_STATELESS_CADENCE_ACCELERATION=1. +# Cleanup fails loud if kubectl set env, rollout, or exact readback fails; +# a nominal test result can never hide a leaked one-minute profile policy. # Reaching state=draining after a served turn therefore needs idle # floor (60s) + one emitter tick (30s) + one HCC poll (5s) + slack => # DRAINING_WAIT default 150s. Full suspend adds drain grace (20s) => @@ -134,7 +142,8 @@ # - state.db row observable resolvable (exactly-once assertion in R3) # # Usage: -# KUBECONTEXT=clerum-test bash scripts/e2e/e2e-stateless-wake-recovery.sh +# E2E_ALLOW_STATELESS_CADENCE_ACCELERATION=1 \ +# KUBECONTEXT=clerum-test bash scripts/e2e/e2e-stateless-wake-recovery.sh # RECOVERY_CYCLES=5 WARM_RECOVERY_BUDGET_MS=3000 ... # ====================================================================== set -euo pipefail @@ -142,6 +151,13 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # shellcheck source=scripts/e2e/e2e-lib.sh source "${SCRIPT_DIR}/e2e-lib.sh" +# Resolve the canonical seeded credential in-process before any cluster +# mutation. An explicit E2E_USER_PASSWORD remains the journey override. +# shellcheck source=scripts/e2e/load-dotenv.sh +source "${SCRIPT_DIR}/load-dotenv.sh" +dotenv_load_canonical_root "${SCRIPT_DIR}/.." +# shellcheck source=scripts/e2e/admin-credentials.sh +source "${SCRIPT_DIR}/admin-credentials.sh" require_safe_kube_context HOST_REF="${E2E_STATELESS_HOST_REF:-chatllm-stateless}" @@ -150,7 +166,15 @@ RPC_BASE="${RPC_PROXY_BASE_URL:-http://127.0.0.1:8094}" CONTROL_BASE="${CONTROL_API_BASE_URL:-http://127.0.0.1:8090}" WAKE_BASE="${RPC_GATEWAY_BASE_URL:-${CONTROL_BASE}}" DEV_EMAIL="${E2E_DEV_LOGIN_EMAIL:-test@clerum.io}" -DEV_PASSWORD="${E2E_USER_PASSWORD:-${ADMIN_PASSWORD:-changeme123!}}" +if [[ -z "${E2E_USER_PASSWORD:-}" ]]; then + DEV_PASSWORD="$(e2e_resolve_admin_password "${SCRIPT_DIR}/.." "" || true)" + if [[ -z "${DEV_PASSWORD}" ]]; then + echo "FAIL: canonical E2E credential is unavailable; set E2E_USER_PASSWORD explicitly" >&2 + exit 2 + fi +else + DEV_PASSWORD="${E2E_USER_PASSWORD}" +fi E2E_TURN_TIMEOUT="${E2E_TURN_TIMEOUT:-120}" POD_READY_TIMEOUT="${POD_READY_TIMEOUT:-180}" @@ -165,6 +189,7 @@ WARM_RECOVERY_BUDGET_MS="${WARM_RECOVERY_BUDGET_MS:-5000}" R4_WARM_INFRA_BUDGET_MS="${R4_WARM_INFRA_BUDGET_MS:-1500}" # warm-class infra ceiling for the R4 prewarm claim (warm ~100-250ms, in-band cold ~17000ms) COLD_RECOVERY_BUDGET_MS="${COLD_RECOVERY_BUDGET_MS:-120000}" RECOVERY_ARTIFACT="${RECOVERY_ARTIFACT:-/tmp/stateless-wake-recovery-$(date +%s).json}" +mkdir -p -- "$(dirname -- "$RECOVERY_ARTIFACT")" TURN_TIMING_WAIT="${TURN_TIMING_WAIT:-15}" # bounded wait for the serving pod's [TurnTiming] line after a 200 LLM_EMPTY_MAX_RETRIES="${LLM_EMPTY_MAX_RETRIES:-3}" # bounded re-sends of a turn that 200s with an empty-LLM body (glm-4.7 provider flake, orthogonal to wake) @@ -185,24 +210,38 @@ HCC_ENV_SAVED="" restore_hcc_env() { [ -n "$HCC_ENV_SAVED" ] || return 0 - local args=() key val + local args=() key val actual while IFS='=' read -r key val; do [ -n "$key" ] || continue if [ -n "$val" ]; then args+=("${key}=${val}"); else args+=("${key}-"); fi done <<< "$HCC_ENV_SAVED" [ ${#args[@]} -gt 0 ] || return 0 log "Restoring HCC cadence env on deployment/${HCC_DEPLOY}" - kctl set env "deployment/${HCC_DEPLOY}" -n "$HCC_NS" "${args[@]}" >/dev/null 2>&1 || \ - warn "failed to restore HCC env (manual check advised)" - kctl rollout status "deployment/${HCC_DEPLOY}" -n "$HCC_NS" --timeout=180s >/dev/null 2>&1 || \ + if ! kctl set env "deployment/${HCC_DEPLOY}" -n "$HCC_NS" "${args[@]}" >/dev/null 2>&1; then + warn "failed to restore HCC env (manual check required)" + return 1 + fi + if ! kctl rollout status "deployment/${HCC_DEPLOY}" -n "$HCC_NS" --timeout=180s >/dev/null 2>&1; then warn "HCC rollout did not settle after env restore" + return 1 + fi + while IFS='=' read -r key val; do + [ -n "$key" ] || continue + actual="$(kctl get "deployment/${HCC_DEPLOY}" -n "$HCC_NS" \ + -o jsonpath="{.spec.template.spec.containers[0].env[?(@.name=='${key}')].value}" 2>/dev/null || true)" + if [ "$actual" != "$val" ]; then + warn "HCC restore readback mismatch for ${key}: expected '${val}', got '${actual}'" + return 1 + fi + done <<< "$HCC_ENV_SAVED" + HCC_ENV_SAVED="" } cleanup_on_exit() { local status=$? set +e rm -f "$CYCLES_FILE" >/dev/null 2>&1 - restore_hcc_env + restore_hcc_env || status=1 exit "$status" } trap cleanup_on_exit EXIT @@ -234,7 +273,7 @@ wait_for_ready_pod() { # --- Row observable (exactly-once marker): mirrors the durability exemplar DB_PATH="" ROW_TOOL="" -NODE_COUNT_SNIPPET='const D=require("/app/node_modules/better-sqlite3");const db=new D(process.env.DB,{readonly:true});db.pragma("busy_timeout=5000");const row=db.prepare(process.env.SQL).get();console.log(row?Object.values(row)[0]:0);' +NODE_COUNT_SNIPPET='const D=require("better-sqlite3");const db=new D(process.env.DB,{readonly:true});db.pragma("busy_timeout=5000");const row=db.prepare(process.env.SQL).get();console.log(row?Object.values(row)[0]:0);' resolve_row_tool() { local pod dbdir @@ -244,7 +283,7 @@ resolve_row_tool() { DB_PATH="${dbdir%/}/state.db" if kctl exec "$pod" -n "$MCP_HOST_NS" -- sh -c 'command -v sqlite3' >/dev/null 2>&1; then ROW_TOOL="sqlite3-cli" - elif kctl exec "$pod" -n "$MCP_HOST_NS" -- sh -c 'test -d /app/node_modules/better-sqlite3' >/dev/null 2>&1; then + elif kctl exec "$pod" -n "$MCP_HOST_NS" -- node -e 'require("better-sqlite3")' >/dev/null 2>&1; then ROW_TOOL="node-better-sqlite3" else echo "no row-count reader in pod (${pod}): neither sqlite3 CLI nor better-sqlite3" >&2; return 1 @@ -319,9 +358,13 @@ send_turn_raw() { # hard failure at the caller — an unreachable assertion never passes. RECOVERY_MS="" TURN_T0_RFC3339="" +ADMISSION_FIRST_STATUS="" +ADMISSION_ATTEMPTS="" measure_recovery_turn() { local content=$1 t0 t1 deadline attempt=0 RECOVERY_MS="" + ADMISSION_FIRST_STATUS="" + ADMISSION_ATTEMPTS="" mint_rpc_token || { echo "could not mint RPC token before t0" >&2; return 1; } TURN_T0_RFC3339="$(now_rfc3339)" t0="$(now_ms)" @@ -329,9 +372,11 @@ measure_recovery_turn() { while [ "$SECONDS" -lt "$deadline" ]; do attempt=$((attempt + 1)) send_turn_raw "$content" || return 1 + if [ "$attempt" -eq 1 ]; then ADMISSION_FIRST_STATUS="$TURN_STATUS"; fi if [ "$TURN_STATUS" = "200" ]; then t1="$(now_ms)" RECOVERY_MS=$((t1 - t0)) + ADMISSION_ATTEMPTS="$attempt" return 0 fi if [ "$TURN_STATUS" = "503" ] && echo "$TURN_BODY" | grep -qE 'host_waking|host_draining'; then @@ -490,6 +535,10 @@ force_idle_and_suspend() { } save_and_set_hcc_cadences() { + if [ "${E2E_ALLOW_STATELESS_CADENCE_ACCELERATION:-0}" != 1 ]; then + fail "refusing to accelerate stateless lifecycle cadences; set E2E_ALLOW_STATELESS_CADENCE_ACCELERATION=1 on an owned profile" + return 1 + fi header "PRECONDITION (labeled setup) -- HCC test cadences (idle=1min, drain=${TEST_DRAIN_GRACE_MS}ms, poll=${TEST_POLL_MS}ms)" local keys=(CONTEXT_MAPPER_STATELESS_IDLE_MINUTES CONTEXT_MAPPER_STATELESS_IDLE_FLOOR_MINUTES \ CONTEXT_MAPPER_STATELESS_DRAIN_GRACE_MS CONTEXT_MAPPER_HEARTBEAT_POLL_MS) @@ -535,11 +584,11 @@ drain_window_signal() { } record_cycle() { - local scenario=$1 cycle=$2 ms=$3 resolution=$4 wake_lines=$5 timing_json=${6:-null} prewarm_ms=${7:-null} + local scenario=$1 cycle=$2 ms=$3 resolution=$4 wake_lines=$5 timing_json=${6:-null} prewarm_ms=${7:-null} baseline_json=${8:-null} prewarm_first_ms=${9:-null} jq -cn --arg s "$scenario" --argjson c "$cycle" --argjson ms "$ms" \ --arg r "$resolution" --arg wl "$wake_lines" --argjson tt "$timing_json" \ - --argjson pw "$prewarm_ms" \ - '{scenario:$s, cycle:$c, ms:$ms, resolution:$r, wake_phases:($wl | split("\n") | map(select(length>0))), turn_timing:$tt, prewarm_to_ready_ms:$pw}' \ + --argjson pw "$prewarm_ms" --argjson ab "$baseline_json" --argjson pf "$prewarm_first_ms" \ + '{scenario:$s, cycle:$c, ms:$ms, resolution:$r, wake_phases:($wl | split("\n") | map(select(length>0))), turn_timing:$tt, prewarm_to_ready_ms:$pw, prewarm_first_post_to_ready_ms:$pf, admission_baseline:$ab}' \ >> "$CYCLES_FILE" } @@ -851,6 +900,16 @@ while [ "$cycle" -le "$RECOVERY_CYCLES" ]; do fail "R2 cycle ${cycle}: turn to suspended host did not reach a definitive 200" pod_diagnostics; print_results; exit 1 fi + # Characterization baseline for a later, separately approved proxy + # admission change: first POST HTTP status, POST count through the first + # 200, and the first-200 send->200 latency. first_200_ms is deliberately + # the same first-200 measurement as RECOVERY_MS (LLM-empty re-sends never + # touch it); it is repeated here so the baseline record is self-contained. + r2_baseline_json="$(jq -cn \ + --arg first_status "$ADMISSION_FIRST_STATUS" \ + --argjson attempts "$ADMISSION_ATTEMPTS" \ + --argjson first_200_ms "$RECOVERY_MS" \ + '{first_post_status:$first_status, admission_attempts:$attempts, first_200_ms:$first_200_ms}')" assert_success_response "R2 cycle ${cycle} turn" "Reply with exactly: ${r2_marker}" || { print_results; exit 1; } if wait_for_state "active" "$ACTIVE_WAIT"; then ok "R2 cycle ${cycle}: state=active after wake" @@ -867,6 +926,24 @@ while [ "$cycle" -le "$RECOVERY_CYCLES" ]; do fail "R2 cycle ${cycle}: expected a NEW pod after suspend, got uid '${r2_uid_after:-}' (before ${r2_uid_before})" pod_diagnostics; print_results; exit 1 fi + # Kubernetes-clock timeline for the NEW pod: creation, scheduling, + # initialization, container start, and Ready publication. Deltas are + # computed server-side only below; send-aligned cross-clock subtraction is + # deliberately avoided because client and apiserver clocks are not proven + # synchronized. + r2_pod_timeline="$(kctl get pod "$r2_pod_after" -n "$MCP_HOST_NS" -o json | jq -c '{ + pod_uid: .metadata.uid, + created_at: .metadata.creationTimestamp, + scheduled_at: (.status.conditions[]? | select(.type == "PodScheduled") | .lastTransitionTime), + initialized_at: (.status.conditions[]? | select(.type == "Initialized") | .lastTransitionTime), + ready_at: (.status.conditions[]? | select(.type == "Ready") | .lastTransitionTime), + container_started_at: (.status.containerStatuses[]? | select(.name == "mcp-host") | .state.running.startedAt) + }' 2>/dev/null || true)" + if [ -n "$r2_pod_timeline" ]; then + r2_baseline_json="$(jq -cn --argjson b "$r2_baseline_json" --argjson t "$r2_pod_timeline" '$b + {pod_timeline:$t}')" + else + r2_baseline_json="$(jq -cn --argjson b "$r2_baseline_json" '$b + {pod_timeline_unavailable_reason:"pod timeline read failed"}')" + fi # ATTRIBUTION -- the NEW pod served the resume; its log stream carries the # [TurnTiming] line for the measured turn. if capture_turn_attribution "R2 cycle ${cycle}" "$r2_pod_after"; then @@ -888,7 +965,7 @@ while [ "$cycle" -le "$RECOVERY_CYCLES" ]; do fail "R2 cycle ${cycle}: no [StatelessWake] phase=wake_observed line since ${r2_since} -- the wake path is unproven for this cycle" pod_diagnostics; print_results; exit 1 fi - record_cycle "R2_cold_resume_from_suspended" "$cycle" "$RECOVERY_MS" "cold" "$r2_wake_lines" "$r2_attr" + record_cycle "R2_cold_resume_from_suspended" "$cycle" "$RECOVERY_MS" "cold" "$r2_wake_lines" "$r2_attr" null "$r2_baseline_json" cycle=$((cycle + 1)) done @@ -1000,11 +1077,23 @@ while [ "$cycle" -le "$RECOVERY_CYCLES" ]; do # so the suite's check count stays deterministic. r4_prewarm_attempt=1 r4_prewarm_acked="" + r4_first_ready_file="$(mktemp "${TMPDIR:-/tmp}/r4-first-ready.XXXXXX")" + r4_prewarm_observer_started="" while :; do post_prewarm || { fail "R4 cycle ${cycle}: prewarm POST failed at ${RPC_BASE} (transport-level)"; pod_diagnostics; print_results; exit 1; } case "$PREWARM_STATUS" in 202) if [ "$r4_prewarm_attempt" -eq 1 ]; then + # Observe readiness from the FIRST accepted wake POST, concurrently + # with the bounded re-emission loop below. The legacy + # prewarm_to_ready_ms remains comparable but is censored by any + # mandatory 10s re-emission wait; this observer is not. + ( + r4_observed_pod="$(wait_for_ready_pod "$POD_READY_TIMEOUT")" && \ + printf '%s\t%s\n' "$(now_ms)" "$r4_observed_pod" > "$r4_first_ready_file" + ) & + r4_prewarm_observer=$! + r4_prewarm_observer_started=1 ok "R4 cycle ${cycle}: prewarm accepted (HTTP 202 wake-requested)" else log "R4 cycle ${cycle}: prewarm re-emission ${r4_prewarm_attempt}/3 still 202 -- HCC has not acted (possible lost watch event); annotation re-projected" @@ -1031,13 +1120,30 @@ while [ "$cycle" -le "$RECOVERY_CYCLES" ]; do sleep 10 r4_prewarm_attempt=$((r4_prewarm_attempt + 1)) done + if [ -n "$r4_prewarm_observer_started" ]; then + wait "$r4_prewarm_observer" || { + fail "R4 cycle ${cycle}: concurrent prewarm observer did not produce a Ready pod within ${POD_READY_TIMEOUT}s"; pod_diagnostics; print_results; exit 1; } + IFS=$'\t' read -r r4_first_ready_at r4_pod_warm < "$r4_first_ready_file" + else + # First attempt already returned active (wake race): no uncensored cold + # start exists in this cycle, so record it as unavailable rather than + # inventing one. + r4_pod_warm=$(wait_for_ready_pod "$POD_READY_TIMEOUT") || { + fail "R4 cycle ${cycle}: prewarm did not produce a Ready pod within ${POD_READY_TIMEOUT}s"; pod_diagnostics; print_results; exit 1; } + r4_first_ready_at="" + fi + rm -f "$r4_first_ready_file" # READINESS (OVERLAP time -- models the user reading history / typing # after opening the agent view; NOT the user-perceived wait): the cold - # infra start runs while no message is pending. Recorded per cycle as - # prewarm_to_ready_ms -- informational, no budget. - r4_pod_warm=$(wait_for_ready_pod "$POD_READY_TIMEOUT") || { - fail "R4 cycle ${cycle}: prewarm did not produce a Ready pod within ${POD_READY_TIMEOUT}s"; pod_diagnostics; print_results; exit 1; } + # infra start runs while no message is pending. prewarm_to_ready_ms is the + # legacy censored series; prewarm_first_post_to_ready_ms is the uncensored + # first-POST-to-Ready measurement. Both are informational, no budget. r4_prewarm_ms=$(( $(now_ms) - r4_prewarm_t0 )) + if [ -n "$r4_first_ready_at" ]; then + r4_prewarm_first_ms=$(( r4_first_ready_at - r4_prewarm_t0 )) + else + r4_prewarm_first_ms="" + fi ok "R4 cycle ${cycle}: prewarmed pod Ready in ${r4_prewarm_ms}ms (overlap time hidden from the user -- informational, no budget)" r4_uid_warm="$(pod_uid "$r4_pod_warm")" [ -n "$r4_uid_warm" ] || { fail "R4 cycle ${cycle}: could not read podUid for prewarmed pod ${r4_pod_warm}"; print_results; exit 1; } @@ -1092,7 +1198,7 @@ while [ "$cycle" -le "$RECOVERY_CYCLES" ]; do fail "R4 cycle ${cycle}: infra overhead ${r4_infra_ms}ms > R4_WARM_INFRA_BUDGET_MS=${R4_WARM_INFRA_BUDGET_MS}ms -- an in-band cold start leaked into the measured turn" pod_diagnostics; print_results; exit 1 fi - record_cycle "R4_prewarm_first_message" "$cycle" "$RECOVERY_MS" "warm" "" "$TURN_ATTRIBUTION_JSON" "$r4_prewarm_ms" + record_cycle "R4_prewarm_first_message" "$cycle" "$RECOVERY_MS" "warm" "" "$TURN_ATTRIBUTION_JSON" "$r4_prewarm_ms" null "${r4_prewarm_first_ms:-null}" cycle=$((cycle + 1)) done @@ -1181,12 +1287,106 @@ for name, budget in budgets.items(): continue attribution.append({"cycle": r["cycle"], **tt}) breakdown = [] + baseline = [] for r in recs: parsed = [] for line in r.get("wake_phases", []): m = wake_re.search(line) if m: parsed.append({"generation": m.group(1), "phase": m.group(2), "ts_ms": int(m.group(3))}) + if name == "R2_cold_resume_from_suspended": + raw = r.get("admission_baseline") or {} + for key in ("first_post_status", "admission_attempts", "first_200_ms"): + if key not in raw: + print( + f"scenario {name} cycle {r['cycle']}: admission baseline field '{key}' is missing -- refusing to emit partial baseline data", + file=sys.stderr, + ) + raise SystemExit(1) + # Pair the two HCC phases by wake generation. A missing phase or + # multiple candidate generations stays null with a reason: the + # baseline must characterize the observed signal, never invent it. + generations = {} + for phase in parsed: + generations.setdefault(phase["generation"], {})[phase["phase"]] = phase["ts_ms"] + paired = [ + phases["replicas_patched"] - phases["status_flipped"] + for phases in generations.values() + if "status_flipped" in phases and "replicas_patched" in phases + ] + baseline_entry = { + "cycle": r["cycle"], + "first_post_status": raw["first_post_status"], + "admission_attempts": raw["admission_attempts"], + "first_200_ms": raw["first_200_ms"], + "status_flipped_to_replicas_patched_ms": None, + } + timeline = raw.get("pod_timeline") or {} + if timeline: + baseline_entry["pod_timeline"] = timeline + + def _rfc3339_ms(value): + from datetime import datetime + + return int( + datetime.fromisoformat( + value.replace("Z", "+00:00") + ).timestamp() + * 1000 + ) + + marks = { + key: _rfc3339_ms(timeline[key]) + for key in ( + "created_at", + "scheduled_at", + "initialized_at", + "container_started_at", + "ready_at", + ) + if timeline.get(key) + } + ordered = [ + ("created_to_scheduled_ms", "created_at", "scheduled_at"), + ("scheduled_to_initialized_ms", "scheduled_at", "initialized_at"), + ( + "initialized_to_container_started_ms", + "initialized_at", + "container_started_at", + ), + ("container_started_to_ready_ms", "container_started_at", "ready_at"), + ] + deltas = {} + for label, start_key, end_key in ordered: + if start_key in marks and end_key in marks: + deltas[label] = marks[end_key] - marks[start_key] + if deltas: + baseline_entry["pod_timeline_deltas_ms"] = deltas + else: + baseline_entry["pod_timeline_deltas_unavailable_reason"] = ( + "pod timeline lacked a complete server-side phase pair" + ) + else: + baseline_entry["pod_timeline_deltas_unavailable_reason"] = raw.get( + "pod_timeline_unavailable_reason", + "pod timeline was not captured", + ) + if len(paired) == 1: + if paired[0] >= 0: + baseline_entry["status_flipped_to_replicas_patched_ms"] = paired[0] + else: + baseline_entry["status_flipped_to_replicas_patched_unavailable_reason"] = ( + f"negative timestamp delta ({paired[0]}ms) in [StatelessWake] lines" + ) + elif len(paired) == 0: + baseline_entry["status_flipped_to_replicas_patched_unavailable_reason"] = ( + "no [StatelessWake] generation in this cycle has both phase=status_flipped and phase=replicas_patched" + ) + else: + baseline_entry["status_flipped_to_replicas_patched_unavailable_reason"] = ( + f"{len(paired)} [StatelessWake] generations have both phases -- attribution is ambiguous" + ) + baseline.append(baseline_entry) if parsed: breakdown.append({"cycle": r["cycle"], "wake_phases": parsed}) # R4 overlap series (prewarm POST -> pod Ready). Informational, no @@ -1200,6 +1400,17 @@ for name, budget in budgets.items(): file=sys.stderr, ) raise SystemExit(1) + prewarm_first = [ + r["prewarm_first_post_to_ready_ms"] + for r in recs + if r.get("prewarm_first_post_to_ready_ms") is not None + ] + if name == "R4_prewarm_first_message" and len(prewarm_first) != expected: + print( + f"scenario {name}: expected {expected} uncensored prewarm_first_post_to_ready_ms values, got {len(prewarm_first)} -- refusing to emit a partial series", + file=sys.stderr, + ) + raise SystemExit(1) p95 = percentile(values, 95) if name == "R4_prewarm_first_message": # The R4 gate metric is warm-class infra overhead, immune to LLM @@ -1235,6 +1446,12 @@ for name, budget in budgets.items(): entry["turn_attribution"] = attribution if prewarm: entry["prewarm_to_ready_ms"] = prewarm + if prewarm_first: + entry["prewarm_first_post_to_ready_ms"] = prewarm_first + if baseline: + # Informational characterization only: no budget and no effect on + # the existing R2 recovery verdict. + entry["cold_resume_baseline"] = baseline scenarios[name] = entry artifact = { diff --git a/scripts/e2e/minikube-real-postgres.sh b/scripts/e2e/minikube-real-postgres.sh index 4f8f1a1c2f..a9ceeb5ee1 100755 --- a/scripts/e2e/minikube-real-postgres.sh +++ b/scripts/e2e/minikube-real-postgres.sh @@ -266,6 +266,8 @@ sanitize_file() { [ -f "$file" ] || return 0 T1_REDACT_PASSWORD="${T1_REDACT_PASSWORD:-${PG_PASSWORD:-}}" python3 - "$file" <<'PY' from pathlib import Path +from urllib.parse import quote +import json import os import re import sys @@ -273,9 +275,44 @@ path = Path(sys.argv[1]) text = path.read_text(errors="replace") password = os.environ.get("T1_REDACT_PASSWORD", "") if password: - text = text.replace(password, "") -text = re.sub(r"postgres(?:ql)?://[^\s\"'<>]+", "", text) -text = re.sub(r"(?i)(authorization:\s*bearer\s+)[^\s]+", r"\1", text) + # Every spelling the password can take in a log or in the Vitest JSON + # reporter: verbatim, JSON-escaped once (a reporter message) and twice (a + # JSON document quoted inside a message) in both ensure_ascii spellings + # (JavaScript's JSON.stringify keeps non-ASCII literal; Python's default + # writes \uXXXX), percent-encoded by Python's quote() and by JavaScript's + # encodeURIComponent, which keeps !~*'() literal. + escaped = [] + for ensure_ascii in (True, False): + once = json.dumps(password, ensure_ascii=ensure_ascii)[1:-1] + escaped += [once, json.dumps(once, ensure_ascii=ensure_ascii)[1:-1]] + spellings = ( + password, + *escaped, + quote(password, safe=""), + quote(password, safe="-_.!~*'()"), + ) + # Longest first so a spelling that contains another is replaced whole. + for spelling in sorted(set(spellings), key=len, reverse=True): + text = text.replace(spelling, "") +# A DSN loses its scheme-to-host span, credentials included, whatever the +# password is. The scheme is case-insensitive and the userinfo runs to the first +# "@", including the JSON escapes (\" and \\) a reporter message adds inside a +# password. The host part stops at a backslash: inside the JSON reporter a DSN +# is followed by the escape that closes its string (\"), and consuming it leaves +# invalid JSON. +text = re.sub( + r"(?i)postgres(?:ql)?://(?:(?:[^\s\"\\@]|\\[\"\\])*@)?[^\s\"'<>\\]+", + "", + text, +) +# libpq conninfo (password=...), quoted or bare. A bare value stops at a quote +# or a backslash so it cannot swallow the end of a JSON string. +text = re.sub( + r"(?i)(password\s*=\s*)(?:'(?:\\.|[^'\\])*'|[^\s\"'\\]+)", + r"\1", + text, +) +text = re.sub(r"(?i)(authorization:\s*bearer\s+)[^\s\"'\\]+", r"\1", text) path.write_text(text) PY } @@ -359,6 +396,49 @@ print("postgresql://postgres@127.0.0.1:" + port + "/postgres", end="") ')" } +# Print the reporter counters and every non-passing file and test. The file must +# already have gone through sanitize_file. A suite that fails before any test +# runs (beforeAll, import error) has no assertionResults: its only diagnosis is +# the file-level `message`, so it is printed right after the FAILED FILE line. +print_reporter_failure_details() { + python3 - "$1" <<'PY' +import json +import sys +from pathlib import Path + +result = json.loads(Path(sys.argv[1]).read_text()) +print( + "T1 reporter counters:", + { + key: result.get(key) + for key in ( + "success", + "numTotalTestSuites", + "numPassedTestSuites", + "numFailedTestSuites", + "numPendingTestSuites", + "numTotalTests", + "numPassedTests", + "numFailedTests", + "numPendingTests", + ) + }, +) +for test_result in result.get("testResults", []): + if test_result.get("status") == "passed": + continue + print(f"FAILED FILE: {test_result.get('name')}") + if test_result.get("message"): + print(test_result["message"]) + for assertion in test_result.get("assertionResults", []): + if assertion.get("status") == "passed": + continue + print(f"FAILED TEST: {assertion.get('fullName') or assertion.get('title')}") + for message in assertion.get("failureMessages", []): + print(message) +PY +} + run_suite() { local package="$1" lane="$2" admin_url="$3" local expected_files=0 reported_files=0 file log_file json_file stats @@ -473,6 +553,8 @@ PY if [ "$success" -ne 1 ] || [ "$total_suites" -le 0 ] || \ [ "$passed_suites" -ne "$total_suites" ] || [ "$failed_suites" -ne 0 ] || \ [ "$failed_tests" -ne 0 ]; then + cat "$log_file" >&2 || true + print_reporter_failure_details "$json_file" >&2 || true T1_NEXT_COMMAND='repair the failed or incomplete Real PostgreSQL lane, then re-run T1' die_t1 REAL_PG_SUITE_FAILED "Real PostgreSQL reporter did not pass every suite in $package ($lane)" fi @@ -521,6 +603,8 @@ main() { T1_NEXT_COMMAND="MINIKUBE_PROFILE=$T2_PROFILE make minikube-setup-local" die_t1 POSTGRES_NOT_READY 'control-postgres did not become Ready' fi + # Close the phase t2_evidence_init opened before the suites start. + t2_evidence_write preflight PASS 'marker, process and control-postgres readiness checks passed' T1_TMP_DIR="$(mktemp -d "$T1_TMP_ROOT/evenfire-t1.XXXXXX")" PG_USER="$(secret_field POSTGRES_USER)" diff --git a/scripts/e2e/real-postgres-local-preflight.sh b/scripts/e2e/real-postgres-local-preflight.sh index fa653dae9d..ba4acb104d 100755 --- a/scripts/e2e/real-postgres-local-preflight.sh +++ b/scripts/e2e/real-postgres-local-preflight.sh @@ -40,9 +40,12 @@ real_pg_local_preflight() { done node_major="$(node -p 'Number(process.versions.node.split(".")[0])' 2>/dev/null || true)" - if ! [[ "$node_major" =~ ^[0-9]+$ ]] || (( node_major < 24 )); then + # CI and the local runtime contract pin Node 24. Newer majors have already + # changed deep-JSON behavior and must fail before T0 rather than pass the + # old ">=24" check and break package suites mid-flight. + if ! [[ "$node_major" =~ ^[0-9]+$ ]] || (( node_major != 24 )); then real_pg_preflight_fail LOCAL_DEPENDENCY_MISSING \ - "Node.js >=24 is required for T1; found ${node_major:-unknown}" + "Node.js 24.x is required for the canonical T0/T1/T2 harness; found ${node_major:-unknown}" return 1 fi diff --git a/scripts/minikube/pre-gate-incremental.sh b/scripts/minikube/pre-gate-incremental.sh index 92b6ad4668..6d00f4196c 100755 --- a/scripts/minikube/pre-gate-incremental.sh +++ b/scripts/minikube/pre-gate-incremental.sh @@ -452,8 +452,8 @@ incremental_build_images() { log "Building all images because the change cannot be safely targeted" ( cd "${PROJECT_DIR}" - make minikube-build-images - make minikube-verify-images + T2_SKIP_LOCK=true T2_LOCK_TOKEN="${T2_LOCK_TOKEN}" make minikube-build-images + T2_SKIP_LOCK=true T2_LOCK_TOKEN="${T2_LOCK_TOKEN}" make minikube-verify-images ) return 0 fi @@ -486,8 +486,8 @@ incremental_build_images_ghcr() { log "Re-pulling the release image set (${IMAGE_TAG}) before shadowing the changed images" ( cd "${PROJECT_DIR}" - make minikube-pull-images - make minikube-verify-images + T2_SKIP_LOCK=true T2_LOCK_TOKEN="${T2_LOCK_TOKEN}" make minikube-pull-images + T2_SKIP_LOCK=true T2_LOCK_TOKEN="${T2_LOCK_TOKEN}" make minikube-verify-images ) fi diff --git a/scripts/minikube/t2.sh b/scripts/minikube/t2.sh index 4468ccc816..d9bec234cb 100755 --- a/scripts/minikube/t2.sh +++ b/scripts/minikube/t2.sh @@ -137,6 +137,9 @@ PY printf '[minikube-t2] transition=%s reason=%s\n' "$T2_PLAN_STATE" "$T2_PLAN_REASON" t2_evidence_write planner PASS \ "state=$T2_PLAN_STATE duration=$((SECONDS - phase_started_seconds))s" + # Close the phase t2_evidence_init opened: local-preflight and the planner + # have both passed, so this run's preflight is complete. + t2_evidence_write preflight PASS "local-preflight and planner passed; transition=$T2_PLAN_STATE" } run_pvc_reset_if_authorized() { diff --git a/scripts/tests/hcc-watch-api-proxy.test.mjs b/scripts/tests/hcc-watch-api-proxy.test.mjs index 3edfde08dc..a4c705541a 100644 --- a/scripts/tests/hcc-watch-api-proxy.test.mjs +++ b/scripts/tests/hcc-watch-api-proxy.test.mjs @@ -30,10 +30,108 @@ const allowedPath = '/api/v1/namespaces/mcp-server/services/runtime-witness' const mcpPath = '/apis/clerum.io/v1alpha1/namespaces/mcp-server/mcpservers?watch=true' const contextPath = '/apis/clerum.io/v1alpha1/namespaces/mcp-server/contexts?watch=true' const otherPath = '/api/v1/namespaces/mcp-server/pods?watch=true' +const channelPath = '/apis/clerum.io/v1alpha1/namespaces/channels/communicationchannels' let front let upstreamTls let publicConfig +test('channel hold accepts only a bounded duration', () => { + assert.throws(() => validateCommand({ id: 'hold', action: 'hold-channel', durationMs: 0 }, [])) + assert.throws(() => validateCommand({ id: 'hold', action: 'hold-channel', durationMs: 60001 }, [])) + assert.deepEqual(validateCommand({ id: 'hold', action: 'hold-channel', durationMs: 60000 }, []), + { id: 'hold', action: 'hold-channel', durationMs: 60000 }) +}) + +test('channel hold cuts only its watch and keeps Host API requests available', { timeout: 10000 }, async () => { + const controlDir = mkdtempSync(join(tmpdir(), 'hcc-channel-hold-')) + const seen = [] + const upstream = https.createServer(upstreamTls, (request, response) => { + seen.push(request.url) + if (request.url.includes('watch=true')) { + response.writeHead(200) + response.flushHeaders() + } else { + response.writeHead(200, { 'content-type': 'application/json' }) + response.end('{}') + } + }) + let proxy + let watchRequest + const get = path => new Promise((resolveResponse, rejectResponse) => { + https.get({ hostname: '127.0.0.1', port: proxy.server.address().port, path, + servername: 'proxy-fixture.test-fixture.svc', ca: front.cert }, response => { + response.resume() + response.once('end', () => resolveResponse(response.statusCode)) + }).once('error', rejectResponse) + }) + const ack = async (id, state) => { + for (let attempt = 0; attempt < 50; attempt++) { + if (existsSync(join(controlDir, 'ack.json'))) { + const value = JSON.parse(readFileSync(join(controlDir, 'ack.json'), 'utf8')) + if (value.id === id && value.state === state) return value + } + await new Promise(resolveWait => setTimeout(resolveWait, 20)) + } + throw new Error(`missing ${state} acknowledgement`) + } + try { + await new Promise(resolveListen => upstream.listen(0, '127.0.0.1', resolveListen)) + proxy = createProxy({ ...front, upstreamCa: upstreamTls.cert, + upstreamHost: '127.0.0.1', upstreamPort: upstream.address().port, + allowedPaths: [], controlDir, periodMs: 60000, minAgeMs: 60000, channelPath }) + await new Promise(resolveListen => proxy.server.listen(0, '127.0.0.1', resolveListen)) + await new Promise((resolveHeaders, rejectHeaders) => { + watchRequest = https.get({ hostname: '127.0.0.1', port: proxy.server.address().port, + path: `${channelPath}?watch=true`, servername: 'proxy-fixture.test-fixture.svc', + ca: front.cert }, resolveHeaders) + watchRequest.once('error', rejectHeaders) + }) + writeFileSync(join(controlDir, 'command.json'), JSON.stringify({ + id: 'hold-1', action: 'hold-channel', durationMs: 5000, + })) + const held = await ack('hold-1', 'held') + assert.equal(held.count, 1, 'live CommunicationChannel watch was cut') + const initialHold = JSON.parse(readFileSync(join(controlDir, 'channel-hold.json'), 'utf8')) + assert.equal(await get(channelPath), 503, 'channel LIST remains unavailable') + assert.equal(await get('/apis/clerum.io/v1alpha1/namespaces/mcp-host/hosts/test'), 200) + assert.equal(await get('/api/v1/namespaces/mcp-host/pods'), 200) + assert.equal(seen.filter(path => path.startsWith(channelPath)).length, 1, + 'blocked channel retries never reached upstream') + writeFileSync(join(controlDir, 'command.json'), JSON.stringify({ + id: 'hold-2', action: 'hold-channel', durationMs: 5000, + })) + const renewed = await ack('hold-2', 'held') + assert.equal(renewed.count, 1, 'renewal retains the original watch-cut witness') + assert.equal(renewed.renewed, true) + assert.equal(renewed.renewals, 1) + assert.ok(renewed.deadlineAtMs > initialHold.deadlineAtMs, + 'renewal extends the live hold deadline without a timing sleep') + const renewedHold = JSON.parse(readFileSync(join(controlDir, 'channel-hold.json'), 'utf8')) + assert.deepEqual(renewedHold, { + id: 'hold-2', + state: 'held', + cut: 1, + rejected: 1, + renewals: 1, + deadlineAtMs: renewed.deadlineAtMs, + }) + assert.equal(await get(channelPath), 503, 'channel LIST remains unavailable after renewal') + assert.equal(seen.filter(path => path.startsWith(channelPath)).length, 1, + 'renewed channel hold still blocks upstream channel requests') + writeFileSync(join(controlDir, 'command.json'), JSON.stringify({ + id: 'release-1', action: 'release-channel', + })) + await ack('release-1', 'released') + assert.equal(await get(channelPath), 200, 'authoritative LIST works after release') + } finally { + watchRequest?.destroy() + proxy?.close() + upstream.closeAllConnections() + await new Promise(resolveClose => upstream.close(resolveClose)) + rmSync(controlDir, { recursive: true, force: true }) + } +}) + test( 'quiet upstream WATCH headers reach the client before its first body event', { timeout: 5000 }, diff --git a/scripts/tests/test-autoresearch-gates.sh b/scripts/tests/test-autoresearch-gates.sh new file mode 100755 index 0000000000..334ec26229 --- /dev/null +++ b/scripts/tests/test-autoresearch-gates.sh @@ -0,0 +1,128 @@ +#!/usr/bin/env bash +set -u + +FAIL=0 +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd -P)" +BENCHMARK_WRAPPER="${ROOT}/scripts/dev/autoresearch-benchmark.sh" +CHECKS_WRAPPER="${ROOT}/scripts/dev/autoresearch-checks.sh" +HOST_HEAD="$(git -C "${ROOT}" rev-parse HEAD)" +HOST_BRANCH="$(git -C "${ROOT}" branch --show-current)" +HOST_STATUS="$(git -C "${ROOT}" status --porcelain=v1)" +TMP_ROOT="$(mktemp -d "${TMPDIR:-/tmp}/evenfire-autoresearch-gates.XXXXXX")" + +cleanup() { + local status=$? + rm -rf "${TMP_ROOT}" + if [[ "$(git -C "${ROOT}" rev-parse HEAD)" != "${HOST_HEAD}" || + "$(git -C "${ROOT}" branch --show-current)" != "${HOST_BRANCH}" || + "$(git -C "${ROOT}" status --porcelain=v1)" != "${HOST_STATUS}" ]]; then + echo 'FAIL: gate fixture changed host checkout' + status=1 + fi + exit "${status}" +} +trap cleanup EXIT + +pass() { echo "PASS: $1"; } +fail() { echo "FAIL: $1"; FAIL=1; } + +FIXTURE="${TMP_ROOT}/fixture" +MARKER="${TMP_ROOT}/benchmark-ran" +mkdir -p "${FIXTURE}" +git init -q "${FIXTURE}" +git -C "${FIXTURE}" checkout -q -b test/autoresearch-gates +git -C "${FIXTURE}" config user.email "fixture@example.invalid" +git -C "${FIXTURE}" config user.name "Evenfire Fixture" +git -C "${FIXTURE}" commit --allow-empty -qm base +git -C "${FIXTURE}" commit --allow-empty -qm candidate +git -C "${FIXTURE}" checkout -q -b fixture-origin-dev +git -C "${FIXTURE}" commit --allow-empty -qm "origin dev ahead" +ORIGIN_DEV_HEAD="$(git -C "${FIXTURE}" rev-parse HEAD)" +git -C "${FIXTURE}" checkout -q test/autoresearch-gates +git -C "${FIXTURE}" update-ref refs/remotes/origin/dev "${ORIGIN_DEV_HEAD}" + +OUTPUT="$(cd "${FIXTURE}" && bash "${BENCHMARK_WRAPPER}" -- bash -c "touch '${MARKER}'" 2>&1)" +STATUS=$? + +if [[ "${STATUS}" -eq 2 ]]; then + pass "stale origin/dev blocks the benchmark before execution" +else + fail "stale origin/dev blocks the benchmark before execution" + echo "${OUTPUT}" +fi + +if [[ ! -e "${MARKER}" ]]; then + pass "blocked benchmark command was not executed" +else + fail "blocked benchmark command was not executed" +fi + +if grep -Fq 'branch_missing_origin/dev_commits' <<<"${OUTPUT}"; then + pass "stale-base blocker is preserved in gate output" +else + fail "stale-base blocker is preserved in gate output" + echo "${OUTPUT}" +fi + +if ! grep -Fq 'AUTORESEARCH_GATE branch_freshness=pass' <<<"${OUTPUT}"; then + pass "blocked freshness gate does not report success" +else + fail "blocked freshness gate does not report success" +fi + +git -C "${FIXTURE}" update-ref refs/remotes/origin/dev "$(git -C "${FIXTURE}" rev-parse HEAD)" +FRESH_OUTPUT="$(cd "${FIXTURE}" && bash "${BENCHMARK_WRAPPER}" -- bash -c "touch '${MARKER}'")" +FRESH_STATUS=$? + +if [[ "${FRESH_STATUS}" -eq 0 ]]; then + pass "fresh origin/dev allows the benchmark" +else + fail "fresh origin/dev allows the benchmark" + echo "${FRESH_OUTPUT}" +fi + +if [[ -e "${MARKER}" ]]; then + pass "benchmark arguments pass through after the freshness gate" +else + fail "benchmark arguments pass through after the freshness gate" +fi + +if grep -Fq 'AUTORESEARCH_GATE branch_freshness=pass' <<<"${FRESH_OUTPUT}"; then + pass "freshness pass is explicit in gate output" +else + fail "freshness pass is explicit in gate output" + echo "${FRESH_OUTPUT}" +fi + +mkdir -p "${FIXTURE}/scripts/dev" "${FIXTURE}/.github/workflows" +cp "${ROOT}/Makefile" "${FIXTURE}/Makefile" +cp "${ROOT}/scripts/dev/check-test-services.cjs" "${FIXTURE}/scripts/dev/check-test-services.cjs" +cp "${ROOT}/.github/workflows/ci-public.yml" "${FIXTURE}/.github/workflows/ci-public.yml" + +CHECKS_OUTPUT="$(cd "${FIXTURE}" && bash "${CHECKS_WRAPPER}" 2>&1)" +CHECKS_STATUS=$? +if [[ "${CHECKS_STATUS}" -eq 0 ]] && grep -Fq 'AUTORESEARCH_GATE service_matrix_parity=pass' <<<"${CHECKS_OUTPUT}"; then + pass "matching CI and Makefile service matrices pass" +else + fail "matching CI and Makefile service matrices pass" + echo "${CHECKS_OUTPUT}" +fi + +sed 's/^ - channel-reader$/ - fixture-only/' \ + "${ROOT}/.github/workflows/ci-public.yml" >"${FIXTURE}/.github/workflows/ci-public.yml" +DRIFT_OUTPUT="$(cd "${FIXTURE}" && bash "${CHECKS_WRAPPER}" 2>&1)" +DRIFT_STATUS=$? +if [[ "${DRIFT_STATUS}" -ne 0 ]] && grep -Fq 'Test service drift:' <<<"${DRIFT_OUTPUT}"; then + pass "service-matrix drift blocks the checks gate" +else + fail "service-matrix drift blocks the checks gate" + echo "${DRIFT_OUTPUT}" +fi + +if ! grep -Fq 'AUTORESEARCH_GATE service_matrix_parity=pass' <<<"${DRIFT_OUTPUT}"; then + pass "blocked service-matrix gate does not report success" +else + fail "blocked service-matrix gate does not report success" +fi + +exit "${FAIL}" diff --git a/scripts/tests/test-hcc-stateless-cache-containment.sh b/scripts/tests/test-hcc-stateless-cache-containment.sh new file mode 100755 index 0000000000..e9ed9919c3 --- /dev/null +++ b/scripts/tests/test-hcc-stateless-cache-containment.sh @@ -0,0 +1,96 @@ +#!/usr/bin/env bash +# Static contract for the branch-owned selective watch gate. Runtime behavior +# of the TLS fault is exercised by hcc-watch-api-proxy.test.mjs. +set -euo pipefail +# shellcheck disable=SC2034,SC2329 # Fixture globals/functions are consumed by the sourced helper. +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +GATE="${ROOT}/scripts/e2e/e2e-hcc-stateless-cache-containment.sh" +PROXY="${ROOT}/scripts/e2e/_lib/hcc-watch-api-proxy.mjs" +PROXY_TEST="${ROOT}/scripts/tests/hcc-watch-api-proxy.test.mjs" +HELPER="${ROOT}/scripts/e2e/_lib/hcc-watch-pr-a.sh" +MAKEFILE="${ROOT}/Makefile" +WORKFLOW="${ROOT}/.github/workflows/ci-public.yml" + +bash -n "$GATE" "$HELPER" "$0" +node --check "$PROXY" +grep -Fq 'require_branch_owned_hcc_gate' "$GATE" +grep -Fq 'E2E_BRANCH_PROFILE_ENV' "$GATE" +grep -Fq 'E2E_PROFILE_PORTS_ENV' "$GATE" +grep -Fq 'profile_port_value CONTROL_API_URL' "$GATE" +grep -Fq 'acquire_hcc_watch_gate_lock' "$GATE" +# shellcheck disable=SC2016 # Match literal shell source. +grep -Fq 'finalize_hcc_watch_gate_lock "$cleanup_failed" "$restore_ok"' "$GATE" +grep -Fq 'hcc_pr_a_command hold-channel 60000' "$GATE" +grep -Fq 'hcc_pr_a_command release-channel' "$GATE" +grep -Fq 'wait_log_count '\''CommunicationChannel watch ended;' "$GATE" +grep -Fq 'CommunicationChannelCacheUnsynced' "$GATE" +# shellcheck disable=SC2016 # Match literal shell source. +grep -Fq 'host_runtime_is_always_on "$HOST_REF"' "$GATE" +grep -Fq 'jq -Sc '\''.spec.template' "$GATE" +grep -Fq 'baseline_pod' "$GATE" +grep -Fq 'CLERUM_SESSION_DB_DIR' "$GATE" +# shellcheck disable=SC2016 # Match literal shell source. +grep -Fq '/sessions/${HOST_REF}/${THREAD_ID}/messages' "$GATE" +grep -Fq '/sessions?agent=${HOST_REF}&limit=100' "$GATE" +grep -Fq 'session_visible || die "cycle' "$GATE" +grep -Fq 'session_listed || die "cycle' "$GATE" +grep -Fq '"host:wake:write"' "$GATE" +grep -Fq '/api/v1/rpc/hosts/${HOST_REF}/messages' "$GATE" +grep -Fq 'channel_hold_is_active' "$GATE" +[ "$(grep -Fc 'channel_hold_is_active "$HCC_PR_A_CHANNEL_HOLD_ID" ||' "$GATE")" -ge 6 ] +[ "$(grep -Fc 'hcc_pr_a_command hold-channel 60000' "$GATE")" -ge 2 ] +grep -Fq 'MAX_CHANNEL_HOLD_BUDGET_MS = 120_000' "$PROXY" +grep -Fq 'channelHold.renewals++' "$PROXY" +grep -Fq 'deadlineAtMs' "$PROXY" +grep -Fq "renewal extends the live hold deadline without a timing sleep" "$PROXY_TEST" +grep -Fq "renewed channel hold still blocks upstream channel requests" "$PROXY_TEST" +grep -Fq 'HCC_PR_A_SELECTIVE_CHANNEL=1 hcc_pr_a_enable_proxy' "$GATE" +grep -Fq 'response.writeHead(503' "$PROXY" +grep -Fq 'url.pathname === channelPath' "$PROXY" +grep -Fq 'CONTROL_UI_BASE_URL="$(profile_port_value CONTROL_UI_URL)"' "$GATE" +grep -Fq 'minikube-t2-hcc-stateless-cache-containment' "$MAKEFILE" +grep -Fq 'T2_HEALTHCHECK_COMMAND=' "$MAKEFILE" +grep -Fq 'MINIKUBE_PROFILE="$$T2_PROFILE"' "$MAKEFILE" +grep -Fq 'T2_GATE_ID=minikube-t2' "$MAKEFILE" +grep -Fq 'MINIKUBE_PROFILE)" = "$$context"' "$MAKEFILE" +grep -Fq 'E2E_EXPECTED_PRE_GATE_GATE=minikube-t2' "$MAKEFILE" +grep -Fq 'bash scripts/tests/test-hcc-stateless-cache-containment.sh' "$WORKFLOW" +if grep -Eq 'scale_proxy|kill .*HCC|kubectl .*delete pod' "$GATE"; then + echo 'Selective gate contains a broad API or pod fault' >&2 + exit 1 +fi + +# The proxy image has a read-only root filesystem. Exercise the real fixture +# builder and inspect the Deployment patch, so command/ack files have an owned +# writable mount in the effective pod configuration. +capture="$(mktemp "${TMPDIR:-/tmp}/hcc-channel-patch.XXXXXX")" +trap 'rm -f "$capture"' EXIT +( + # shellcheck source=scripts/e2e/_lib/hcc-watch-pr-a.sh + source "$HELPER" + PROXY_NAME=fixture-proxy HCC_NS=control-plane MCP_NS=mcp-server CHANNEL_NS=channels RUN_ID=fixture-run + HCC_PR_A_SELECTIVE_CHANNEL=1 + CAPTURE="$capture" + truncate_rfc1123() { printf '%.63s' "$1"; } + die() { echo "$*" >&2; exit 1; } + node() { printf '{}'; } + kctl() { + if [ "$1" = patch ]; then + while [ "$#" -gt 0 ]; do + if [ "$1" = -p ]; then printf '%s' "$2" > "$CAPTURE"; break; fi + shift + done + fi + } + hcc_pr_a_enable_proxy +) +jq -e ' + .spec.template.spec as $pod | + $pod.securityContext.fsGroup==1000 and + (any($pod.volumes[]; .name=="churn-control" and .emptyDir=={})) and + (any($pod.containers[]; .name=="proxy" and + any(.volumeMounts[]; .name=="churn-control" and .mountPath=="/churn-ctl" and (.readOnly!=true)) and + any(.env[]; .name=="CHURN_DISABLED" and .value=="1"))) +' "$capture" >/dev/null +grep -Fq 'readOnlyRootFilesystem: true' "${ROOT}/scripts/e2e/_lib/hcc-watch-recovery-fixture.sh" +echo 'Selective stateless-cache containment shell contract: PASS' diff --git a/scripts/tests/test-hcc-watch-recovery-gate.sh b/scripts/tests/test-hcc-watch-recovery-gate.sh index a971107223..c2326519a5 100755 --- a/scripts/tests/test-hcc-watch-recovery-gate.sh +++ b/scripts/tests/test-hcc-watch-recovery-gate.sh @@ -409,7 +409,7 @@ fi # shellcheck disable=SC2016 marker_bindings=( "$WATCH_GATE" 'CommunicationChannel watch ended;' - "$HCC_K8S_CLIENT" 'CommunicationChannel watch ended; holding stateless lifecycle active' + "$HCC_K8S_CLIENT" 'CommunicationChannel watch ended; preserving durable Host lifecycle state' "$WATCH_GATE" 'Starting CommunicationChannel watch' "$HCC_K8S_CLIENT" '[K8s] Starting CommunicationChannel watch' @@ -418,7 +418,7 @@ marker_bindings=( "$HCC_K8S_CLIENT" 'Recovered ${snapshot.channels.length} CommunicationChannel(s) into cache' "$WATCH_GATE" 'cache recovery failed;' - "$HCC_K8S_CLIENT" 'CommunicationChannel cache recovery failed; stateless lifecycle remains held active' + "$HCC_K8S_CLIENT" 'CommunicationChannel cache recovery failed; preserving durable Host lifecycle state' "$WATCH_GATE" 'Listing all CommunicationChannels in namespace ${CHANNEL_NS}' "$HCC_K8S_CLIENT" 'Listing all CommunicationChannels in namespace ${config.channelsNamespace}' @@ -436,7 +436,7 @@ marker_bindings=( "$HCC_K8S_CLIENT" 'CommunicationChannel watch event: ${type} for ${cc.name}' "$LOG_HELPER" 'CommunicationChannel watch ended;' - "$HCC_K8S_CLIENT" 'CommunicationChannel watch ended; holding stateless lifecycle active' + "$HCC_K8S_CLIENT" 'CommunicationChannel watch ended; preserving durable Host lifecycle state' "$LOG_HELPER" 'Recovered [0-9]+ CommunicationChannel\(s\) into cache' "$HCC_K8S_CLIENT" 'Recovered ${snapshot.channels.length} CommunicationChannel(s) into cache' @@ -1385,7 +1385,7 @@ else fi printf '%s\n' \ - '[K8s] CommunicationChannel watch ended; holding stateless lifecycle active until snapshot recovery' \ + '[K8s] CommunicationChannel watch ended; preserving durable Host lifecycle state until snapshot recovery' \ '[K8s] Listing all Hosts in namespace mcp-host' \ '[K8s] Recovered 2 CommunicationChannel(s) into cache (ccCacheSynced=true)' \ '[K8s] Reconciling 5 Host(s) for lifecycle after CommunicationChannel recovery' \ @@ -1397,7 +1397,7 @@ else fi printf '%s\n' \ - '[K8s] CommunicationChannel watch ended; holding stateless lifecycle active until snapshot recovery' \ + '[K8s] CommunicationChannel watch ended; preserving durable Host lifecycle state until snapshot recovery' \ '[K8s] Listing all Hosts in namespace mcp-host' \ '[K8s] Recovered 2 CommunicationChannel(s) into cache (ccCacheSynced=true)' \ '[K8s] Reconciling 5 Host(s) after Host watch recovery convergence' \ @@ -1409,7 +1409,7 @@ else fi printf '%s\n' \ - '[K8s] CommunicationChannel watch ended; holding stateless lifecycle active until snapshot recovery' \ + '[K8s] CommunicationChannel watch ended; preserving durable Host lifecycle state until snapshot recovery' \ '[K8s] Listing all Hosts in namespace mcp-host' \ '[K8s] Reconciling 5 Host(s) after Host watch recovery convergence' \ '[K8s] Recovered 2 CommunicationChannel(s) into cache (ccCacheSynced=true)' \ @@ -1422,7 +1422,7 @@ fi printf '%s\n' \ '[K8s] Listing all Hosts in namespace mcp-host' \ - '[K8s] CommunicationChannel watch ended; holding stateless lifecycle active until snapshot recovery' \ + '[K8s] CommunicationChannel watch ended; preserving durable Host lifecycle state until snapshot recovery' \ '[K8s] Recovered 2 CommunicationChannel(s) into cache (ccCacheSynced=true)' \ '[K8s] Reconciling 5 Host(s) for lifecycle after CommunicationChannel recovery' \ '[K8s] Completed Host reconciliation after CommunicationChannel recovery' >"$MOCK_LOG_FILE" @@ -1433,7 +1433,7 @@ else fi printf '%s\n' \ - '[K8s] CommunicationChannel watch ended; holding stateless lifecycle active until snapshot recovery' \ + '[K8s] CommunicationChannel watch ended; preserving durable Host lifecycle state until snapshot recovery' \ '[K8s] Listing all Hosts in namespace mcp-host' \ '[K8s] Recovered 2 CommunicationChannel(s) into cache (ccCacheSynced=true)' \ '[K8s] Reconciling 4 Host(s) for lifecycle after CommunicationChannel recovery' \ @@ -1445,12 +1445,12 @@ else fi printf '%s\n' \ - '[K8s] CommunicationChannel watch ended; holding stateless lifecycle active until snapshot recovery' \ + '[K8s] CommunicationChannel watch ended; preserving durable Host lifecycle state until snapshot recovery' \ '[K8s] Listing all Hosts in namespace mcp-host' \ '[K8s] Recovered 1 CommunicationChannel(s) into cache (ccCacheSynced=true)' \ '[K8s] Reconciling 4 Host(s) for lifecycle after CommunicationChannel recovery' \ '[K8s] Completed Host reconciliation after CommunicationChannel recovery' \ - '[K8s] CommunicationChannel watch ended; holding stateless lifecycle active until snapshot recovery' \ + '[K8s] CommunicationChannel watch ended; preserving durable Host lifecycle state until snapshot recovery' \ '[K8s] Listing all Hosts in namespace mcp-host' \ '[K8s] Recovered 2 CommunicationChannel(s) into cache (ccCacheSynced=true)' \ '[K8s] Reconciling 5 Host(s) for lifecycle after CommunicationChannel recovery' \ diff --git a/scripts/tests/test-minikube-pre-gate-shadow.sh b/scripts/tests/test-minikube-pre-gate-shadow.sh index 7f1f3d9489..c667b21928 100755 --- a/scripts/tests/test-minikube-pre-gate-shadow.sh +++ b/scripts/tests/test-minikube-pre-gate-shadow.sh @@ -127,7 +127,9 @@ exit 0 STUB cat > "$d/bin/make" <<'STUB' #!/usr/bin/env bash -printf 'make %s\n' "$*" >>"${TEST_LOG_FILE:?}" +printf 'make %s T2_SKIP_LOCK=%s T2_LOCK_TOKEN_SET=%s\n' \ + "$*" "${T2_SKIP_LOCK:-false}" "$([ -n "${T2_LOCK_TOKEN:-}" ] && printf true || printf false)" \ + >>"${TEST_LOG_FILE:?}" exit 0 STUB chmod +x "$d/bin/docker" "$d/bin/minikube" "$d/bin/kubectl" "$d/bin/make" @@ -366,12 +368,13 @@ assert_a_full_sync_in_ghcr_mode_repulls_instead_of_building_everything() { prepare_repo "$d" out="$( PATH="$d/bin:$PATH" TEST_LOG_FILE="$d/ops.log" \ - IMAGE_SOURCE=ghcr IMAGE_TAG="$PIN_TAG" \ + IMAGE_SOURCE=ghcr IMAGE_TAG="$PIN_TAG" T2_LOCK_TOKEN=test-token \ run_incremental "$d" 'INCREMENTAL_REPULL_ALL=true; INCREMENTAL_TARGETS=("control-api|control-plane|control-api"); incremental_build_images' )" rc=$? if [ "$rc" -eq 0 ] \ && grep -Fq "make minikube-pull-images" "$d/ops.log" \ + && grep -Fq 'make minikube-pull-images T2_SKIP_LOCK=true T2_LOCK_TOKEN_SET=true' "$d/ops.log" \ && ! grep -Fq "make minikube-build-images" "$d/ops.log" \ && grep -Fq "docker tag $(local_ref_for_image control-api) ghcr.io/evenfire-ai/control-api:${PIN_TAG}" "$d/ops.log"; then pass "a ghcr full sync re-pulls the release set, then shadows the changed set on top" @@ -388,12 +391,13 @@ assert_local_mode_still_builds_everything_on_a_full_image_build() { prepare_repo "$d" out="$( PATH="$d/bin:$PATH" TEST_LOG_FILE="$d/ops.log" \ - IMAGE_SOURCE=local IMAGE_TAG="" \ + IMAGE_SOURCE=local IMAGE_TAG="" T2_LOCK_TOKEN=test-token \ run_incremental "$d" 'INCREMENTAL_FULL_IMAGE_BUILD=true; incremental_build_images' )" rc=$? if [ "$rc" -eq 0 ] \ && grep -Fq "make minikube-build-images" "$d/ops.log" \ + && grep -Fq 'make minikube-build-images T2_SKIP_LOCK=true T2_LOCK_TOKEN_SET=true' "$d/ops.log" \ && ! grep -Fq "make minikube-pull-images" "$d/ops.log"; then pass "local mode still builds every image when the change cannot be targeted" else diff --git a/scripts/tests/test-minikube-t1-reporter-report.sh b/scripts/tests/test-minikube-t1-reporter-report.sh new file mode 100755 index 0000000000..21e818c142 --- /dev/null +++ b/scripts/tests/test-minikube-t1-reporter-report.sh @@ -0,0 +1,370 @@ +#!/usr/bin/env bash +# T1 must keep the Vitest JSON reporter parseable after redaction and must print +# the file-level message of a suite that failed before any test ran. Only the +# T1 function definitions are sourced; nothing here touches a cluster or Docker. +set -euo pipefail + +ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)" +fail() { printf 'FAIL: %s\n' "$*" >&2; exit 1; } + +TEST_DIR="$(mktemp -d "${TMPDIR:-/tmp}/evenfire-t1-reporter.XXXXXX")" +trap 'rm -rf -- "$TEST_DIR"' EXIT + +( + set +e + export MINIKUBE_PROFILE=t1-reporter-contract + export T2_CONTEXT=t1-reporter-contract + source "$ROOT/scripts/e2e/minikube-real-postgres.sh" >/dev/null 2>&1 + trap - EXIT INT TERM + set +e + declare -F sanitize_file >/dev/null || fail 'sanitize_file is not defined' + declare -F print_reporter_failure_details >/dev/null || fail 'print_reporter_failure_details is not defined' + + assert_valid_json() { + python3 -c 'import json, sys; json.load(open(sys.argv[1]))' "$1" \ + || fail "$2: sanitized reporter is not valid JSON" + } + + # 1. A DSN followed by an escaped quote must not swallow the escape. + # The synthetic DSNs are assembled at run time so this file carries no + # literal scheme://user:password@host, which the public-boundary gate rejects. + dsn_scheme=postgresql + dsn_json="$TEST_DIR/dsn.json" + printf '{"failureMessages":["expected \\"%s://u:p@h/db\\" to be undefined"]}' "$dsn_scheme" >"$dsn_json" + T1_REDACT_PASSWORD='' sanitize_file "$dsn_json" + assert_valid_json "$dsn_json" 'DSN inside a JSON string' + grep -Fq '' "$dsn_json" \ + || fail 'DSN was not redacted (liveness witness missing)' + grep -Fq 'u:p@h' "$dsn_json" && fail 'DSN credential survived redaction' + + # 2. A password with JSON-special characters is redacted in every spelling. + pw_json="$TEST_DIR/password.json" + password='s3cr"et\pw' + python3 - "$pw_json" "$password" <<'PY' +import json +import sys +from urllib.parse import quote + +path, password = sys.argv[1], sys.argv[2] +message = ( + f"auth failed for {password} " + f"see http://example.invalid/?p={quote(password, safe='')}" +) +with open(path, "w") as handle: + json.dump({"testResults": [{"message": message}]}, handle) +PY + grep -Fq 's3cr' "$pw_json" || fail 'fixture does not contain the password (liveness witness missing)' + T1_REDACT_PASSWORD="$password" sanitize_file "$pw_json" + assert_valid_json "$pw_json" 'JSON-escaped password' + grep -Fq '' "$pw_json" || fail 'password was not redacted' + grep -Fq 's3cr' "$pw_json" && fail 'a spelling of the password survived redaction' + + # 2a. A non-ASCII password written by the Vitest reporter's real encoder. + # JSON.stringify keeps non-ASCII characters literal (Python's default + # ensure_ascii=True would write ä instead), so the redactor must know + # both spellings, escaped once (a reporter message) and twice (a JSON + # document quoted inside a message). + for non_ascii_password in 'Qzä"ss' 'Qzä\ss'; do + non_ascii_json="$TEST_DIR/non-ascii.json" + node -e ' + const fs = require("fs") + const [file, password] = process.argv.slice(1) + const inner = JSON.stringify({ failureMessages: ["auth failed for " + password] }) + fs.writeFileSync(file, JSON.stringify({ testResults: [{ message: "auth failed for " + password, inner }] })) + ' "$non_ascii_json" "$non_ascii_password" + [ "$(grep -o 'Qz' "$non_ascii_json" | wc -l | tr -d ' ')" = 2 ] \ + || fail "non-ASCII fixture does not contain both spellings of $non_ascii_password (liveness witness missing)" + T1_REDACT_PASSWORD="$non_ascii_password" sanitize_file "$non_ascii_json" + assert_valid_json "$non_ascii_json" "non-ASCII password $non_ascii_password" + [ "$(grep -o '' "$non_ascii_json" | wc -l | tr -d ' ')" = 2 ] \ + || fail "non-ASCII password $non_ascii_password was not redacted in both spellings" + grep -Fq 'Qz' "$non_ascii_json" && fail "a spelling of the non-ASCII password $non_ascii_password survived redaction" + done + + # 2b. Spellings of a secret the first two cases did not cover. Each fixture + # is built from real producers (json.dump, quote, a DSN template) and + # first shown to contain the secret, so a redactor that never ran cannot + # pass; the reporter must still parse afterwards. + # 2b-i. JavaScript's encodeURIComponent keeps !~*'() literal, which + # urllib's quote(safe="") does not, so 'ss%20word(x)' used to survive. + uri_json="$TEST_DIR/uri.json" + uri_password="pa'ss word(x)" + python3 - "$uri_json" "$uri_password" <<'PY' +import json +import sys +from urllib.parse import quote + +path, password = sys.argv[1], sys.argv[2] +js_encoded = quote(password, safe="-_.!~*'()") # encodeURIComponent output +with open(path, "w") as handle: + json.dump({"testResults": [{"message": f"connect failed, retry with ?pw={js_encoded}"}]}, handle) +PY + grep -Fq "ss%20word(x)" "$uri_json" || fail 'encodeURIComponent fixture lacks the password (liveness witness missing)' + T1_REDACT_PASSWORD="$uri_password" sanitize_file "$uri_json" + assert_valid_json "$uri_json" 'encodeURIComponent password' + grep -Fq '' "$uri_json" || fail 'encodeURIComponent password was not redacted' + grep -Fq 'ss%20word' "$uri_json" && fail 'the encodeURIComponent spelling of the password survived' + + # 2b-ii. A JSON document quoted inside a reporter message escapes the + # backslash twice. + twice_json="$TEST_DIR/twice.json" + twice_password='Zq7\Xv3' + python3 - "$twice_json" "$twice_password" <<'PY' +import json +import sys + +path, password = sys.argv[1], sys.argv[2] +inner = json.dumps({"failureMessages": [f"auth failed for {password}"]}) +with open(path, "w") as handle: + json.dump({"testResults": [{"message": inner}]}, handle) +PY + grep -Fq 'Zq7' "$twice_json" || fail 'double-escaped fixture lacks the password (liveness witness missing)' + T1_REDACT_PASSWORD="$twice_password" sanitize_file "$twice_json" + assert_valid_json "$twice_json" 'doubly JSON-escaped password' + grep -Fq '' "$twice_json" || fail 'doubly escaped password was not redacted' + grep -Fq 'Zq7' "$twice_json" && fail 'the doubly escaped spelling of the password survived' + grep -Fq 'Xv3' "$twice_json" && fail 'the tail of the doubly escaped password survived' + + # 2b-iii. An upper-case scheme is still a DSN. No password is registered, so + # the DSN credentials must be removed on their own. + upper_scheme="$(printf '%s' "$dsn_scheme" | tr '[:lower:]' '[:upper:]')" + upper_json="$TEST_DIR/upper.json" + python3 - "$upper_json" "$upper_scheme" <<'PY' +import json +import sys + +path, scheme = sys.argv[1], sys.argv[2] +message = f"connect failed for {scheme}://gfs_runtime:UpperSecret1@db.internal:5432/app" +with open(path, "w") as handle: + json.dump({"testResults": [{"message": message}]}, handle) +PY + grep -Fq 'UpperSecret1' "$upper_json" || fail 'upper-case DSN fixture lacks the secret (liveness witness missing)' + T1_REDACT_PASSWORD='' sanitize_file "$upper_json" + assert_valid_json "$upper_json" 'upper-case DSN scheme' + grep -Fq '' "$upper_json" || fail 'upper-case DSN was not redacted' + grep -Fq 'UpperSecret1' "$upper_json" && fail 'the credential of an upper-case DSN survived' + + # 2b-iv. libpq conninfo carries the password as password=..., bare or quoted. + # A bare value at the end of a JSON string must not eat the closing quote. + conninfo_json="$TEST_DIR/conninfo.json" + python3 - "$conninfo_json" <<'PY' +import json +import sys + +# The quoted value carries the word "Fixture", the marker the public-boundary +# gate accepts for a synthetic credential assignment. +messages = [ + "connection to server failed: host=db.internal user=gfs password=BareSecret2", + "connection to server failed: host=db.internal password='Quoted Fixture3' dbname=app", + "PGPASSWORD=EnvSecret4 psql failed", +] +with open(sys.argv[1], "w") as handle: + json.dump({"testResults": [{"message": message} for message in messages]}, handle) +PY + for secret in BareSecret2 'Quoted Fixture3' EnvSecret4; do + grep -Fq "$secret" "$conninfo_json" || fail "conninfo fixture lacks '$secret' (liveness witness missing)" + done + T1_REDACT_PASSWORD='' sanitize_file "$conninfo_json" + assert_valid_json "$conninfo_json" 'libpq conninfo password' + grep -Fq '' "$conninfo_json" || fail 'conninfo password was not redacted' + for secret in BareSecret2 'Quoted Fixture3' EnvSecret4; do + grep -Fq "$secret" "$conninfo_json" && fail "conninfo password '$secret' survived" + done + grep -Fq 'host=db.internal' "$conninfo_json" || fail 'redaction removed the non-secret conninfo fields' + + # 2b-v. A password with a double quote inside a DSN is JSON-escaped, so the + # backslash used to end the redaction and leave the tail readable. No + # password is registered: the DSN userinfo must go on its own. + tail_json="$TEST_DIR/tail.json" + python3 - "$tail_json" "$dsn_scheme" <<'PY' +import json +import sys + +path, scheme = sys.argv[1], sys.argv[2] +message = f'connect failed for {scheme}://gfs_runtime:Zq"TailSecret5@db.internal:5432/app' +with open(path, "w") as handle: + json.dump({"testResults": [{"message": message}]}, handle) +PY + grep -Fq 'TailSecret5' "$tail_json" || fail 'escaped-DSN fixture lacks the secret (liveness witness missing)' + T1_REDACT_PASSWORD='' sanitize_file "$tail_json" + assert_valid_json "$tail_json" 'JSON-escaped DSN with a quote in the password' + grep -Fq '' "$tail_json" || fail 'escaped DSN was not redacted' + grep -Fq 'TailSecret5' "$tail_json" && fail 'the tail of a JSON-escaped DSN survived' + grep -Fq 'db.internal' "$tail_json" && fail 'the DSN host survived redaction' + + # 3. A suite that failed before any test has no assertionResults; its reason + # is the file-level message and must be printed after the FAILED FILE line. + report="$TEST_DIR/report.json" + cat >"$report" <"$details" 2>&1 || fail 'print_reporter_failure_details failed' + grep -Fq 'FAILED FILE: /repo/control-api/early.realPostgres.test.ts' "$details" \ + || fail 'the failed file was not printed (liveness witness missing)' + grep -Fq 'FAILED FILE: /repo/control-api/late.realPostgres.test.ts' "$details" \ + || fail 'the assertion-failure file was not printed' + grep -Fq 'LATE-DETAIL' "$details" || fail 'assertion failure messages are no longer printed' + grep -Fq 'ok.realPostgres' "$details" && fail 'a passing file was printed as failed' + early_next="$(grep -A1 -F 'FAILED FILE: /repo/control-api/early.realPostgres.test.ts' "$details" | tail -n 1)" + case "$early_next" in + 'beforeAll failed: connect ECONNREFUSED '*) ;; + *) fail "pre-test failure reason was not printed after FAILED FILE: '$early_next'" ;; + esac + grep -Fq 'hunter2' "$details" && fail 'a credential reached the failure dump' + grep -Fq '' "$details" \ + || fail 'the dumped message was not sanitized' + # The counters line is the only place the suite/test totals reach the log. + grep -Fq 'T1 reporter counters:' "$details" || fail 'the reporter counters line is missing' + grep -Fq "'numTotalTestSuites': 3" "$details" || fail 'the counters line does not carry the suite total' + grep -Fq "'numFailedTestSuites': 2" "$details" || fail 'the counters line does not carry the failed suite count' + + # 3b. The raw password, with JSON-special characters, as it appears in a + # plain-text log (not a JSON string). Only the verbatim spelling matches + # there: the JSON-escaped and percent-encoded spellings differ from it. + plain_log="$TEST_DIR/plain.log" + plain_password='Lg8"Pl\ain' + printf 'vitest: auth failed for %s (retrying)\n' "$plain_password" >"$plain_log" + grep -Fq "$plain_password" "$plain_log" || fail 'plain log fixture lacks the password (liveness witness missing)' + T1_REDACT_PASSWORD="$plain_password" sanitize_file "$plain_log" + grep -Fq '' "$plain_log" || fail 'the verbatim password was not redacted in a plain log' + grep -Fq 'Pl\ain' "$plain_log" && fail 'the verbatim spelling of the password survived in a plain log' + grep -Fq 'Lg8' "$plain_log" && fail 'the head of the verbatim password survived in a plain log' + + # 3c. An Authorization header inside a reporter message. The token is + # mid-string in one message and ends the JSON string in the other: the + # redaction must stop before the closing quote in both. + bearer_json="$TEST_DIR/bearer.json" + bearer_token='FixtureBearerTok7' + python3 - "$bearer_json" "$bearer_token" <<'PY' +import json +import sys + +path, token = sys.argv[1], sys.argv[2] +messages = [ + f"request rejected: Authorization: Bearer {token} while calling the API", + f"request rejected: Authorization: Bearer {token}", +] +with open(path, "w") as handle: + json.dump({"testResults": [{"message": message} for message in messages]}, handle) +PY + grep -Fq "$bearer_token" "$bearer_json" || fail 'bearer fixture lacks the token (liveness witness missing)' + T1_REDACT_PASSWORD='' sanitize_file "$bearer_json" + assert_valid_json "$bearer_json" 'Authorization bearer header' + grep -Fq '' "$bearer_json" || fail 'the bearer token was not redacted' + grep -Fq "$bearer_token" "$bearer_json" && fail 'the bearer token survived redaction' + grep -Fq 'Authorization: Bearer' "$bearer_json" || fail 'redaction removed the header name' + + # 4. The failure path of run_suite itself. Calling print_reporter_failure_details + # directly (case 3) cannot tell whether the script still calls it when the + # reporter says a suite failed. Here the real run_suite runs against a + # stand-in npm that writes a failing reporter, exactly as Vitest would, and + # the FAILED FILE line, the pre-test message and the counters must reach + # stderr, sanitized. run_suite runs under set -e, as in production. + run_project="$TEST_DIR/run-project" + run_file="$run_project/control-api/early.realPostgres.test.ts" + mkdir -p "$run_project/control-api" + printf 'fixture\n' >"$run_file" + run_password='RunPlain9Secret' + run_bearer='FixtureRunBearer8' + npm_marker="$TEST_DIR/npm-ran" + PROJECT_DIR="$run_project" + T1_TMP_DIR="$TEST_DIR/run-tmp" + mkdir -p "$T1_TMP_DIR" + RUN_FILE="$run_file" RUN_DSN_SCHEME="$dsn_scheme" RUN_BEARER="$run_bearer" NPM_MARKER="$npm_marker" + RUN_PASSWORD="$run_password" + export RUN_FILE RUN_DSN_SCHEME RUN_BEARER NPM_MARKER RUN_PASSWORD + npm() { + local output_file='' + while [ "$#" -gt 0 ]; do + case "$1" in + --outputFile=*) output_file="${1#--outputFile=}" ;; + esac + shift + done + printf 'npm stand-in ran\n' >"$NPM_MARKER" + printf 'vitest log: connect refused for %s\n' "$RUN_PASSWORD" + python3 - "$output_file" <<'PY' +import json +import os +import sys + +message = ( + "beforeAll failed: connect ECONNREFUSED " + f"{os.environ['RUN_DSN_SCHEME']}://admin:{os.environ['RUN_PASSWORD']}@127.0.0.1:5432/postgres " + f"with Authorization: Bearer {os.environ['RUN_BEARER']} on retry" +) +report = { + "success": False, + "numTotalTestSuites": 1, + "numPassedTestSuites": 0, + "numFailedTestSuites": 1, + "numPendingTestSuites": 0, + "numTotalTests": 0, + "numPassedTests": 0, + "numFailedTests": 0, + "numPendingTests": 0, + "testResults": [ + {"name": os.environ["RUN_FILE"], "status": "failed", "message": message, "assertionResults": []} + ], +} +with open(sys.argv[1], "w") as handle: + json.dump(report, handle) +PY + return 1 + } + export -f npm + export RUN_PROJECT="$run_project" RUN_SCRIPT="$ROOT/scripts/e2e/minikube-real-postgres.sh" RUN_TMP="$T1_TMP_DIR" + run_err="$TEST_DIR/run-suite.err" + # A child bash, so that set -e is really in force: this subshell is itself the + # left side of a || list, where set -e is ignored and die_t1 would not stop + # run_suite the way it does in production. + bash -c ' + set -euo pipefail + source "$RUN_SCRIPT" >/dev/null 2>&1 + trap - EXIT INT TERM + PROJECT_DIR="$RUN_PROJECT" + T1_TMP_DIR="$RUN_TMP" + T1_REDACT_PASSWORD="$RUN_PASSWORD" + run_suite control-api isolated "$1" + ' bash "$dsn_scheme://isolated" >"$TEST_DIR/run-suite.out" 2>"$run_err" + run_status=$? + [ -s "$npm_marker" ] || fail 'the npm stand-in never ran (liveness witness missing)' + grep -Fq 'REAL_PG_SUITE_FAILED' "$run_err" || fail 'run_suite did not fail on the failing reporter' + [ "$run_status" -ne 0 ] || fail 'run_suite returned success on a failing reporter' + grep -Fq "FAILED FILE: $run_file" "$run_err" \ + || fail 'run_suite did not print the FAILED FILE line of the failing suite' + grep -Fq 'beforeAll failed: connect ECONNREFUSED' "$run_err" \ + || fail 'run_suite did not print the pre-test failure message' + grep -Fq 'T1 reporter counters:' "$run_err" || fail 'run_suite did not print the reporter counters' + grep -Fq "'numFailedTestSuites': 1" "$run_err" || fail 'run_suite counters lack the failed suite count' + grep -Fq '' "$run_err" || fail 'the failure dump was not bearer-sanitized' + grep -Fq '' "$run_err" || fail 'the vitest log was not password-sanitized' + grep -Fq "$run_password" "$run_err" && fail 'the password reached the run_suite failure output' + grep -Fq "$run_bearer" "$run_err" && fail 'the bearer token reached the run_suite failure output' + unset -f npm + exit 0 +) || fail 'T1 reporter sanitization or failure dump contract broken' + +printf 'PASS: T1 reporter stays valid JSON after redaction and prints pre-test failure reasons\n' diff --git a/scripts/tests/test-minikube-t2-contract.sh b/scripts/tests/test-minikube-t2-contract.sh index 24cb4921e4..ad3d018122 100755 --- a/scripts/tests/test-minikube-t2-contract.sh +++ b/scripts/tests/test-minikube-t2-contract.sh @@ -51,6 +51,7 @@ for file in "$MINIKUBE_DIR/profile-readiness.sh" "$ROOT/scripts/tests/test-minik "$ROOT/scripts/tests/test-minikube-pre-gate-shadow.sh" \ "$ROOT/scripts/tests/test-minikube-fenced-recovery-render.sh" \ "$ROOT/scripts/tests/test-minikube-t2-process-owner.sh" \ + "$ROOT/scripts/tests/test-minikube-t1-reporter-report.sh" \ "$ROOT/scripts/tests/test-minikube-explicit-context.sh" \ "$ROOT/scripts/tests/test-minikube-t2-evidence.sh" \ "$ROOT/scripts/tests/test-minikube-targeted-health.sh" \ @@ -59,6 +60,7 @@ for file in "$MINIKUBE_DIR/profile-readiness.sh" "$ROOT/scripts/tests/test-minik done "$ROOT/scripts/tests/test-minikube-t1-port-forward-owner.sh" "$ROOT/scripts/tests/test-minikube-t2-process-owner.sh" +"$ROOT/scripts/tests/test-minikube-t1-reporter-report.sh" "$ROOT/scripts/tests/test-minikube-explicit-context.sh" "$ROOT/scripts/tests/test-minikube-t2-evidence.sh" "$ROOT/scripts/tests/test-minikube-targeted-health.sh" @@ -362,6 +364,24 @@ if [ -z "$post_runtime_process_check_line" ] || [ -z "$complete_pass_line" ] || echo 'FAIL: T2 does not revalidate port-forward ownership before complete PASS' >&2 exit 1 fi +# t2_evidence_init opens every evidence file with `preflight RUNNING`. The +# planner closes it; each certification writer must close it too, or a PASS +# attestation keeps a phase whose latest status reads as still running. +preflight_plan_body="$(awk '/^run_preflight_plan\(\) \{$/,/^\}$/' "$T2")" +if [ -z "$preflight_plan_body" ] || + ! grep -Fq 't2_evidence_write preflight PASS' <<<"$preflight_plan_body"; then + echo 'FAIL: T2 certification evidence never closes the preflight phase opened by t2_evidence_init' >&2 + exit 1 +fi +# awk prints an empty line number when nothing matches, so the explicit FAIL +# below reports the gap instead of pipefail ending the script silently. +t1_evidence_init_line="$(awk '$0 == " t2_evidence_init" { line = NR } END { print line }' "$T1")" +t1_preflight_pass_line="$(awk 'index($0, "t2_evidence_write preflight PASS") { line = NR } END { print line }' "$T1")" +if [ -z "$t1_evidence_init_line" ] || [ -z "$t1_preflight_pass_line" ] || + [ "$t1_preflight_pass_line" -le "$t1_evidence_init_line" ]; then + echo 'FAIL: T1 certification evidence never closes the preflight phase opened by t2_evidence_init' >&2 + exit 1 +fi if ! grep -Fq 'instead of already-synced' "$T2"; then echo 'FAIL: final T2 preflight does not require already-synced' >&2 exit 1 diff --git a/scripts/tests/test-minikube-t2-process-owner.sh b/scripts/tests/test-minikube-t2-process-owner.sh index a309f2ac52..370466c6dd 100755 --- a/scripts/tests/test-minikube-t2-process-owner.sh +++ b/scripts/tests/test-minikube-t2-process-owner.sh @@ -57,7 +57,8 @@ case "$mode" in pid=) suffix=pid ;; *) exit 1 ;; esac -[ ! -f "${T2_PS_DATA_DIR:?}/${pid}.${suffix}" ] || cat "${T2_PS_DATA_DIR}/${pid}.${suffix}" +[ -f "${T2_PS_DATA_DIR:?}/${pid}.${suffix}" ] || exit 1 +cat "${T2_PS_DATA_DIR}/${pid}.${suffix}" EOF_PS chmod +x "$PS_BIN/ps" @@ -115,13 +116,21 @@ run_process_check "$TEST_DIR/exact.out" # A registered record must remain visible to the verdict even when the live # process disappears from ps(1). This protects the fail-closed ownership scan -# from treating a stale/dead pidfile as "nothing to check". -rm -f "$PS_DATA/$PID.command" +# from treating a stale/dead pidfile as "nothing to check" or deleting it. +rm -f "$PS_DATA/$PID.command" "$PS_DATA/$PID.pid" +if T2_PS_DATA_DIR="$PS_DATA" PATH="$PS_BIN:$PATH" ps -p "$PID" -o pid= >/dev/null 2>&1; then + printf 'FAIL: dead-PID fixture did not make ps fail\n' >&2 + exit 1 +fi if run_process_check "$TEST_DIR/stale-record.out"; then printf 'FAIL: stale registered port-forward record was ignored\n' >&2 exit 1 fi grep -Fq PORT_FORWARD_CONFLICT "$TEST_DIR/stale-record.out" +[ -f "$RECORD" ] || { + printf 'FAIL: final process check removed the dead owner record\n' >&2 + exit 1 +} set_process_fixture kubectl "$exact_command" printf '%s\n' 'different process start' >"$PS_DATA/$PID.start" @@ -140,5 +149,34 @@ if run_process_check "$TEST_DIR/duplicate.out"; then exit 1 fi grep -Fq PORT_FORWARD_CONFLICT "$TEST_DIR/duplicate.out" +rm -f -- "$SECOND_RECORD" + +# A record whose first line is not a PID is an ownership failure and must stay +# on disk. The baseline run proves the fixture alone is accepted, so the refusal +# below can only come from the malformed record. +run_process_check "$TEST_DIR/malformed-baseline.out" || { + printf 'FAIL: exact ownership fixture was rejected before the malformed record existed\n' >&2 + exit 1 +} +MALFORMED_RECORD="$PID_DIR/control-ui-malformed.pid" +printf '%s\n' 'not-a-pid' 'trailing line' >"$MALFORMED_RECORD" +malformed_before="$(cat "$MALFORMED_RECORD")" +if run_process_check "$TEST_DIR/malformed.out"; then + printf 'FAIL: a port-forward record whose first line is not a PID was accepted\n' >&2 + exit 1 +fi +grep -Fq PORT_FORWARD_CONFLICT "$TEST_DIR/malformed.out" +grep -Fq "$MALFORMED_RECORD" "$TEST_DIR/malformed.out" || { + printf 'FAIL: the conflict did not name the malformed record\n' >&2 + exit 1 +} +[ -f "$MALFORMED_RECORD" ] && [ "$(cat "$MALFORMED_RECORD")" = "$malformed_before" ] || { + printf 'FAIL: the process check removed or rewrote the malformed record\n' >&2 + exit 1 +} +[ -f "$RECORD" ] || { + printf 'FAIL: the process check removed the valid owner record\n' >&2 + exit 1 +} printf 'PASS: T2 process inventory requires one exact structured port-forward owner\n'