diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 63529ae..9cee9bb 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -7,10 +7,10 @@ **Key Characteristics:** - Registers as the built-in `openai` provider; OpenCode loads external server plugins after its internal ones, so this package transparently supersedes OpenCode's internal OpenAI auth hook. - Rewrites OpenAI Responses requests into Codex's wire shape (headers, body, tools, turn metadata) so the Codex backend treats traffic as if it came from the official Codex CLI. -- Reactive (not preemptive) account fallback: a `401`/`403`/`429` triggers a retry on the next usable account, respecting routing mode. Enforce the killswitch as a hard circuit-breaker directly on the request path to block requests or filter candidates before spending when cached quota falls below configured thresholds. +- Reactive account fallback for ordered modes, plus sticky-balanced cold-session placement. Sticky pins migrate only on confirmed quota exhaustion or permanent auth failure; there is no mid-session rebalance or Retry-After hold. Enforce the killswitch as a hard circuit-breaker directly on the request path to block requests or filter candidates before spending when cached quota falls below configured thresholds. - Push-only quota tracking: quota comes from `x-codex-*` HTTP response headers or `codex.rate_limits` WS frames — no extra polling during normal traffic. - Three transport modes share the cache-stabilizer behavior: HTTP/SSE, native WebSocket, and a hand-rolled RFC 6455 WebSocket (Bun.connect or node:net/node:tls). -- TUI sidebar reads a serialized `sidebar-state.json` snapshot pushed by the auth loader; the loader and TUI exchange dialogs/notifications over a loopback HTTP RPC bound to a per-process token. +- TUI sidebar reads a serialized, machine-global `sidebar-state.json` snapshot pushed by the auth loader; it owns SHA-256-keyed sticky assignments with a seven-day TTL. The loader and TUI exchange dialogs/notifications over a loopback HTTP RPC bound to a per-process token. - Plugin is split into a generic, provider-agnostic core (`core/`) and Codex-specific seams (`provider.ts`, `oauth.ts`) so the same shape could host another OAuth provider. ## Layers @@ -51,7 +51,7 @@ - Used by: Plugin loader (`/login openai` `methods`), CLI (`login`), `/openai-account add`. **Cache keep-warm:** -- Purpose: Track idle main-agent (and optionally subagent) sessions and replay the last real request as a `store:false` shadow request just before Codex evicts the prompt cache. Employs model-aware TTL (raising GPT-5.6 TTL to 30 min from the 5-min default), gpt-5.6 subagent 2-warm caps, a process clock-bound window (outside of which warming and capture are skipped), and extended subagent idle bounds (75 min for GPT-5.6 subagents). +- Purpose: Track idle main-agent (and optionally subagent) sessions and replay the last real request as a `store:false` shadow request just before Codex evicts the prompt cache. Employs model-aware TTL (raising GPT-5.6 TTL to 30 min from the 5-min default), gpt-5.6 subagent 2-warm caps, a process clock-bound window (outside of which warming and capture are skipped), and extended subagent idle bounds (75 min for GPT-5.6 subagents). `sustain` defaults off and bypasses only main idle pruning; it leaves the clock window, subagent limits, target-count, byte, and LRU caps intact. - Location: `packages/opencode/src/core/cachekeep.ts` - Contains: `CacheKeepManager` class (target map, timer, idle caps, backoff), `buildKeepwarmCapture`, `buildKeepwarmBody`, model-aware TTL matcher (`isGpt56Model`, `ttlForModel`), clock window checker (`isWithinCacheKeepWindow`), SSE/JSON usage extraction. - Depends on: `core/accounts.ts` (`findCachekeepFallbackAccount` exported from `index.ts`), `quota-normalize.ts`. @@ -140,14 +140,13 @@ 5. `auth.loader` constructs `QuotaManager`, `FallbackAccountManager`, and (if any fallback accounts) starts `fallbackManager.startBackgroundRefresh()`. 6. Each refresh runs through `codexRefreshFn` with file-lock + lease concurrency — `core/refresh-file-lock.ts`, `index.ts` `refreshMainWithLease`. Refreshed token persistence retries up to 3 times to prevent transient file locks or API write errors from invalidating sessions. -**Reactive fallback (per request):** +**Routing and request flow (per request):** -1. Plugin loader `auth.fetch` resolves the session affinity ID from the request's headers (`x-session-affinity`, `x-opencode-session`, `x-session-id`, `session-id`) and determines the routing mode (purely mode-driven: `main-first` or `fallback-first`). If `fallback-first` mode is active and the request is replayable, it proactively tries usable fallback accounts before the main account. -2. Strips any existing `authorization` header, refreshes an expired main token via `refreshMainWithLease`, or refreshes a fallback via `fallbackManager.refreshAccount`. Derives the main ChatGPT identity from the access token JWT to ensure correct quota/killswitch tracking after a main-account switch. -3. If a proactive fallback serves, its response is used. If a proactive fallback request throws a transport error (both caller-aborts and indeterminate transport failures), routing halts immediately and the error propagates to prevent request duplication and double-billing. Otherwise (or under `main-first` mode), checks if the primary account is blocked by the killswitch (verifying cached quota against configured thresholds) or by a mid-stream rate-limit mark. If blocked, it synthesizes a 429 response carrying a `Retry-After` header derived from the earliest known reset time across all accounts (or the mid-stream mark's own reset time, whichever is tighter). -4. If the request is not blocked by the killswitch, calls `sendWithAccessToken` which rewrites headers/body via `prepareCodexRequest`, picks HTTP or WS transport, and optionally tracks the body for cachekeep — `packages/opencode/src/index.ts`. -5. If the primary request fails with a fallback status (`401`/`403`/`429`), is blocked by the killswitch, or encounters mid-stream rate-limit exhaustion before streaming starts, and the request is replayable, `tryFallbackAccounts` reactively iterates usable fallback accounts (filtering candidates below their killswitch thresholds and unconditionally excluding those with active mid-stream rate-limit marks) and retries each candidate — `packages/opencode/src/index.ts`. Indeterminate transport failures on reactive fallbacks halt routing immediately to prevent duplication. If a fallback attempt fails, its advisory quota headers are pushed to the cache. Reactive fallback is skipped if the proactive gate already tried all fallbacks in `fallback-first` mode. -6. The final response's `x-codex-*` headers are normalized via `normalizeQuotaHeaders` and pushed into `QuotaManager` (main or per-account). The loader then calls `writeRequestSidebarRouting` to write the routing snapshot to `sidebar-state.json`. If a session ID is present, it registers the active account to that session in `activeRouting` (pruned to 128 entries and 1 hour age); otherwise, it falls back to legacy routing. The TUI sidebar resolves the active account for its session via `resolveSessionSidebarRouting`. +1. Resolve the session key from `x-session-affinity`, `x-opencode-session`, `x-session-id`, or `session-id`. +2. Read the shared sidebar state once. In `sticky-balanced` mode, retain a valid pin or create one using least projected pressure against fresh quota; stale and unknown quota are excluded, and an empty weighted set fails open to configured order. Equal scores use configured order then account id, while a shared pending-byte bridge accounts for concurrent cold sessions. +3. Send on the chosen account. Ordered modes retain their normal main-first or fallback-first retry behavior; sticky-balanced does not rebalance a live session and has no Retry-After hold. +4. Classify a sticky break after the send. Only confirmed exhaustion and permanent auth failure migrate the pin; transient, stale, and unknown outcomes retain it. +5. Repin immediately on migration, then write the served account to the display state. Main and fallback quota headers are normalized into `QuotaManager`; `activeRouting` remains a short-lived display record, while `stickyAssignments` owns the seven-day, SHA-256-keyed session pins. A child session uses its own pin, and the parent display keeps its own pin rather than mirroring the child. **Quota push (no extra polling during normal traffic):** @@ -168,7 +167,7 @@ 1. Every main-agent (and optionally subagent) request is captured by `buildKeepwarmCapture` from `sendWithAccessToken`. Outside of the configured clock window, capture is skipped. 2. `cacheKeepManager.track` stores the body + replay headers per session, computing `cacheExpiresAt` using model-aware TTL (30 min for GPT-5.6 models, 5 min otherwise). -3. A 60s timer fires; if the current hour is within the clock window, it checks each tracked session. For sessions within `leadMs` of expiry and within their respective idle caps (1 h main, 30 min subagent, or 75 min for GPT-5.6 subagents), it calls `buildKeepwarmBody(body)` (`store:false`, token caps removed) and replays via `fetchImpl`. +3. A 60s timer fires; if the current hour is within the clock window, it checks each tracked session. For sessions within `leadMs` of expiry and within their respective idle caps (1 h main, 30 min subagent, or 75 min for GPT-5.6 subagents), it calls `buildKeepwarmBody(body)` (`store:false`, token caps removed) and replays via `fetchImpl`. With `sustain on`, only the main 1-hour idle pruning bound is bypassed; the window and all memory/LRU caps still apply. 4. Successful warms increment `warmCount`. A GPT-5.6 subagent session is immediately removed/evicted from tracking once its `warmCount` reaches the 2-warm cap. 5. Failures trigger a 10-min backoff per session. @@ -190,7 +189,7 @@ - Pattern: Push-only (no `fetchQuotaFn` injected — quota comes via `setMain`/`setFallback`); active refresh is orchestrated by `refreshAllQuota`. **`CacheKeepManager`:** -- Purpose: Idle prompt-cache warmer with per-session targets, idle caps (1 h main / 30 min subagent, extended to 75 min for GPT-5.6 subagents), clock window checks, and 10-min backoff after a failed warm. +- Purpose: Idle prompt-cache warmer with per-session targets, idle caps (1 h main / 30 min subagent, extended to 75 min for GPT-5.6 subagents), clock window checks, and 10-min backoff after a failed warm. Main-only `sustain` bypasses idle pruning without affecting the other bounds. - Location: `packages/opencode/src/core/cachekeep.ts` - Pattern: Target map keyed by session id; interval timer; bounded (`maxTargets`, `maxBytes`) so a long-lived process cannot leak; model-aware TTL adjustment (30-min TTL for GPT-5.6 models) and gpt-5.6 subagent 2-warm limits. @@ -205,9 +204,9 @@ - Pattern: HTTP server on `127.0.0.1:` with a 32-byte bearer token written to `port-.json`; client discovers via pid-liveness scan of the dir. **Sidebar snapshot:** -- Purpose: Loader → TUI surface for quota/killswitch/routing without coupling the TUI to the auth storage schema. +- Purpose: Loader → TUI surface for quota/killswitch/routing without coupling the TUI to the auth storage schema; also owns machine-global sticky assignments. - Location: `packages/opencode/src/sidebar-state.ts` -- Pattern: Promise-chained writes (no interleaved/stale writes); file path bound at loader-run time; `normalizeSidebarState` is the tolerant-read entry point so a malformed file never crashes the TUI. Writes machine-wide quota state via `setSidebarMachineState` and session-specific active routing records via `upsertSidebarActiveRouting`, preserving concurrency through a file-level write lock and a promise serialization chain. +- Pattern: Promise-chained writes (no interleaved/stale writes); file path bound at loader-run time; `normalizeSidebarState` is the tolerant-read entry point so a malformed file never crashes the TUI. Writes machine-wide quota state via `setSidebarMachineState`, short-lived session display records via `upsertSidebarActiveRouting`, and SHA-256-keyed sticky assignments via `resolveSidebarStickyAssignment`; file locking and a promise serialization chain preserve concurrent placement and pending-byte accounting. Pins expire after seven days. ## Entry Points @@ -256,10 +255,12 @@ - **In-memory quota cache:** `QuotaManager` (per-account fingerprint; 5-min refresh-after default; `respectBackoff` gates active polling). - **Prompt cache keep-warm:** `CacheKeepManager` tracks per-session last request and replays as `store:false` before the Codex ~5-min eviction window. +`/openai-cachekeep sustain on|off` is main-agent-only and defaults off. It is orthogonal to the clock window, retains memory/LRU limits, and never warms a non-active account. GPT-5.6 targets warm about twice per hour per session (about 1K output tokens/hour at about 99.4% cache hit); non-5.6 targets warm about twelve times per hour. Before enabling it for a main-session model, the operator must preserve existing entries in `~/.config/cortexkit/magic-context.jsonc` and set that model's `cache_ttl` to `"never"`. Magic Context does not run in subagent sessions. An indefinitely live cache invalidates elapsed-time assumptions that a cache is cold and that mutation is free. The sibling anthropic plugin's `always` means ignore the clock schedule; this plugin's `sustain` means bypass main idle pruning. + **Storage:** Config and state are stored in two separate files under `$OPENCODE_CONFIG_DIR`: config at `openai-auth.json` (default `~/.config/opencode/openai-auth.json`, overridable via `OPENCODE_OPENAI_AUTH_FILE`) containing settings and metadata without credentials, and state at `openai-auth-state.json` (overridable via `OPENCODE_OPENAI_AUTH_STATE_FILE`) containing access/refresh tokens and API keys. Atomic writes via `writeJsonAtomic` (temp + `rename`, mode `0o600`). File-level locks at `.save.lock` and `.main-refresh.lock` coordinate cross-process refresh and quota seed. A separate `openai-auth-sessions.json` persists Codex UUIDv7 thread/turn ids for prompt-cache continuity. Sidebar state lives at `tmpdir/opencode-openai-auth/sidebar-state.json` (override `OPENCODE_OPENAI_AUTH_SIDEBAR_STATE_FILE`). Loopback RPC port files live in `$XDG_STATE_HOME/cortexkit/openai-auth/rpc//port-.json`. **Configuration resolution (`config.ts`):** Env wins over config file wins over default. The `webSearch` cache fix is default-on and gated by a NEGATIVE env (`CORTEXKIT_OPENAI_AUTH_NO_WEB_SEARCH`). Booleans accept `1`/`true`/`yes`/`on` and `0`/`false`/`no`/`off`/empty. Settings are memoized per process; tests call `resetSettingsForTest`. **Versioning & build:** `packages/opencode/src/version.ts` exposes `PackageVersion` (currently `0.3.4`); the TUI plugin header reads `package.json` at runtime via `import.meta.url` so the version badge tracks the package version without baking it into the dist. Use `packages/opencode/scripts/build-tui.ts` during the build to precompile TUI Solid JSX source files into `packages/opencode/src/tui-compiled/` using the `@opentui/solid` compiler transform, binding Solid/OpenTUI imports to the host's virtual runtime registry (`opentui:runtime-module:`) so the TUI shares the host's single Solid/OpenTUI runtime. The release pipeline is tag-driven (`.github/workflows` + `scripts/release.sh`); see `README.md` for the exact command surface. -**Formatting/linting:** Biome 2.4.16 (single quotes, no semicolons, trailing commas, 2-space indent). Lefthook runs `biome check` on staged files. Tests run via `bun test src/tests`; typecheck via `tsc`. \ No newline at end of file +**Formatting/linting:** Biome 2.4.16 (single quotes, no semicolons, trailing commas, 2-space indent). Lefthook runs `biome check` on staged files. Tests run via `bun test src/tests`; typecheck via `tsc`. diff --git a/README.md b/README.md index a8ec47d..f5dcf6a 100644 --- a/README.md +++ b/README.md @@ -61,7 +61,7 @@ The plugin supports more than one ChatGPT account: a single **main** account (th - **Remove** a fallback with `/openai-account remove `. - Each account is identified by its stable ChatGPT account id, so the same account is never added twice. -Which account serves is decided by [routing](#routing) mode — there is no manual "active account" to pin. A request that a fallback can serve is buffered so the retry is safe, and selection skips accounts the [killswitch](#killswitch) has gated out. +Which account serves is decided by [routing](#routing) mode. There is no manual active-account selector: `sticky-balanced` creates a session pin automatically, while the other modes follow their configured order. A request that a fallback can serve is buffered so the retry is safe, and selection skips accounts the [killswitch](#killswitch) has gated out. ### Routing @@ -71,6 +71,11 @@ Which account serves is decided by [routing](#routing) mode — there is no manu | --- | --- | | `main-first` (default) | Send on the main account; on a `401`/`403`/`429`, transparently retry on the next usable fallback. | | `fallback-first` | Try usable fallback accounts first (preserving the main account's quota), and fall through to the main account only if no fallback can serve. | +| `sticky-balanced` | For a cold session, choose the account with the lowest projected pressure against its usable quota. Keep that account pinned for the session. | + +`sticky-balanced` does not rebalance mid-session. A pin stays in place until fresh quota confirms exhaustion or the account has a permanent authentication failure (`401`/`403`). A transient failure, stale or unknown quota, and `429` without confirmed exhaustion retain the pin; there is no Retry-After hold. This avoids changing an account while a session's continuation and cache context still belong to the first account. + +Pins use a SHA-256 hash of the session id as their key in the machine-global sidebar state and expire after seven days. Cold placement excludes accounts with stale or unknown quota from weighted selection. If every account is excluded, it fails open to the configured mode order. Equal projected-pressure scores break deterministically by configured order, then account id; a shared pending-byte bridge makes simultaneous cold placements account for each other. Subagents receive independent pins, and a resumed subagent reuses its own pin. `/openai-routing reset` clears only the current session's pin, so the next placement may legitimately choose the same account again. ### Killswitch @@ -94,9 +99,29 @@ Codex reports usage on **two rolling windows** — a 5-hour primary window and a - `/openai-cachekeep on` / `off` — enable or disable. The setting is **persisted**, so it stays on across restarts and applies to every session until you turn it off. - `/openai-cachekeep subagents on` / `off` — also keep subagent sessions warm (off by default). Useful when the same subagent is reused repeatedly. Subagent sessions warm only while recently active (a 30-minute idle cap, versus one hour for the main session). +- `/openai-cachekeep sustain on` / `off` — bypass main-session idle pruning. It defaults to off, is main-agent-only, and does not change subagent limits. - `/openai-cachekeep` — show status: enabled state, subagent mode, tracked sessions, and last-warm cost. -Keep-warm only ever runs for **main-agent** sessions unless subagent mode is on, and an idle session stops warming once it passes its idle cap (then resumes if it becomes active again). Each warm reuses the session's own cached prefix, so its marginal cost is small (typically a near-100% cache hit plus a few dozen output tokens). +Keep-warm only ever runs for **main-agent** sessions unless subagent mode is on, and it never warms an account that is not serving that session. `sustain` is orthogonal to the clock window: the configured window still controls capture and warming. `sustain` bypasses only the main idle-pruning bound; the target-count, memory-byte, and LRU caps still apply. Subagent idle and warm-count limits remain unchanged. + +Cost matters. A GPT-5.6 session needs about two warms per hour, roughly 1K output tokens per hour at about a 99.4% cache hit rate. Non-5.6 sessions need about twelve warms per hour, which is the higher-cost case. + +`sustain` deliberately does not use the sibling `anthropic-auth` plugin's `always` term. In that plugin, `always` means ignore the clock schedule; here, `sustain` means bypass main idle pruning. They control different axes. + +### Sustain prerequisite: Magic Context + +Before enabling sustain for a main-session model, update `~/.config/cortexkit/magic-context.jsonc` with Magic Context's supported schema. Preserve existing per-model entries and set only models used by sustained **main** sessions to `"never"`: + +```jsonc +{ + "cache_ttl": { + "default": "5m", + "openai/gpt-5.6-sol": "never" + } +} +``` + +Magic Context does not run in subagent sessions, so no subagent exception is needed. This setting is required because keeping a cache alive indefinitely falsifies an elapsed-time heuristic that assumes a cache is cold and therefore safe to mutate for free. ## Logging @@ -117,9 +142,9 @@ All commands open an interactive control surface in the TUI (a selectable dialog | --- | --- | --- | | `/openai-quota` | — | Show 5h + weekly quota for all accounts (polls the usage endpoint). | | `/openai-account` | `add [label]` · `remove ` · `order ` | List and manage accounts; add runs OAuth, order swaps fallback positions. | -| `/openai-routing` | `main-first` · `fallback-first` | Set account preference order. | +| `/openai-routing` | `main-first` · `fallback-first` · `sticky-balanced` · `reset` | Set account preference order, enable sticky balanced routing, or clear the current session pin. | | `/openai-killswitch` | `on` · `off` · `set :<5h>,<1w> ...` | Hard-block accounts below per-window quota thresholds. | -| `/openai-cachekeep` | `on` · `off` · `subagents on` · `subagents off` | Idle prompt-cache keep-warm; optional subagent mode. | +| `/openai-cachekeep` | `on` · `off` · `subagents on` · `subagents off` · `sustain on` · `sustain off` | Idle prompt-cache keep-warm; optional subagent mode and main-only idle-pruning bypass. | | `/openai-logging` | `` | Set log level (`error`/`warn`/`info`/`debug`/`trace`) live. | | `/openai-dump` | `on` · `off` | Toggle transport request dumps for cache debugging. | diff --git a/STRUCTURE.md b/STRUCTURE.md index c53c61a..a903c47 100644 --- a/STRUCTURE.md +++ b/STRUCTURE.md @@ -19,7 +19,7 @@ │ │ │ ├── logger.ts # Leveled, redacting, rotating logger │ │ │ ├── model-costs.ts # Dev catalog parser and cost restorer │ │ │ ├── quota-normalize.ts # HTTP/WS/wham → OAuthQuotaSnapshot -│ │ │ ├── sidebar-state.ts # Loader→TUI snapshot file +│ │ │ ├── sidebar-state.ts # Loader→TUI snapshot and sticky-pin state │ │ │ ├── tui-preferences.ts # Shared tui-preferences.jsonc reader/writer/watcher │ │ │ ├── tui.tsx # TUI sidebar component │ │ │ ├── ws.ts # Low-level WS connect/stream @@ -74,7 +74,7 @@ - Key files: - `packages/opencode/src/core/accounts.ts` — `loadAccounts`/`mutateAccounts` (authoritative read-modify-write), `saveAccounts` (test seeding only), `saveAccountState` (updates state secrets, gated by config roster), `FallbackAccountManager`, account types - `packages/opencode/src/core/quota-manager.ts` — in-memory quota cache, backoff, and mid-stream rate limit marking - - `packages/opencode/src/core/cachekeep.ts` — `CacheKeepManager` (idle prompt-cache warmer with model-aware TTLs, subagent 2-warm limits, clock windows, and idle pruning) + - `packages/opencode/src/core/cachekeep.ts` — `CacheKeepManager` (idle prompt-cache warmer with model-aware TTLs, subagent 2-warm limits, clock windows, idle pruning, and main-only sustain) - `packages/opencode/src/core/oauth.ts` — PKCE, callback server, device-code flow, JWT parsing - `packages/opencode/src/core/backoff.ts` — refresh/quota backoff math + `hashRefreshToken` - `packages/opencode/src/core/refresh-file-lock.ts` — single-writer eviction-marker lock @@ -136,7 +136,7 @@ - `packages/opencode/src/core/accounts.ts` — multi-account store, `FallbackAccountManager`. - `packages/opencode/src/core/oauth.ts` — PKCE, OAuth flow, JWT parsing. - `packages/opencode/src/core/quota-manager.ts` — quota cache, backoff, and mid-stream rate limit marking. -- `packages/opencode/src/core/cachekeep.ts` — prompt-cache warmer with model-aware TTL, clock window, and subagent warm caps. +- `packages/opencode/src/core/cachekeep.ts` — prompt-cache warmer with model-aware TTL, clock window, subagent warm caps, and main-only sustain that bypasses idle pruning but not memory/LRU caps. - `packages/opencode/src/prompt-context.ts` — assistant model/variant resolver for synthetic command replies. - `packages/opencode/src/core/provider.ts` — Codex injection seam (`codexRefreshFn`, `whamUsageFn`). - `packages/opencode/src/core/backoff.ts` — retry/backoff math. @@ -147,7 +147,7 @@ - `packages/opencode/src/raw-ws-bun.ts` / `packages/opencode/src/raw-ws-node.ts` — hand-rolled RFC 6455 clients. - `packages/opencode/src/hosted-web-search.ts` — provider-hosted `web_search` tool + replay/SSE translation. - `packages/opencode/src/quota-normalize.ts` — HTTP/WS/wham → `OAuthQuotaSnapshot`. -- `packages/opencode/src/sidebar-state.ts` — loader→TUI snapshot file + tolerant reader. +- `packages/opencode/src/sidebar-state.ts` — loader→TUI snapshot, tolerant reader, and SHA-256-keyed sticky session assignments with seven-day TTL. - `packages/opencode/src/dump.ts` — optional transport request dumps for cache debugging. - `packages/opencode/src/logger.ts` — leveled, secret-redacting, size-rotating logger. - `packages/opencode/src/model-costs.ts` — model cost resolution and restoration from `models.dev` catalog. @@ -201,4 +201,4 @@ Example: `CORTEXKIT_OPENAI_AUTH_WEBSOCKETS`, `CORTEXKIT_OPENAI_AUTH_RAW_WS`, `OP **New package (sibling to `opencode` or `pi`):** create `packages//` with its own `package.json`, `src/`, `tsconfig.json`, `tsconfig.build.json`, and add it under `workspaces` in the root `package.json`. Mirror the existing `opencode` or `pi` layout — Bun workspaces, `bun run build`, `bun run typecheck`. -**New plugin command constant / TUI preferences key:** add to `packages/opencode/src/tui-preferences.ts` (`DEFAULT_PREFS` + a typed key under the `PLUGIN_KEY = 'openai-auth'` top-level key in `~/.config/opencode/tui-preferences.jsonc`); the schema-validated reader will accept the new key automatically because `resolveOpenaiAuthPrefs` per-key defaults. \ No newline at end of file +**New plugin command constant / TUI preferences key:** add to `packages/opencode/src/tui-preferences.ts` (`DEFAULT_PREFS` + a typed key under the `PLUGIN_KEY = 'openai-auth'` top-level key in `~/.config/opencode/tui-preferences.jsonc`); the schema-validated reader will accept the new key automatically because `resolveOpenaiAuthPrefs` per-key defaults. diff --git a/packages/opencode/README.md b/packages/opencode/README.md index cce4ce0..6f88fbf 100644 --- a/packages/opencode/README.md +++ b/packages/opencode/README.md @@ -26,9 +26,9 @@ Restart OpenCode after changing plugin config, then authenticate: - Codex request rewriting for OAuth requests, with Codex identity parity. - OAuth model filtering and zero-cost display. - Prompt-cache stabilizer (`web_search`) that keeps tool-continuation requests on the backend's cached path (on by default). -- Multiple ChatGPT accounts with automatic reactive fallback on rate limits, configurable routing, and a per-account quota killswitch. +- Multiple ChatGPT accounts with automatic reactive fallback on rate limits, `main-first`, `fallback-first`, or sticky-balanced routing, and a per-account quota killswitch. - Per-turn quota tracking (5-hour + weekly windows) on both transports, with a sidebar readout and an explicit all-accounts refresh. -- Idle prompt-cache keep-warm, with an optional subagent mode. +- Idle prompt-cache keep-warm, with an optional subagent mode and main-only sustain mode. - Leveled, secret-redacting, rotating log file. - Interactive in-TUI control surfaces for every command, plus an `openai-auth` CLI for managing fallback accounts headlessly. - Optional OpenAI Responses WebSocket transport (HTTP is the default). @@ -41,9 +41,9 @@ Each opens an interactive dialog in the TUI and also accepts explicit arguments: | --- | --- | --- | | `/openai-quota` | — | Show 5h + weekly quota for all accounts. | | `/openai-account` | `add [label]` · `remove ` · `order ` | Manage main + fallback accounts. | -| `/openai-routing` | `main-first` · `fallback-first` | Routing mode: which account is tried first. | +| `/openai-routing` | `main-first` · `fallback-first` · `sticky-balanced` · `reset` | Routing order, sticky balanced session pins, or clear the current pin. | | `/openai-killswitch` | `on` · `off` · `set :<5h>,<1w> ...` | Hard-block accounts below quota thresholds. | -| `/openai-cachekeep` | `on` · `off` · `subagents on` · `subagents off` | Idle prompt-cache keep-warm. | +| `/openai-cachekeep` | `on` · `off` · `subagents on` · `subagents off` · `sustain on` · `sustain off` | Idle prompt-cache keep-warm; sustain bypasses only main idle pruning. | | `/openai-logging` | `` | Set log level live. | | `/openai-dump` | `on` · `off` | Toggle transport request dumps. | @@ -71,6 +71,10 @@ Settings resolve as environment variable → config file (`~/.config/opencode/op See the [repository README](https://github.com/cortexkit/openai-auth#readme) for transport differences and why `web_search` is needed. +`sticky-balanced` places a cold session by least projected quota pressure, then keeps its SHA-256-keyed sidebar-state pin for up to seven days. It does not rebalance mid-session or use a Retry-After hold; it migrates only after confirmed exhaustion or permanent auth failure. Stale or unknown quota is excluded from weighted placement; when the killswitch is enabled, accounts below their per-account threshold are also excluded from both weighted placement AND the mode-fallback fail-open branch — that branch otherwise orders by `resetCreditsApplicable` first, then configured order, then account id. Subagents have separate pins and reuse them when resumed. + +`/openai-cachekeep sustain on|off` defaults off, applies only to main sessions, remains subject to the clock window and memory/LRU caps, and never warms non-active accounts. It costs about two GPT-5.6 warms per hour per session (about 1K output tokens/hour at about 99.4% cache hit); non-5.6 sessions warm about twelve times per hour. Before enabling it for a main-session model, preserve existing entries in `~/.config/cortexkit/magic-context.jsonc` and set that model's `cache_ttl` to `"never"`; Magic Context does not run in subagents. `sustain` means bypass main idle pruning, unlike the sibling anthropic plugin's `always`, which means ignore the clock schedule. + ## License MIT diff --git a/packages/opencode/src/commands.ts b/packages/opencode/src/commands.ts index 426d1f6..0bb9e8c 100644 --- a/packages/opencode/src/commands.ts +++ b/packages/opencode/src/commands.ts @@ -76,8 +76,14 @@ export interface CommandContext { setCacheKeepEnabled?: (enabled: boolean) => void /** Updates the live loader's persisted-subagent cachekeep gate. */ setCacheKeepSubagents?: (enabled: boolean) => void + /** Updates the live loader's main-agent idle-cap bypass gate. */ + setCacheKeepSustain?: (enabled: boolean) => void /** Updates the live loader's clock-hour warm window. undefined = no window. */ setCacheKeepWindow?: (window: CacheKeepWindow | undefined) => void + /** Clears only the sticky account assignment for one OpenCode session. */ + clearStickyRouting?: (sessionId: string) => Promise + /** Resolves the current session's usable sticky account, if one exists. */ + getStickyRouting?: (sessionId: string) => Promise } const log = createLogger('commands') @@ -87,9 +93,13 @@ const log = createLogger('commands') // --------------------------------------------------------------------------- function routingDescription(mode: RoutingMode) { - return mode === 'fallback-first' - ? 'Try usable fallback accounts before the main account.' - : 'Try the main account first. Use fallback accounts only when required.' + if (mode === 'fallback-first') { + return 'Try usable fallback accounts before the main account.' + } + if (mode === 'sticky-balanced') { + return 'Keep each session on its assigned account while balancing new sessions.' + } + return 'Try the main account first. Use fallback accounts only when required.' } // --------------------------------------------------------------------------- @@ -181,7 +191,9 @@ async function executeAccountCommand( ) } else { const mode: RoutingMode = storage.routing?.mode ?? 'main-first' - lines.push(`Routing: \`${mode}\` (set with \`/openai-routing\`).`) + lines.push( + `Routing: \`${mode}\` (set with \`/openai-routing\`). Modes: main-first, fallback-first, or sticky-balanced. \`/openai-routing reset\` clears this session's pin.`, + ) lines.push('') for (const a of accounts) { const type = (a as { type?: string }).type ?? 'oauth' @@ -358,7 +370,7 @@ async function executeAccountCommand( return { command: 'openai-account', - text: '## Account Commands\n\n- `/openai-account` — show accounts\n- `/openai-account add [label]` — add a new account\n- `/openai-account remove ` — remove\n- `/openai-account order ` — swap fallback positions\n\nRouting is set with `/openai-routing` (main-first / fallback-first).', + text: '## Account Commands\n\n- `/openai-account` — show accounts\n- `/openai-account add [label]` — add a new account\n- `/openai-account remove ` — remove\n- `/openai-account order ` — swap fallback positions\n\nRouting modes are `main-first`, `fallback-first`, and `sticky-balanced`. `/openai-routing reset` clears the current session pin.', knobs: { accounts }, } } @@ -374,9 +386,35 @@ async function executeRoutingCommand( } const currentMode: RoutingMode = storage.routing?.mode ?? 'main-first' + if (tokens.length === 1 && tokens[0] === 'reset') { + if (!ctx.sessionId) { + return { + command: 'openai-routing', + text: '## OpenAI Routing Reset\n\nNo current session is available, so no pin was changed.', + knobs: { mode: currentMode }, + } + } + if (!ctx.clearStickyRouting) { + return { + command: 'openai-routing', + text: '## OpenAI Routing Reset\n\nThis runtime cannot clear the current session pin.', + knobs: { mode: currentMode }, + } + } + await ctx.clearStickyRouting(ctx.sessionId) + log.info('routing session pin cleared') + return { + command: 'openai-routing', + text: "## OpenAI Routing Reset\n\nThis session's pin was cleared. The next request may choose the same account if it remains the best selection.", + knobs: { mode: currentMode }, + } + } + if ( tokens.length === 1 && - (tokens[0] === 'main-first' || tokens[0] === 'fallback-first') + (tokens[0] === 'main-first' || + tokens[0] === 'fallback-first' || + tokens[0] === 'sticky-balanced') ) { const mode = tokens[0] as RoutingMode // Scalar-field write MUST go through mutateAccounts (read-fresh under lock, @@ -390,14 +428,25 @@ async function executeRoutingCommand( log.info('routing mode changed', { mode }) return { command: 'openai-routing', - text: `## OpenAI Routing Updated\n\nMode: \`${mode}\`\n- ${routingDescription(mode)}\n\nUsage: \`/openai-routing\`, \`/openai-routing main-first\`, or \`/openai-routing fallback-first\`.`, + text: `## OpenAI Routing Updated\n\nMode: \`${mode}\`\n- ${routingDescription(mode)}\n\nUsage: \`/openai-routing\`, \`/openai-routing main-first\`, \`/openai-routing fallback-first\`, or \`/openai-routing sticky-balanced\`.`, knobs: { mode }, } } + const stickyPin = + currentMode === 'sticky-balanced' && ctx.sessionId + ? await ctx.getStickyRouting?.(ctx.sessionId) + : undefined + const stickyPinDescription = + currentMode === 'sticky-balanced' + ? stickyPin + ? `\n- Session pin: \`${stickyPin}\`. Use \`/openai-routing reset\` to clear it.` + : '\n- Session pin: none yet. A request will choose one when a usable account is available.' + : '' + return { command: 'openai-routing', - text: `## OpenAI Routing\n\n- Mode: \`${currentMode}\`\n- ${routingDescription(currentMode)}\n\nUsage: \`/openai-routing\`, \`/openai-routing main-first\`, or \`/openai-routing fallback-first\`.`, + text: `## OpenAI Routing\n\n- Mode: \`${currentMode}\`\n- ${routingDescription(currentMode)}${stickyPinDescription}\n\nUsage: \`/openai-routing\`, \`/openai-routing main-first\`, \`/openai-routing fallback-first\`, or \`/openai-routing sticky-balanced\`.`, knobs: { mode: currentMode }, } } @@ -687,6 +736,7 @@ async function executeCachekeepCommand( `Status: **${enabled ? 'ON' : 'OFF'}**`, `Timer: **${status.running ? 'armed' : 'idle'}**`, `Subagent warming: **${storage?.cachekeep?.subagents === true ? 'ON' : 'OFF'}**`, + `Idle policy: **sustain ${status.sustain ? 'ON' : 'OFF'} (main only)**`, `Window: **${windowLabel}**`, ] lines.push(`Tracked sessions: **${status.tracked}**`) @@ -717,7 +767,7 @@ async function executeCachekeepCommand( ) lines.push('') lines.push( - 'Commands: `/openai-cachekeep on` | `/openai-cachekeep off` | `/openai-cachekeep HH-HH` | `/openai-cachekeep window clear` | `/openai-cachekeep subagents on` | `/openai-cachekeep subagents off` | `/openai-cachekeep`', + 'Commands: `/openai-cachekeep on` | `/openai-cachekeep off` | `/openai-cachekeep sustain on` | `/openai-cachekeep sustain off` | `/openai-cachekeep HH-HH` | `/openai-cachekeep window clear` | `/openai-cachekeep subagents on` | `/openai-cachekeep subagents off` | `/openai-cachekeep`', ) const lastWarmAt = Math.max( 0, @@ -729,6 +779,7 @@ async function executeCachekeepCommand( knobs: { enabled, subagents: storage?.cachekeep?.subagents === true, + sustain: status.sustain, window: liveWindow, running: status.running, tracked: status.tracked, @@ -766,6 +817,7 @@ async function executeCachekeepCommand( knobs: { enabled: true, subagents: storage?.cachekeep?.subagents === true, + sustain: status.sustain, window: status.window, running: status.running, tracked: status.tracked, @@ -788,7 +840,51 @@ async function executeCachekeepCommand( return { command: 'openai-cachekeep', text: '## Cachekeep Disabled', - knobs: { enabled: false, running: false, tracked: 0 }, + knobs: { + enabled: false, + sustain: storage?.cachekeep?.sustain === true, + running: false, + tracked: 0, + }, + } + } + + if (tokens[0] === 'sustain') { + const sustainCmd = tokens[1] + if (tokens.length !== 2 || (sustainCmd !== 'on' && sustainCmd !== 'off')) { + return { + command: 'openai-cachekeep', + text: 'Usage: `/openai-cachekeep sustain on` | `/openai-cachekeep sustain off`', + knobs: {}, + } + } + const value = sustainCmd === 'on' + await mutateAccounts((current) => { + current.cachekeep = { + ...(current.cachekeep ?? {}), + sustain: value, + } + return current + }, ctx.accountStoragePath) + log.info(`cachekeep sustain ${value ? 'enabled' : 'disabled'}`) + ctx.setCacheKeepSustain?.(value) + const nextStatus = mgr?.status() + return { + command: 'openai-cachekeep', + text: value + ? '## Cachekeep Sustain Enabled\n\nSustain keeps main-agent sessions warming past the idle cap for this process. Clock windows still apply.\n\nBefore enabling sustain with Magic Context, set a non-expiring `cache_ttl` for models used by main sessions; elapsed-time cold-cache assumptions are no longer valid.' + : '## Cachekeep Sustain Disabled\n\nMain-agent sessions again stop warming at the configured idle cap.', + knobs: { + enabled, + subagents: storage?.cachekeep?.subagents === true, + sustain: value, + window: nextStatus?.window, + running: nextStatus?.running ?? false, + tracked: nextStatus?.tracked ?? 0, + generatedAt: nextStatus?.generatedAt ?? Date.now(), + maxIdleWarmMs: nextStatus?.maxIdleWarmMs ?? 60 * 60 * 1000, + maxSubagentIdleMs: nextStatus?.maxSubagentIdleMs ?? 30 * 60 * 1000, + }, } } @@ -822,6 +918,7 @@ async function executeCachekeepCommand( knobs: { enabled, subagents: value, + sustain: nextStatus?.sustain ?? storage?.cachekeep?.sustain === true, window: nextStatus?.window, running: nextStatus?.running ?? false, tracked: nextStatus?.tracked ?? 0, @@ -853,6 +950,7 @@ async function executeCachekeepCommand( knobs: { enabled, subagents: storage?.cachekeep?.subagents === true, + sustain: nextStatus?.sustain ?? storage?.cachekeep?.sustain === true, window: undefined, running: nextStatus?.running ?? false, tracked: nextStatus?.tracked ?? 0, @@ -899,6 +997,7 @@ async function executeCachekeepCommand( knobs: { enabled, subagents: storage?.cachekeep?.subagents === true, + sustain: nextStatus?.sustain ?? storage?.cachekeep?.sustain === true, window: { startHour, endHour }, running: nextStatus?.running ?? false, tracked: nextStatus?.tracked ?? 0, @@ -911,7 +1010,7 @@ async function executeCachekeepCommand( return { command: 'openai-cachekeep', - text: 'Usage: `/openai-cachekeep`, `/openai-cachekeep on`, `/openai-cachekeep off`, `/openai-cachekeep HH-HH`, `/openai-cachekeep window clear`, `/openai-cachekeep subagents on`, `/openai-cachekeep subagents off`', + text: 'Usage: `/openai-cachekeep`, `/openai-cachekeep on`, `/openai-cachekeep off`, `/openai-cachekeep sustain on`, `/openai-cachekeep sustain off`, `/openai-cachekeep HH-HH`, `/openai-cachekeep window clear`, `/openai-cachekeep subagents on`, `/openai-cachekeep subagents off`', knobs: {}, } } diff --git a/packages/opencode/src/core/accounts.ts b/packages/opencode/src/core/accounts.ts index 9b0f6b5..a6fca4a 100644 --- a/packages/opencode/src/core/accounts.ts +++ b/packages/opencode/src/core/accounts.ts @@ -161,7 +161,7 @@ export function isValidApiBaseURL(value: string | undefined) { // Storage types // --------------------------------------------------------------------------- -export type RoutingMode = 'main-first' | 'fallback-first' +export type RoutingMode = 'main-first' | 'fallback-first' | 'sticky-balanced' export type KillswitchThresholds = Partial< Record @@ -180,9 +180,8 @@ export type AccountStorage = { provider: 'openai' } routing?: { - // Routing is purely mode-driven (main-first | fallback-first). There is no - // persisted "active account" pin — the account that serves each request is - // decided per-request by the mode + quota/killswitch policy. + // Sticky-balanced retains a per-session pin in sidebar state; configuration + // here selects only the routing policy, never the serving account itself. mode?: RoutingMode } fallbackOn?: number[] @@ -220,6 +219,7 @@ export type AccountStorage = { cachekeep?: { enabled?: boolean subagents?: boolean + sustain?: boolean /** Clock-hour window start (0-23, inclusive) — keeps cachekeep idle warming * inside `[startHour, endHour)` local hours. Omit to warm unconditionally. */ startHour?: number @@ -1126,7 +1126,7 @@ function normalizeKillswitchThresholds( } } -function getKillswitchThresholdsForAccount( +export function getKillswitchThresholdsForAccount( storage: AccountStorage | null, accountId?: string, ): { primary: number; secondary: number } { diff --git a/packages/opencode/src/core/cachekeep.ts b/packages/opencode/src/core/cachekeep.ts index 02f0f49..09e1da2 100644 --- a/packages/opencode/src/core/cachekeep.ts +++ b/packages/opencode/src/core/cachekeep.ts @@ -52,6 +52,8 @@ export interface CacheKeepManagerOptions { maxBytes?: number /** Returns the configured clock-hour window; undefined means always warm. */ getWindow?: () => CacheKeepWindow | undefined + /** Returns whether main-agent targets bypass only the idle warm cap. */ + getSustain?: () => boolean } export interface CacheKeepStatus { @@ -63,6 +65,7 @@ export interface CacheKeepStatus { maxSubagentIdleMs: number ttlMs: number leadMs: number + sustain: boolean window?: CacheKeepWindow targets: Array<{ sessionKey: string @@ -317,6 +320,7 @@ export class CacheKeepManager { private readonly maxTargets: number private readonly maxBytes: number private readonly getWindow?: () => CacheKeepWindow | undefined + private readonly getSustain?: () => boolean private timer: ReturnType | null = null private startedAt: number | null = null @@ -360,6 +364,7 @@ export class CacheKeepManager { this.maxTargets = options.maxTargets ?? DEFAULT_MAX_TARGETS this.maxBytes = options.maxBytes ?? DEFAULT_MAX_BYTES this.getWindow = options.getWindow + this.getSustain = options.getSustain } // -- public API ------------------------------------------------------------ @@ -484,7 +489,10 @@ export class CacheKeepManager { ? this.maxSubagentIdleMs : this.maxIdleWarmMs } - if (target.lastRealRequestAt < bound) { + if ( + !(target.isSubagent !== true && this.getSustain?.() === true) && + target.lastRealRequestAt < bound + ) { this.log.debug( 'cachekeep pruned idle target', this.logPayload({ @@ -570,6 +578,7 @@ export class CacheKeepManager { maxSubagentIdleMs: this.maxSubagentIdleMs, ttlMs: this.ttlMs, leadMs: this.leadMs, + sustain: this.getSustain?.() === true, window: this.getWindow?.(), targets, } diff --git a/packages/opencode/src/core/sticky-routing.ts b/packages/opencode/src/core/sticky-routing.ts new file mode 100644 index 0000000..1846d77 --- /dev/null +++ b/packages/opencode/src/core/sticky-routing.ts @@ -0,0 +1,274 @@ +import { + type AccountQuota, + getPresentQuotaWindows, + type QuotaWindow, + type QuotaWindowKey, +} from '../sidebar-state' + +export const QUOTA_STALENESS_MS = 15 * 60_000 +export const MIN_RESET_HOURS = 1 / 60 +export const MIN_WEIGHT = 1e-6 + +// Both primary and secondary windows are consulted, then the snapshot-level +// timestamp, then the cache entry's timestamp. An account whose only fresh +// window is the secondary must not be judged on the older primary stamp — the +// admission path in particular relies on this to avoid staling out a snapshot +// the wire just refreshed against a tighter window. +export function snapshotCheckedAt( + quota: AccountQuota | null | undefined, + entryCheckedAt?: number, +): number | undefined { + for (const checkedAt of [ + quota?.primary?.checkedAt, + quota?.secondary?.checkedAt, + quota?.checkedAt, + entryCheckedAt, + ]) { + if (typeof checkedAt === 'number' && Number.isFinite(checkedAt)) { + return checkedAt + } + } + return undefined +} + +export type StickyBreakDecision = + | { action: 'retain'; reason: 'unknown' | 'stale' | 'healthy' | 'transient' } + | { + action: 'migrate' + reason: 'exhausted' | 'permanent' | 'killswitch' + windowKey?: QuotaWindowKey + resetsAt?: string + } + +export function decideStickyBreak(input: { + quota: AccountQuota | null | undefined + quotaCheckedAt?: number + status?: number + now: number + killswitchPasses?: boolean +}): StickyBreakDecision { + if (input.status === 401 || input.status === 403) { + return { action: 'migrate', reason: 'permanent' } + } + if (!input.quota) return { action: 'retain', reason: 'unknown' } + + const checkedAt = snapshotCheckedAt(input.quota, input.quotaCheckedAt) + if ( + checkedAt === undefined || + !Number.isFinite(checkedAt) || + input.now - checkedAt > QUOTA_STALENESS_MS + ) { + return { action: 'retain', reason: 'stale' } + } + + // Placed AFTER the stale check so a stale snapshot never judges the account + // on a snap the killswitch would consider below floor. The caller is expected + // to pre-resolve the killswitch result using the non-invalidating policy peek + // so a routine token refresh does not flip a killed account to "unknown". + if (input.killswitchPasses === false) { + return { action: 'migrate', reason: 'killswitch' } + } + + const windows = getPresentQuotaWindows(input.quota).sort((left, right) => { + const leftWindowMs = left.windowMs + const rightWindowMs = right.windowMs + const leftKnown = + typeof leftWindowMs === 'number' && Number.isFinite(leftWindowMs) + const rightKnown = + typeof rightWindowMs === 'number' && Number.isFinite(rightWindowMs) + if (leftKnown !== rightKnown) return leftKnown ? -1 : 1 + if (leftKnown && rightKnown) return rightWindowMs - leftWindowMs + return 0 + }) + for (const { key, window } of windows) { + if ( + Number.isFinite(window.remainingPercent) && + window.remainingPercent <= 0 + ) { + return { + action: 'migrate', + reason: 'exhausted', + windowKey: key, + ...(typeof window.resetsAt === 'string' + ? { resetsAt: window.resetsAt } + : {}), + } + } + } + + if ( + input.status === undefined || + input.status === 0 || + !Number.isFinite(input.status) || + (input.status >= 500 && input.status <= 599) || + input.status === 429 + ) { + return { action: 'retain', reason: 'transient' } + } + return { action: 'retain', reason: 'healthy' } +} + +export function sustainableWindowWeight( + window: Pick, + reservePercent: number, + now: number, +): number { + const spendable = Math.max(0, window.remainingPercent - reservePercent) + if (spendable <= 0) return 0 + if (!window.resetsAt) return spendable + const resetMs = Date.parse(window.resetsAt) + // A lapsed reset can't yield a meaningful spend rate: the elapsed-hours divisor + // would clamp to MIN_RESET_HOURS and inflate the weight ~60x, favoring an account + // whose window has already rolled over on stale information. Fall back to the + // un-rate-adjusted spendable capacity instead. + if (!Number.isFinite(resetMs) || resetMs <= now) return spendable + const hours = Math.max((resetMs - now) / 3_600_000, MIN_RESET_HOURS) + return spendable / hours +} + +export interface StickySelectionCandidate { + accountId: string + quota: AccountQuota | null | undefined + quotaCheckedAt?: number + reservePercent: Record + configuredOrder: number + resetCreditsApplicable?: number + // Opt-in killswitch gate. When `false`, the candidate is excluded from both + // weighted placement AND the mode-fallback fail-open branch. Undefined or + // `true` is a no-op — the dominant path with killswitch disabled is + // byte-identical to the pre-killswitch behaviour. + killswitchPasses?: boolean +} + +export interface StickySelectionInput { + candidates: readonly StickySelectionCandidate[] + pendingBytes: ReadonlyMap + requestBytes: number + now: number + onEmptyWeightedSet?: () => void +} + +type WeightedCandidate = { + candidate: StickySelectionCandidate + quotaCheckedAt: number + weight: number +} + +function compareAccountIds(left: string, right: string): number { + if (left < right) return -1 + if (left > right) return 1 + return 0 +} + +function candidateWeight( + candidate: StickySelectionCandidate, + now: number, +): WeightedCandidate | undefined { + if (!candidate.quota) return undefined + const quotaCheckedAt = snapshotCheckedAt( + candidate.quota, + candidate.quotaCheckedAt, + ) + if ( + quotaCheckedAt === undefined || + now - quotaCheckedAt > QUOTA_STALENESS_MS + ) { + return undefined + } + const weights = getPresentQuotaWindows(candidate.quota).map( + ({ key, window }) => { + // Missing reserve data must leave the window usable rather than silently excluding its account. + return sustainableWindowWeight( + window, + candidate.reservePercent[key] ?? 0, + now, + ) + }, + ) + const weight = weights.length > 0 ? Math.min(...weights) : 0 + return weight > 0 ? { candidate, quotaCheckedAt, weight } : undefined +} + +export function selectStickyCandidate(input: StickySelectionInput): + | { + accountId: string + quotaCheckedAt?: number + source: 'weighted' | 'mode-fallback' + } + | undefined { + // Killswitch filter: a candidate whose stored killswitch result is `false` + // is excluded from BOTH weighted placement and the mode-fallback fail-open + // branch. The branch must never become a way to spend on a killed account. + // Undefined / `true` is a no-op so the killswitch-disabled path is + // byte-identical to the pre-killswitch behaviour. + const eligibleCandidates = input.candidates.filter( + (candidate) => candidate.killswitchPasses !== false, + ) + + // Empty input is a programmer error — preserve the existing throw. + // Every candidate killed by the killswitch filter is a routable state: the + // caller is expected to hand the same `killswitchBlockedResponse` the + // ordered modes produce rather than letting the resolver fall through. + if (input.candidates.length === 0) { + throw new Error( + 'Cannot select a sticky candidate: input.candidates is empty', + ) + } + if (eligibleCandidates.length === 0) { + return undefined + } + + const weighted = eligibleCandidates + .map((candidate) => candidateWeight(candidate, input.now)) + .filter( + (candidate): candidate is WeightedCandidate => candidate !== undefined, + ) + + if (weighted.length > 0) { + weighted.sort((left, right) => { + // MIN_WEIGHT only guards this division after the weight > 0 eligibility filter; it is not a tuning parameter. + const leftScore = + ((input.pendingBytes.get(left.candidate.accountId) ?? 0) + + input.requestBytes) / + Math.max(left.weight, MIN_WEIGHT) + const rightScore = + ((input.pendingBytes.get(right.candidate.accountId) ?? 0) + + input.requestBytes) / + Math.max(right.weight, MIN_WEIGHT) + return ( + leftScore - rightScore || + left.candidate.configuredOrder - right.candidate.configuredOrder || + compareAccountIds(left.candidate.accountId, right.candidate.accountId) + ) + }) + const selected = weighted[0] + if (selected) { + return { + accountId: selected.candidate.accountId, + quotaCheckedAt: selected.quotaCheckedAt, + source: 'weighted', + } + } + } + + input.onEmptyWeightedSet?.() + const fallback = [...eligibleCandidates].sort((left, right) => { + const leftHasCredits = (left.resetCreditsApplicable ?? 0) > 0 ? 1 : 0 + const rightHasCredits = (right.resetCreditsApplicable ?? 0) > 0 ? 1 : 0 + return ( + rightHasCredits - leftHasCredits || + left.configuredOrder - right.configuredOrder || + compareAccountIds(left.accountId, right.accountId) + ) + })[0] + if (!fallback) { + throw new Error( + 'Cannot select a sticky candidate: input.candidates is empty', + ) + } + return { + accountId: fallback.accountId, + quotaCheckedAt: snapshotCheckedAt(fallback.quota, fallback.quotaCheckedAt), + source: 'mode-fallback', + } +} diff --git a/packages/opencode/src/dump.ts b/packages/opencode/src/dump.ts index 970826e..fdfb360 100644 --- a/packages/opencode/src/dump.ts +++ b/packages/opencode/src/dump.ts @@ -200,6 +200,7 @@ export async function dumpCodexRequest(input: { transport: DumpTransport phase: DumpPhase bodyText: string + accountId?: string url?: string method?: string headers?: DumpHeaders @@ -231,6 +232,7 @@ export async function dumpCodexRequest(input: { session: shortSession(sessionID), transport: input.transport, phase: input.phase, + accountId: input.accountId, status: input.status, error: input.error, bodyBytes: input.bodyText.length, @@ -251,6 +253,7 @@ export async function dumpCodexRequest(input: { redactForDump({ url: input.url, method: input.method, + accountId: input.accountId, headers: headersToRecord(input.headers), }), null, diff --git a/packages/opencode/src/index.ts b/packages/opencode/src/index.ts index 90b28f3..1661216 100644 --- a/packages/opencode/src/index.ts +++ b/packages/opencode/src/index.ts @@ -26,6 +26,7 @@ import { type AccountStorage, type FallbackAccount, FallbackAccountManager, + getKillswitchThresholdsForAccount, isCostZeroingEnabled, isKillswitchEnabled, isOAuthAccount, @@ -72,6 +73,12 @@ import { } from './core/quota-manager' import { refreshAllQuota } from './core/refresh-all-quota' import { acquireRefreshFileLock } from './core/refresh-file-lock' +import { + decideStickyBreak, + type StickyBreakDecision, + selectStickyCandidate, + snapshotCheckedAt, +} from './core/sticky-routing' import { DUMP_SESSION_HEADER, dumpCodexRequest } from './dump' import { HostedWebSearchTool, @@ -99,11 +106,15 @@ import { getRpcDir } from './rpc/rpc-dir' import { type RpcServerHandle, startRpcServer } from './rpc/rpc-server' import { type AccountQuota, + clearSidebarStickyAssignment, exhaustedQuotaResetAt, getSidebarState, getSidebarStateFile, + hashSidebarSessionId, isQuotaExhausted, removeSidebarActiveRouting, + resolveSessionStickyAccount, + resolveSidebarStickyAssignment, type SidebarMachineState, type SidebarState, setSidebarLegacyRouting, @@ -761,7 +772,7 @@ export async function CodexAuthPlugin( } if (codexSessions.delete(info.id)) persistCodexSessions() if (sidebarStateFileForEvents) { - const accounts = (await loadAccounts(getConfigPath()))?.accounts ?? [] + const accounts = (await loadAccounts(getConfigPath()))?.accounts await removeSidebarActiveRouting( info.id, accounts, @@ -974,6 +985,7 @@ export async function CodexAuthPlugin( const cacheKeepLogger = createLogger('cachekeep') let cacheKeepEnabled = storage?.cachekeep?.enabled === true let cacheKeepSubagents = storage?.cachekeep?.subagents === true + let cacheKeepSustain = storage?.cachekeep?.sustain === true let cacheKeepWindow = getCacheKeepWindow(storage) let mainRefreshPromise: | Promise<{ access: string; refresh: string; expires: number }> @@ -1219,6 +1231,7 @@ export async function CodexAuthPlugin( logger: cacheKeepLogger, now: Date.now, getWindow: () => cacheKeepWindow, + getSustain: () => cacheKeepSustain, }) cacheKeepGlobal.__openaiAuthCacheKeepManager = cacheKeepManager @@ -1424,7 +1437,7 @@ export async function CodexAuthPlugin( parentSessionId: string | undefined, activeId: string, route: RoutingMode, - accounts: readonly { id: string; enabled?: boolean }[], + accounts: readonly { id: string; enabled?: boolean }[] | undefined, ) { const input = { activeId, route, updatedAt: Date.now() } if (sessionId) { @@ -1434,6 +1447,31 @@ export async function CodexAuthPlugin( boundSidebarFile, ) if (parentSessionId && parentSessionId !== sessionId) { + if (route === 'sticky-balanced') { + const parentPinnedId = (await getSidebarState(boundSidebarFile)) + .stickyAssignments?.[hashSidebarSessionId(parentSessionId)] + ?.accountId + const parentPinIsUsable = + accounts === undefined || + parentPinnedId === 'main' || + accounts.some( + (account) => + account.enabled !== false && + account.id === parentPinnedId, + ) + if (!parentPinnedId || !parentPinIsUsable) return + await upsertSidebarActiveRouting( + { + sessionId: parentSessionId, + activeId: parentPinnedId, + route, + updatedAt: Date.now(), + }, + accounts, + boundSidebarFile, + ) + return + } await upsertSidebarActiveRouting( { sessionId: parentSessionId, ...input }, accounts, @@ -1461,9 +1499,19 @@ export async function CodexAuthPlugin( setCacheKeepSubagents: (enabled) => { cacheKeepSubagents = enabled }, + setCacheKeepSustain: (enabled) => { + cacheKeepSustain = enabled + }, setCacheKeepWindow: (window) => { cacheKeepWindow = window }, + clearStickyRouting: (sessionId) => + clearSidebarStickyAssignment(sessionId, boundSidebarFile), + getStickyRouting: async (sessionId) => + resolveSessionStickyAccount( + await getSidebarState(boundSidebarFile), + sessionId, + ), refreshSidebar: async () => { const store = await loadAccounts(getConfigPath()) await writeMachineSidebarState(quotaManager, store) @@ -1500,11 +1548,15 @@ export async function CodexAuthPlugin( dir: getRpcDir(input.directory), drain: drainNotifications, apply: async (request: ApplyRequest): Promise => { + const callCtx: CommandContext = { + // biome-ignore lint/style/noNonNullAssertion: cmdCtx is set in the loader before RPC server starts, and command.execute.before has a null guard + ...cmdCtx!, + sessionId: request.sessionId, + } const payload = await buildDialogPayload( request.command, request.arguments, - // biome-ignore lint/style/noNonNullAssertion: cmdCtx is set in the loader before RPC server starts, and command.execute.before has a null guard - cmdCtx!, + callCtx, ) return { text: payload.text, knobs: payload.knobs } }, @@ -1606,6 +1658,7 @@ export async function CodexAuthPlugin( logT.debug('WS transport', { pid: process.pid, pathname: parsed.pathname, + accountId: keepwarmAccountKey, }) if (keepwarmCapture) { cacheKeepManager.track( @@ -1639,6 +1692,7 @@ export async function CodexAuthPlugin( logT.debug('HTTP transport', { pid: process.pid, pathname: parsed.pathname, + accountId: keepwarmAccountKey, }) try { const response = await fetch(url, finalInit) @@ -1647,6 +1701,7 @@ export async function CodexAuthPlugin( transport: 'http', phase: 'http', bodyText: finalInit.body, + accountId: keepwarmAccountKey, url: url.toString(), method: finalInit.method, headers: finalInit.headers, @@ -1659,6 +1714,7 @@ export async function CodexAuthPlugin( transport: 'http', phase: 'http', bodyText: finalInit.body, + accountId: keepwarmAccountKey, url: url.toString(), method: finalInit.method, headers: finalInit.headers, @@ -1722,26 +1778,38 @@ export async function CodexAuthPlugin( resetAtMs: number } - // Freshness key for a quota snapshot: each window's own checkedAt, - // then the snapshot-level checkedAt, then the cache entry's checkedAt. - // Both primary and secondary windows are consulted — an account whose - // only fresh window is the secondary must not be judged on the older - // primary timestamp. - function quotaCheckedAt( + function freshestQuotaSnapshot( quota: AccountQuota | null | undefined, - entryCheckedAt?: number, - ): number | undefined { - for (const checkedAt of [ - quota?.primary?.checkedAt, - quota?.secondary?.checkedAt, - quota?.checkedAt, - entryCheckedAt, - ]) { - if (typeof checkedAt === 'number' && Number.isFinite(checkedAt)) { - return checkedAt - } + entryCheckedAt: number | undefined, + fileQuota: AccountQuota | null | undefined, + fileAccountId?: string, + currentAccountId?: string, + ): { + quota: AccountQuota | null | undefined + quotaCheckedAt: number | undefined + source: 'memory' | 'file' + } { + // snapshotCheckedAt consults primary AND secondary, so the freshness + // comparison is true both-window aware (PR #57 fix #2). A file row is + // trusted only when its identity matches the live caller's — an + // unstamped file with a known live identity is treated as absent + // (PR #57 fix #1: the re-login bug), and a stamped file whose + // identity differs is treated as absent for the same reason. Both + // sides unknown fails open (both undefined compare equal). + const memoryCheckedAt = snapshotCheckedAt(quota, entryCheckedAt) + const fileCheckedAt = snapshotCheckedAt(fileQuota) + const useFile = + fileAccountId === currentAccountId && + fileQuota != null && + (quota === undefined || + (fileCheckedAt !== undefined && + (memoryCheckedAt === undefined || + fileCheckedAt > memoryCheckedAt))) + return { + quota: useFile ? fileQuota : quota, + quotaCheckedAt: useFile ? fileCheckedAt : memoryCheckedAt, + source: useFile ? 'file' : 'memory', } - return undefined } // Selects the fresher quota source and judges it. The file wins only @@ -1760,20 +1828,15 @@ export async function CodexAuthPlugin( currentAccountId?: string, ): AdmissionQuotaDecision { const memoryQuota = memoryEntry?.quota as AccountQuota | undefined - const memoryCheckedAt = quotaCheckedAt( + const freshest = freshestQuotaSnapshot( memoryQuota, memoryEntry?.checkedAt, + fileQuota, + fileAccountId, + currentAccountId, ) - const fileCheckedAt = quotaCheckedAt(fileQuota) - const useFile = - fileAccountId === currentAccountId && - fileQuota != null && - (memoryQuota === undefined || - (fileCheckedAt !== undefined && - (memoryCheckedAt === undefined || - fileCheckedAt > memoryCheckedAt))) - const source = useFile ? 'file' : 'memory' - const quota = useFile ? fileQuota : memoryQuota + const source = freshest.source + const quota = freshest.quota if (!isQuotaExhausted(quota, now)) return { exhausted: false } const reset = exhaustedQuotaResetAt(quota, now) @@ -1887,6 +1950,285 @@ export async function CodexAuthPlugin( }> } + type StickyRouteCandidate = { + accountId: string + wireAccountId?: string + access: string + keepwarmAccountKey: string + fallback?: FallbackAccount + quota: AccountQuota | null | undefined + quotaCheckedAt?: number + reservePercent: { primary: number; secondary: number } + configuredOrder: number + resetCreditsApplicable?: number + // Killswitch gate resolved at roster build, using the non-invalidating + // policy peek so a routine token refresh does not flip a killed + // account to "unknown". `false` excludes the candidate from both + // weighted placement and the mode-fallback fail-open branch. + killswitchPasses?: boolean + } + + function resetCreditsApplicable(value: unknown): number | undefined { + // Field key fix: the storage shape is `resetCreditsAvailable` on + // both OAuthQuotaSnapshot (core/accounts.ts:88) and AccountQuota + // (sidebar-state.ts:13). The previous read of + // `resetCreditsApplicable` was always undefined and the + // credit-priority sort in selectStickyCandidate never fired in + // production. The candidate field stays named + // `resetCreditsApplicable` so the sort comparator downstream is + // unchanged. + const credits = (value as { resetCreditsAvailable?: unknown } | null) + ?.resetCreditsAvailable + return typeof credits === 'number' && Number.isFinite(credits) + ? credits + : undefined + } + + async function buildStickyRouteRoster(input: { + storage: Awaited> + sidebarState: SidebarState + primaryAccess: string + mainAccountIdentity?: string + }): Promise { + const killswitchEnabled = isKillswitchEnabled(input.storage) + const killswitchNow = Date.now() + const mainMemory = quotaManager.peekMainForPolicy( + input.mainAccountIdentity, + ) + const mainFreshest = freshestQuotaSnapshot( + mainMemory?.quota as AccountQuota | undefined, + mainMemory?.checkedAt, + input.sidebarState.main.quota, + input.sidebarState.main.mainAccountId, + input.mainAccountIdentity, + ) + // Killswitch (opt-in): pre-resolve the gate for each candidate so the + // placement selector (weighted + mode-fallback) and the break decision + // can both honour it without redoing the read. Undefined = passes — + // the dominant path with killswitch disabled is byte-identical. + const mainKillswitchPasses = killswitchEnabled + ? killswitchPassesPolicy( + mainMemory?.quota, + input.storage, + undefined, + killswitchNow, + ) + : undefined + const roster: StickyRouteCandidate[] = [ + { + accountId: 'main', + wireAccountId: input.mainAccountIdentity, + access: input.primaryAccess, + keepwarmAccountKey: 'main', + quota: mainFreshest.quota, + quotaCheckedAt: mainFreshest.quotaCheckedAt, + reservePercent: getKillswitchThresholdsForAccount(input.storage), + configuredOrder: 0, + resetCreditsApplicable: resetCreditsApplicable( + mainFreshest.quota, + ), + killswitchPasses: mainKillswitchPasses, + }, + ] + const usableFallbacks = + await fallbackManager.getUsableFallbackAccounts(input.storage) + for (const fallback of usableFallbacks) { + if (!fallback.access) continue + const fileEntry = input.sidebarState.fallbacks.find( + (account) => account.id === fallback.id, + ) + const memoryEntry = quotaManager.peekFallbackForPolicy( + fallback.id, + fallback.accountId, + ) + const freshest = freshestQuotaSnapshot( + memoryEntry?.quota as AccountQuota | undefined, + memoryEntry?.checkedAt, + fileEntry?.quota, + fileEntry?.accountId, + fallback.accountId, + ) + const fallbackKillswitchPasses = killswitchEnabled + ? killswitchPassesPolicy( + memoryEntry?.quota, + input.storage, + fallback.id, + killswitchNow, + ) + : undefined + roster.push({ + accountId: fallback.id, + wireAccountId: fallback.accountId, + access: fallback.access, + keepwarmAccountKey: fallback.id, + fallback, + quota: freshest.quota, + quotaCheckedAt: freshest.quotaCheckedAt, + reservePercent: getKillswitchThresholdsForAccount( + input.storage, + fallback.id, + ), + configuredOrder: roster.length, + resetCreditsApplicable: resetCreditsApplicable(freshest.quota), + killswitchPasses: fallbackKillswitchPasses, + }) + } + return roster + } + + function stickyBreakDecision( + candidate: StickyRouteCandidate, + sidebarState: SidebarState, + status: number | undefined, + now: number, + storage: Awaited> | null, + ): StickyBreakDecision { + // Pre-resolved by the roster builder using the non-invalidating + // policy peek. Pass it through so a retained pin whose account has + // fallen below floor since the pin was created migrates the same + // way an exhausted pin does. Undefined = passes (killswitch + // disabled or no quota seen). + const killswitchPasses = isKillswitchEnabled(storage) + ? candidate.killswitchPasses + : undefined + if (candidate.accountId === 'main') { + const memoryEntry = quotaManager.peekMainForPolicy( + candidate.wireAccountId, + ) + const freshest = freshestQuotaSnapshot( + memoryEntry?.quota as AccountQuota | undefined, + memoryEntry?.checkedAt, + sidebarState.main.quota, + sidebarState.main.mainAccountId, + candidate.wireAccountId, + ) + return decideStickyBreak({ + quota: freshest.quota, + quotaCheckedAt: freshest.quotaCheckedAt, + status, + now, + killswitchPasses, + }) + } + const fileEntry = sidebarState.fallbacks.find( + (account) => account.id === candidate.accountId, + ) + const memoryEntry = quotaManager.peekFallbackForPolicy( + candidate.accountId, + candidate.wireAccountId, + ) + const freshest = freshestQuotaSnapshot( + memoryEntry?.quota as AccountQuota | undefined, + memoryEntry?.checkedAt, + fileEntry?.quota, + fileEntry?.accountId, + candidate.wireAccountId, + ) + return decideStickyBreak({ + quota: freshest.quota, + quotaCheckedAt: freshest.quotaCheckedAt, + status, + now, + killswitchPasses, + }) + } + + function stickyRateLimitKey(candidate: StickyRouteCandidate): string { + return candidate.accountId === 'main' + ? 'main' + : candidate.keepwarmAccountKey + } + + function isStickyRouteCandidateRateLimited( + candidate: StickyRouteCandidate, + ): boolean { + return quotaManager.isRateLimited(stickyRateLimitKey(candidate)) + } + + async function resolveStickyRouteCandidate(input: { + sessionId: string + requestBytes: number + candidates: readonly StickyRouteCandidate[] + excludeAccountIds?: readonly string[] + now: number + }): Promise { + const eligibleCandidates = input.candidates.filter( + (candidate) => !isStickyRouteCandidateRateLimited(candidate), + ) + const candidatesById = new Map( + eligibleCandidates.map((candidate) => [ + candidate.accountId, + candidate, + ]), + ) + const quotaCheckedAtByAccount = Object.fromEntries( + eligibleCandidates.map((candidate) => [ + candidate.accountId, + candidate.quotaCheckedAt, + ]), + ) + const excluded = new Set(input.excludeAccountIds) + let placement: + | { + accountId: string + source: 'weighted' | 'mode-fallback' + pendingBytes: number + } + | undefined + const assignment = await resolveSidebarStickyAssignment( + { + sessionId: input.sessionId, + requestBytes: input.requestBytes, + now: input.now, + validPinnedAccountIds: [...candidatesById.keys()], + excludeAccountIds: input.excludeAccountIds, + quotaCheckedAtByAccount, + choose: (pendingBytes) => { + const eligible = eligibleCandidates.filter( + (candidate) => !excluded.has(candidate.accountId), + ) + if (eligible.length === 0) return undefined + const selected = selectStickyCandidate({ + candidates: eligible, + pendingBytes, + requestBytes: input.requestBytes, + now: input.now, + onEmptyWeightedSet: () => { + logA.debug( + 'sticky routing: no fresh weighted candidates; using configured order', + ) + }, + }) + // Every candidate killed by the killswitch filter. The caller + // will translate the placed-pin absence into the shared + // `killswitchBlockedResponse`, the same shape the ordered + // modes produce. + if (!selected) return undefined + placement = { + accountId: selected.accountId, + source: selected.source, + pendingBytes: pendingBytes.get(selected.accountId) ?? 0, + } + return selected + }, + }, + boundSidebarFile, + ) + if (placement && assignment?.accountId === placement.accountId) { + logA.debug('sticky routing: placed session pin', { + pid: process.pid, + sessionHash: hashSidebarSessionId(input.sessionId), + accountId: placement.accountId, + source: placement.source, + requestBytes: input.requestBytes, + pendingBytes: placement.pendingBytes, + }) + } + return assignment + ? candidatesById.get(assignment.accountId) + : undefined + } + async function usableFallbackCandidates( fallbackStorage: Awaited>, sidebarState: SidebarState, @@ -2193,9 +2535,8 @@ export async function CodexAuthPlugin( const sidebarSessionId = resolveSidebarSessionId(requestHeaders) const sidebarParentSessionId = requestHeaders.get('x-parent-session-id')?.trim() || undefined - // Routing is purely mode-driven. The primary is ALWAYS the main - // account; fallback-first is handled by a proactive gate below that - // tries usable fallbacks before main. There is no per-account pin. + // Main-first and fallback-first select per request; sticky-balanced + // resolves a per-session pin from sidebar state before sending. const reqStorage = await loadRequestAccounts() // Main primary uses opencode's auth slot. @@ -2261,6 +2602,151 @@ export async function CodexAuthPlugin( return requestSidebarStatePromise } const sidebarState = await requestSidebarState() + + if ( + mode === 'sticky-balanced' && + sidebarSessionId && + isReplayableRequest(requestInput, init) && + typeof init?.body === 'string' + ) { + const requestBytes = Buffer.byteLength(init.body, 'utf8') + const stickyRoster = await buildStickyRouteRoster({ + storage: reqStorage, + sidebarState, + primaryAccess, + mainAccountIdentity, + }) + let stickyCandidate = await resolveStickyRouteCandidate({ + sessionId: sidebarSessionId, + requestBytes, + candidates: stickyRoster, + now: Date.now(), + }) + + if (stickyCandidate) { + const preSendBreak = isStickyRouteCandidateRateLimited( + stickyCandidate, + ) + ? { action: 'migrate' as const, reason: 'exhausted' as const } + : stickyBreakDecision( + stickyCandidate, + sidebarState, + undefined, + Date.now(), + reqStorage, + ) + if (preSendBreak.action === 'migrate') { + const replacement = await resolveStickyRouteCandidate({ + sessionId: sidebarSessionId, + requestBytes, + candidates: stickyRoster, + excludeAccountIds: [stickyCandidate.accountId], + now: Date.now(), + }) + if (replacement) { + logA.debug('sticky routing: migrated session pin', { + pid: process.pid, + sessionHash: hashSidebarSessionId(sidebarSessionId), + fromAccountId: stickyCandidate.accountId, + toAccountId: replacement.accountId, + reason: preSendBreak.reason, + }) + stickyCandidate = replacement + } + } + + let stickyResponse = await sendWithAccessToken( + requestInput, + init, + stickyCandidate.access, + stickyCandidate.wireAccountId, + stickyCandidate.keepwarmAccountKey, + ) + + const pushStickyQuota = async ( + response: Response, + candidate: StickyRouteCandidate, + ) => { + try { + const snapshot = normalizeQuotaHeaders(response.headers) + await pushQuota( + snapshot as Record, + candidate.access, + candidate.accountId === 'main' + ? undefined + : candidate.accountId, + candidate.accountId === 'main' + ? candidate.wireAccountId + : undefined, + isCompleteQuotaHeaderFrame(response.headers), + ) + } catch { + // Quota push is advisory; preserve the provider response. + } + } + + await pushStickyQuota(stickyResponse, stickyCandidate) + const responseBreak = stickyBreakDecision( + stickyCandidate, + sidebarState, + stickyResponse.status, + Date.now(), + reqStorage, + ) + const retryableStickyFailure = + stickyResponse.status === 401 || + stickyResponse.status === 403 || + stickyResponse.status === 429 + if ( + retryableStickyFailure && + responseBreak.action === 'migrate' + ) { + const replacement = await resolveStickyRouteCandidate({ + sessionId: sidebarSessionId, + requestBytes, + candidates: stickyRoster, + excludeAccountIds: [stickyCandidate.accountId], + now: Date.now(), + }) + if (replacement) { + logA.debug('sticky routing: migrated session pin', { + pid: process.pid, + sessionHash: hashSidebarSessionId(sidebarSessionId), + fromAccountId: stickyCandidate.accountId, + toAccountId: replacement.accountId, + reason: responseBreak.reason, + }) + const previousResponse = stickyResponse + stickyResponse = await sendWithAccessToken( + requestInput, + init, + replacement.access, + replacement.wireAccountId, + replacement.keepwarmAccountKey, + ) + previousResponse.body?.cancel().catch(() => {}) + stickyCandidate = replacement + await pushStickyQuota(stickyResponse, stickyCandidate) + } + } + + if ( + stickyCandidate.fallback && + !shouldFallbackStatus(stickyResponse.status, reqStorage) + ) { + await fallbackManager.markUsed(stickyCandidate.fallback) + } + await writeRequestSidebarRouting( + sidebarSessionId, + sidebarParentSessionId, + stickyCandidate.accountId, + mode, + reqStorage?.accounts, + ).catch(() => {}) + return stickyResponse + } + } + const mainQuotaDecision = admissionQuotaDecision( quotaManager.peekMainForPolicy(mainAccountIdentity), sidebarState.main.quota, @@ -2468,7 +2954,7 @@ export async function CodexAuthPlugin( sidebarParentSessionId, servedActiveId, mode, - reqStorage?.accounts ?? [], + reqStorage?.accounts, ).catch(() => {}) return finalResponse }, @@ -2596,7 +3082,7 @@ export async function CodexAuthPlugin( [OPENAI_ROUTING_COMMAND_NAME]: { template: OPENAI_ROUTING_COMMAND_NAME, description: - 'Show or change OpenAI account routing between main-first and fallback-first.', + 'Show or change OpenAI account routing between main-first, fallback-first, and sticky-balanced.', }, [OPENAI_KILLSWITCH_COMMAND_NAME]: { template: OPENAI_KILLSWITCH_COMMAND_NAME, diff --git a/packages/opencode/src/rpc/protocol.ts b/packages/opencode/src/rpc/protocol.ts index 9ae19df..c78d4a0 100644 --- a/packages/opencode/src/rpc/protocol.ts +++ b/packages/opencode/src/rpc/protocol.ts @@ -23,6 +23,7 @@ export interface RpcNotification { export interface ApplyRequest { command: CommandModalName arguments: string + sessionId?: string } export interface ApplyResult { diff --git a/packages/opencode/src/sidebar-state.ts b/packages/opencode/src/sidebar-state.ts index b35bf92..8d98e41 100644 --- a/packages/opencode/src/sidebar-state.ts +++ b/packages/opencode/src/sidebar-state.ts @@ -98,6 +98,33 @@ export interface ActiveRoutingEntry { export type ActiveRoutingMap = Record +export interface StickyAssignment { + accountId: string + assignedAt: number + lastSeenAt: number + inputBytes: number + quotaCheckedAt?: number +} + +export type StickyAssignmentMap = Record + +export interface StickyAssignmentChoice { + accountId: string + quotaCheckedAt?: number +} + +export interface ResolveStickyAssignmentInput { + sessionId: string + requestBytes: number + now: number + validPinnedAccountIds: readonly string[] + excludeAccountIds?: readonly string[] + quotaCheckedAtByAccount: Readonly> + choose: ( + pendingBytes: ReadonlyMap, + ) => StickyAssignmentChoice | undefined +} + export interface SidebarState { main: { quota: AccountQuota | null @@ -116,13 +143,14 @@ export interface SidebarState { /** Machine-global routing mode and compatibility value for older readers. */ route: string activeRouting?: ActiveRoutingMap + stickyAssignments?: StickyAssignmentMap planType?: string credits?: number lastUpdated: number } -import { randomUUID } from 'node:crypto' -import { mkdir, readFile, rename, rm, writeFile } from 'node:fs/promises' +import { createHash, randomUUID } from 'node:crypto' +import { chmod, mkdir, readFile, rename, rm, writeFile } from 'node:fs/promises' import { tmpdir } from 'node:os' import { dirname, join } from 'node:path' import { acquireRefreshFileLock } from './core/refresh-file-lock' @@ -133,6 +161,14 @@ const logSb = createLogger('sidebar') const STATE_FILE_ENV = 'OPENCODE_OPENAI_AUTH_SIDEBAR_STATE_FILE' const DEFAULT_STATE_DIR = join(tmpdir(), 'opencode-openai-auth') const DEFAULT_STATE_FILE = join(DEFAULT_STATE_DIR, 'sidebar-state.json') +const SESSION_HASH_PATTERN = /^[a-f0-9]{64}$/ +export const STICKY_ASSIGNMENT_MAX_AGE_MS = 7 * 24 * 60 * 60 * 1000 +export const STICKY_ASSIGNMENT_MAX_ENTRIES = 256 +const STICKY_ASSIGNMENT_LAST_SEEN_TOUCH_MS = 60 * 60 * 1000 + +export function hashSidebarSessionId(sessionId: string): string { + return createHash('sha256').update(sessionId).digest('hex') +} function normalizeResetCredits(value: unknown): number | undefined { return typeof value === 'number' && Number.isFinite(value) && value >= 0 @@ -178,6 +214,60 @@ function normalizeActiveRouting(value: unknown): ActiveRoutingMap | undefined { return Object.keys(normalized).length > 0 ? normalized : undefined } +function normalizeStickyAssignments( + value: unknown, +): StickyAssignmentMap | undefined { + if (value === null || typeof value !== 'object' || Array.isArray(value)) { + return undefined + } + + const normalized: StickyAssignmentMap = {} + for (const [sessionHash, rawAssignment] of Object.entries(value)) { + if ( + !SESSION_HASH_PATTERN.test(sessionHash) || + rawAssignment === null || + typeof rawAssignment !== 'object' || + Array.isArray(rawAssignment) + ) { + continue + } + const assignment = rawAssignment as Record + if ( + typeof assignment.accountId !== 'string' || + assignment.accountId.length === 0 || + typeof assignment.assignedAt !== 'number' || + !Number.isFinite(assignment.assignedAt) || + assignment.assignedAt < 0 || + typeof assignment.lastSeenAt !== 'number' || + !Number.isFinite(assignment.lastSeenAt) || + assignment.lastSeenAt < 0 || + typeof assignment.inputBytes !== 'number' || + !Number.isFinite(assignment.inputBytes) || + assignment.inputBytes < 0 + ) { + continue + } + const quotaCheckedAt = assignment.quotaCheckedAt + if ( + quotaCheckedAt !== undefined && + (typeof quotaCheckedAt !== 'number' || + !Number.isFinite(quotaCheckedAt) || + quotaCheckedAt < 0) + ) { + continue + } + normalized[sessionHash] = { + accountId: assignment.accountId, + assignedAt: assignment.assignedAt, + lastSeenAt: assignment.lastSeenAt, + inputBytes: assignment.inputBytes, + ...(quotaCheckedAt === undefined ? {} : { quotaCheckedAt }), + } + } + + return Object.keys(normalized).length > 0 ? normalized : undefined +} + export function getSidebarStateFile(): string { return process.env[STATE_FILE_ENV] || DEFAULT_STATE_FILE } @@ -280,6 +370,7 @@ export function normalizeSidebarState(raw: unknown): SidebarState { const planType = typeof r.planType === 'string' ? r.planType : undefined const credits = typeof r.credits === 'number' ? r.credits : undefined const activeRouting = normalizeActiveRouting(r.activeRouting) + const stickyAssignments = normalizeStickyAssignments(r.stickyAssignments) return { main, fallbacks, @@ -287,6 +378,7 @@ export function normalizeSidebarState(raw: unknown): SidebarState { route, lastUpdated, ...(activeRouting !== undefined ? { activeRouting } : {}), + ...(stickyAssignments !== undefined ? { stickyAssignments } : {}), ...(planType !== undefined ? { planType } : {}), ...(credits !== undefined ? { credits } : {}), } @@ -314,12 +406,13 @@ export type SidebarRoutingAccount = { export function isUsableRoutingEntry( entry: ActiveRoutingEntry, - accounts: readonly SidebarRoutingAccount[], + accounts: readonly SidebarRoutingAccount[] | undefined, now = Date.now(), ): boolean { const fresh = entry.updatedAt >= now - ACTIVE_ROUTING_MAX_AGE_MS && entry.updatedAt <= now if (!fresh) return false + if (accounts === undefined) return true return ( entry.activeId === 'main' || accounts.some( @@ -367,6 +460,38 @@ export function isQuotaExhausted( return exhaustedQuotaResetAt(quota, now) !== undefined } +export function resolveSessionStickyAccount( + state: SidebarState, + sessionId: string | undefined, + now = Date.now(), +): string | undefined { + if (!sessionId || state.route !== 'sticky-balanced') return undefined + const assignment = state.stickyAssignments?.[hashSidebarSessionId(sessionId)] + if ( + !assignment || + assignment.lastSeenAt < now - STICKY_ASSIGNMENT_MAX_AGE_MS + ) { + return undefined + } + if (assignment.accountId === 'main') { + return state.main?.killed || isQuotaExhausted(state.main?.quota, now) + ? undefined + : 'main' + } + const fallback = state.fallbacks?.find( + (account) => account.id === assignment.accountId, + ) + if ( + !fallback || + fallback.enabled === false || + fallback.killed === true || + isQuotaExhausted(fallback.quota, now) + ) { + return undefined + } + return fallback.id +} + export function resolveSessionSidebarRouting( state: SidebarState, sessionId?: string, @@ -388,6 +513,11 @@ export function resolveSessionSidebarRouting( return { activeId: own.activeId, route: own.route } } + const stickyAccountId = resolveSessionStickyAccount(state, sessionId, now) + if (stickyAccountId) { + return { activeId: stickyAccountId, route: state.route } + } + const enabledFallbacks = state.fallbacks.filter( (account) => account.enabled && !account.killed, ) @@ -403,7 +533,7 @@ export function resolveSessionSidebarRouting( export function pruneActiveRouting( activeRouting: ActiveRoutingMap | undefined, - accounts: readonly SidebarRoutingAccount[], + accounts: readonly SidebarRoutingAccount[] | undefined, now = Date.now(), removedSessionId?: string, ): ActiveRoutingMap | undefined { @@ -422,6 +552,175 @@ export function pruneActiveRouting( return bounded.length > 0 ? Object.fromEntries(bounded) : undefined } +export function pruneStickyAssignments( + assignments: StickyAssignmentMap | undefined, + validAccountIds: ReadonlySet | undefined, + now = Date.now(), + removedSessionHash?: string, +): StickyAssignmentMap | undefined { + if (!assignments) return undefined + let accountNotInRoster = 0 + let expired = 0 + let explicitRemoval = 0 + const kept: [string, StickyAssignment][] = [] + for (const [sessionHash, assignment] of Object.entries(assignments)) { + if (sessionHash === removedSessionHash) { + explicitRemoval += 1 + continue + } + if (assignment.lastSeenAt < now - STICKY_ASSIGNMENT_MAX_AGE_MS) { + expired += 1 + continue + } + if ( + validAccountIds !== undefined && + !validAccountIds.has(assignment.accountId) + ) { + accountNotInRoster += 1 + continue + } + kept.push([sessionHash, assignment]) + } + const removed = accountNotInRoster + expired + explicitRemoval + if (removed > 0) { + logSb.debug('pruned sticky assignments', { + pid: process.pid, + removed, + reasons: { + 'account-not-in-roster': accountNotInRoster, + expired, + 'explicit-removal': explicitRemoval, + }, + }) + } + return kept.length > 0 ? Object.fromEntries(kept) : undefined +} + +function limitStickyAssignments( + assignments: StickyAssignmentMap, + protectedSessionHash: string, +): StickyAssignmentMap { + const overflow = + Object.keys(assignments).length - STICKY_ASSIGNMENT_MAX_ENTRIES + if (overflow <= 0) return assignments + const evicted = new Set( + Object.entries(assignments) + .filter(([sessionHash]) => sessionHash !== protectedSessionHash) + .sort( + ([leftHash, left], [rightHash, right]) => + left.lastSeenAt - right.lastSeenAt || + leftHash.localeCompare(rightHash), + ) + .slice(0, overflow) + .map(([sessionHash]) => sessionHash), + ) + return Object.fromEntries( + Object.entries(assignments).filter( + ([sessionHash]) => !evicted.has(sessionHash), + ), + ) +} + +function stickyAssignmentsEqual( + left: StickyAssignmentMap | undefined, + right: StickyAssignmentMap | undefined, +): boolean { + if (left === right) return true + if (!left || !right) return false + const leftEntries = Object.entries(left) + if (leftEntries.length !== Object.keys(right).length) return false + return leftEntries.every(([sessionHash, assignment]) => + Object.is(right[sessionHash], assignment), + ) +} + +function isValidStickyAssignment( + assignment: StickyAssignment | undefined, + validPinnedAccountIds: ReadonlySet, + excludedAccountIds: ReadonlySet, + now: number, +): assignment is StickyAssignment { + return ( + assignment !== undefined && + validPinnedAccountIds.has(assignment.accountId) && + !excludedAccountIds.has(assignment.accountId) && + assignment.lastSeenAt >= now - STICKY_ASSIGNMENT_MAX_AGE_MS + ) +} + +function stickyAssignmentNeedsMetadataUpdate( + assignment: StickyAssignment, + requestBytes: number, + now: number, +): boolean { + return ( + requestBytes > assignment.inputBytes || + now - assignment.lastSeenAt >= STICKY_ASSIGNMENT_LAST_SEEN_TOUCH_MS + ) +} + +function readonlyPendingBytes( + pendingBytes: Map, +): ReadonlyMap { + const snapshot = new Map(pendingBytes) + const view: ReadonlyMap = { + get size() { + return snapshot.size + }, + has: (key: string) => snapshot.has(key), + get: (key: string) => snapshot.get(key), + entries: () => snapshot.entries(), + keys: () => snapshot.keys(), + values: () => snapshot.values(), + forEach: ( + callback: ( + value: number, + key: string, + map: ReadonlyMap, + ) => void, + thisArg?: unknown, + ) => { + snapshot.forEach((value, key) => { + callback.call(thisArg, value, key, view) + }) + }, + [Symbol.iterator]: () => snapshot[Symbol.iterator](), + } + return Object.freeze(view) +} + +function pendingBytesForAssignments( + assignments: StickyAssignmentMap | undefined, + quotaCheckedAtByAccount: Readonly>, +): ReadonlyMap { + const pendingBytes = new Map() + for (const assignment of Object.values(assignments ?? {})) { + if ( + assignment.quotaCheckedAt !== + quotaCheckedAtByAccount[assignment.accountId] + ) { + continue + } + pendingBytes.set( + assignment.accountId, + (pendingBytes.get(assignment.accountId) ?? 0) + assignment.inputBytes, + ) + } + return readonlyPendingBytes(pendingBytes) +} + +function usableRoutingAccountIds( + accounts: readonly SidebarRoutingAccount[] | undefined, +): ReadonlySet | undefined { + if (accounts === undefined) return undefined + return new Set([ + 'main', + ...accounts + .filter((account) => account.enabled !== false && account.killed !== true) + .map((account) => account.id), + ]) +} + // Serialization chain: concurrent calls are queued so a stale background // write cannot land after a newer one and corrupt the file. let sidebarWriteChain: Promise = Promise.resolve() @@ -484,7 +783,7 @@ function parseSidebarState(raw: string): SidebarState { } async function acquireSidebarWriteLock(file: string) { - await mkdir(dirname(file), { recursive: true }) + await ensureSidebarStateDirectory(file) const deadline = Date.now() + SIDEBAR_WRITE_LOCK_WAIT_MS while (Date.now() <= deadline) { const lock = await acquireRefreshFileLock({ @@ -499,9 +798,21 @@ async function acquireSidebarWriteLock(file: string) { throw new Error('Timed out waiting for the sidebar state write lock') } +async function ensureSidebarStateDirectory(file: string): Promise { + const dir = dirname(file) + await mkdir(dir, { recursive: true, mode: 0o700 }) + if (file !== DEFAULT_STATE_FILE) return + await chmod(dir, 0o700).catch((error) => { + logSb.warn('sidebar directory permission remediation failed', { + pid: process.pid, + error: error instanceof Error ? error.message : String(error), + }) + }) +} + async function writeMergedSidebarState( file: string, - merge: (latest: SidebarState) => SidebarState, + merge: (latest: SidebarState) => SidebarState | undefined, hooks?: SidebarMergeHooks, ): Promise { const lock = await acquireSidebarWriteLock(file) @@ -511,6 +822,7 @@ async function writeMergedSidebarState( for (let attempt = 0; attempt < MAX_MERGE_ATTEMPTS; attempt += 1) { const rawBefore = await readRawSidebar(file) const next = merge(parseSidebarState(rawBefore)) + if (!next) return if (attempt === 0) await hooks?.beforeRecheck?.() const rawRecheck = await readRawSidebar(file) if (rawRecheck !== rawBefore) continue @@ -519,7 +831,8 @@ async function writeMergedSidebarState( } const latest = await readSidebarState(file) - await doWriteSidebarState(merge(latest), file) + const next = merge(latest) + if (next) await doWriteSidebarState(next, file) } finally { await lock.release() } @@ -570,6 +883,12 @@ export function setSidebarMachineState( machineState.main.quota, latest.main.quota, ) + const now = Date.now() + const stickyAssignments = pruneStickyAssignments( + latest.stickyAssignments, + usableRoutingAccountIds(machineState.fallbacks), + now, + ) return { ...latest, ...machineState, @@ -602,7 +921,8 @@ export function setSidebarMachineState( }), activeId: latest.activeId, activeRouting: latest.activeRouting, - lastUpdated: Math.max(Date.now(), latest.lastUpdated + 1), + stickyAssignments, + lastUpdated: Math.max(now, latest.lastUpdated + 1), } }, hooks, @@ -612,7 +932,7 @@ export function setSidebarMachineState( export function upsertSidebarActiveRouting( input: { sessionId: string } & ActiveRoutingEntry, - accounts: readonly SidebarRoutingAccount[], + accounts: readonly SidebarRoutingAccount[] | undefined, file = getSidebarStateFile(), hooks?: SidebarMergeHooks, ): Promise { @@ -632,11 +952,17 @@ export function upsertSidebarActiveRouting( accounts, Date.now(), ) + const stickyAssignments = pruneStickyAssignments( + latest.stickyAssignments, + usableRoutingAccountIds(accounts), + Date.now(), + ) return { ...latest, activeId: input.activeId, route: input.route, activeRouting, + stickyAssignments, lastUpdated: Math.max(Date.now(), latest.lastUpdated + 1), } }, @@ -661,7 +987,7 @@ export function setSidebarLegacyRouting( export function removeSidebarActiveRouting( sessionId: string, - accounts: readonly SidebarRoutingAccount[], + accounts: readonly SidebarRoutingAccount[] | undefined, file = getSidebarStateFile(), hooks?: SidebarMergeHooks, ): Promise { @@ -676,9 +1002,16 @@ export function removeSidebarActiveRouting( now, sessionId, ) + const stickyAssignments = pruneStickyAssignments( + latest.stickyAssignments, + usableRoutingAccountIds(accounts), + now, + hashSidebarSessionId(sessionId), + ) return { ...latest, activeRouting, + stickyAssignments, lastUpdated: Math.max(Date.now(), latest.lastUpdated + 1), } }, @@ -687,14 +1020,164 @@ export function removeSidebarActiveRouting( }) } +export async function resolveSidebarStickyAssignment( + input: ResolveStickyAssignmentInput, + file = getSidebarStateFile(), + hooks?: SidebarMergeHooks, +): Promise { + const sessionHash = hashSidebarSessionId(input.sessionId) + const validPinnedAccountIds = new Set(input.validPinnedAccountIds) + const excludedAccountIds = new Set(input.excludeAccountIds) + const existing = (await readSidebarState(file)).stickyAssignments?.[ + sessionHash + ] + if ( + isValidStickyAssignment( + existing, + validPinnedAccountIds, + excludedAccountIds, + input.now, + ) && + !stickyAssignmentNeedsMetadataUpdate( + existing, + input.requestBytes, + input.now, + ) + ) { + return existing + } + + let resolved: StickyAssignment | undefined + await enqueueSidebarWrite(async () => { + await writeMergedSidebarState( + file, + (latest) => { + const stickyAssignments = pruneStickyAssignments( + latest.stickyAssignments, + validPinnedAccountIds, + input.now, + ) + const assignmentsPruned = !stickyAssignmentsEqual( + latest.stickyAssignments, + stickyAssignments, + ) + const current = stickyAssignments?.[sessionHash] + if ( + isValidStickyAssignment( + current, + validPinnedAccountIds, + excludedAccountIds, + input.now, + ) + ) { + const metadataNeedsUpdate = stickyAssignmentNeedsMetadataUpdate( + current, + input.requestBytes, + input.now, + ) + const assignment = metadataNeedsUpdate + ? { + ...current, + inputBytes: Math.max(current.inputBytes, input.requestBytes), + ...(input.now - current.lastSeenAt >= + STICKY_ASSIGNMENT_LAST_SEEN_TOUCH_MS + ? { lastSeenAt: input.now } + : {}), + } + : current + resolved = assignment + if (!assignmentsPruned && !metadataNeedsUpdate) return undefined + return { + ...latest, + stickyAssignments: { + ...stickyAssignments, + [sessionHash]: assignment, + }, + lastUpdated: Math.max(input.now, latest.lastUpdated + 1), + } + } + + const choice = input.choose( + pendingBytesForAssignments( + stickyAssignments, + input.quotaCheckedAtByAccount, + ), + ) + if (!choice) { + resolved = undefined + if (!assignmentsPruned) return undefined + return { + ...latest, + stickyAssignments, + lastUpdated: Math.max(input.now, latest.lastUpdated + 1), + } + } + + resolved = { + accountId: choice.accountId, + assignedAt: input.now, + lastSeenAt: input.now, + inputBytes: input.requestBytes, + ...(choice.quotaCheckedAt === undefined + ? {} + : { quotaCheckedAt: choice.quotaCheckedAt }), + } + return { + ...latest, + stickyAssignments: limitStickyAssignments( + { + ...stickyAssignments, + [sessionHash]: resolved, + }, + sessionHash, + ), + lastUpdated: Math.max(input.now, latest.lastUpdated + 1), + } + }, + hooks, + ) + }) + return resolved +} + +export async function clearSidebarStickyAssignment( + sessionId: string, + file = getSidebarStateFile(), +): Promise { + const sessionHash = hashSidebarSessionId(sessionId) + let removed = false + await enqueueSidebarWrite(async () => { + await writeMergedSidebarState(file, (latest) => { + const assignments = latest.stickyAssignments + if (assignments?.[sessionHash] === undefined) { + removed = false + return undefined + } + removed = true + const { [sessionHash]: _removed, ...remaining } = assignments + return { + ...latest, + ...(Object.keys(remaining).length > 0 + ? { stickyAssignments: remaining } + : { stickyAssignments: undefined }), + lastUpdated: Math.max(Date.now(), latest.lastUpdated + 1), + } + }) + }) + return removed +} + async function doWriteSidebarState( state: SidebarState, file: string, ): Promise { const tempPath = `${file}.${randomUUID()}.tmp` try { - await mkdir(dirname(file), { recursive: true }) - await writeFile(tempPath, JSON.stringify(state), 'utf8') + await ensureSidebarStateDirectory(file) + await writeFile(tempPath, JSON.stringify(normalizeSidebarState(state)), { + encoding: 'utf8', + mode: 0o600, + }) await rename(tempPath, file) } catch (e) { await rm(tempPath, { force: true }).catch(() => {}) diff --git a/packages/opencode/src/tests/accounts-store.test.ts b/packages/opencode/src/tests/accounts-store.test.ts index 9835cea..09de72c 100644 --- a/packages/opencode/src/tests/accounts-store.test.ts +++ b/packages/opencode/src/tests/accounts-store.test.ts @@ -134,6 +134,38 @@ describe('accounts store', () => { expect(state.accounts[account.id].access).toBe('acc-token') }) + it('round-trips sticky-balanced routing mode', async () => { + const { loadAccounts, saveAccounts } = await import('../core/accounts.ts') + + await saveAccounts( + { + version: 1, + main: { type: 'opencode', provider: 'openai' }, + routing: { mode: 'sticky-balanced' }, + accounts: [], + }, + cfgPath, + ) + + expect((await loadAccounts(cfgPath))?.routing?.mode).toBe('sticky-balanced') + }) + + it('round-trips cachekeep sustain', async () => { + const { loadAccounts, saveAccounts } = await import('../core/accounts.ts') + + await saveAccounts( + { + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [], + cachekeep: { enabled: true, sustain: true }, + }, + cfgPath, + ) + + expect((await loadAccounts(cfgPath))?.cachekeep?.sustain).toBe(true) + }) + it('state file has 0600 permissions', async () => { const { saveAccounts } = await import('../core/accounts.ts') const { statSync } = await import('node:fs') diff --git a/packages/opencode/src/tests/cachekeep.test.ts b/packages/opencode/src/tests/cachekeep.test.ts index 0b25078..f7e749f 100644 --- a/packages/opencode/src/tests/cachekeep.test.ts +++ b/packages/opencode/src/tests/cachekeep.test.ts @@ -391,6 +391,137 @@ describe('CacheKeepManager.track', () => { expect(status.targets[0]!.sessionKey).toBe('sess-2') }) + test('sustain toggles main idle pruning at runtime without recreating the manager', () => { + let sustain = false + const mgr = new CacheKeepManager({ + fetchImpl, + getMainToken, + refreshFallback, + codexResponsesUrl: CODEX_URL, + logger: log, + now: clock.now, + ttlMs: TTL_MS, + maxIdleWarmMs: 60_000, + getSustain: () => sustain, + }) + + mgr.track('pruned-while-off', JSON.stringify({ input: 'old' }), 'main') + clock.advance(60_001) + mgr.track('trigger-off', JSON.stringify({ input: 'new' }), 'main') + expect(mgr.status().targets.map((target) => target.sessionKey)).toEqual([ + 'trigger-off', + ]) + + sustain = true + mgr.track('kept-while-on', JSON.stringify({ input: 'kept' }), 'main') + clock.advance(60_001) + mgr.track('trigger-on', JSON.stringify({ input: 'newer' }), 'main') + expect(mgr.status().targets.map((target) => target.sessionKey)).toEqual([ + 'trigger-off', + 'kept-while-on', + 'trigger-on', + ]) + expect(mgr.status().sustain).toBe(true) + + sustain = false + clock.advance(60_001) + mgr.track('trigger-off-again', JSON.stringify({ input: 'latest' }), 'main') + expect(mgr.status().targets.map((target) => target.sessionKey)).toEqual([ + 'trigger-off-again', + ]) + expect(mgr.status().sustain).toBe(false) + }) + + test('sustain bypasses idle pruning but leaves maxTargets and maxBytes eviction active', async () => { + const body = JSON.stringify({ input: 'x'.repeat(100) }) + const mgr = new CacheKeepManager({ + fetchImpl, + getMainToken, + refreshFallback, + codexResponsesUrl: CODEX_URL, + logger: log, + now: clock.now, + ttlMs: TTL_MS, + maxIdleWarmMs: 1, + maxTargets: 8, + maxBytes: body.length * 2 - 1, + getSustain: () => true, + }) + + mgr.track('sustained-old', body, 'main') + clock.advance(2) + await mgr.tick() + expect(mgr.status().targets.map((target) => target.sessionKey)).toEqual([ + 'sustained-old', + ]) + + mgr.track('newer', body, 'main') + expect(mgr.status().targets.map((target) => target.sessionKey)).toEqual([ + 'newer', + ]) + + const capped = new CacheKeepManager({ + fetchImpl, + getMainToken, + refreshFallback, + codexResponsesUrl: CODEX_URL, + logger: log, + now: clock.now, + ttlMs: TTL_MS, + maxIdleWarmMs: 1, + maxTargets: 1, + getSustain: () => true, + }) + + capped.track('sustained-old', JSON.stringify({ input: 'old' }), 'main') + clock.advance(2) + await capped.tick() + expect(capped.status().targets.map((target) => target.sessionKey)).toEqual([ + 'sustained-old', + ]) + + capped.track('newer', JSON.stringify({ input: 'new' }), 'main') + expect(capped.status().targets.map((target) => target.sessionKey)).toEqual([ + 'newer', + ]) + }) + + test('sustain leaves the configured clock window in control of capture and warming', async () => { + let window: { startHour: number; endHour: number } | undefined + const mgr = new CacheKeepManager({ + fetchImpl, + getMainToken, + refreshFallback, + codexResponsesUrl: CODEX_URL, + logger: log, + now: clock.now, + ttlMs: TTL_MS, + leadMs: LEAD_MS, + getSustain: () => true, + getWindow: () => window, + }) + const outsideHour = new Date(clock.now()).getHours() + window = undefined + mgr.track( + 'captured-before-window', + JSON.stringify({ input: 'old' }), + 'main', + ) + window = { + startHour: (outsideHour + 1) % 24, + endHour: (outsideHour + 2) % 24, + } + + mgr.track('blocked-by-window', JSON.stringify({ input: 'new' }), 'main') + expect(mgr.status().targets.map((target) => target.sessionKey)).toEqual([ + 'captured-before-window', + ]) + + clock.advance(TTL_MS - LEAD_MS + 1) + await mgr.tick() + expect(fetchImpl).not.toHaveBeenCalled() + }) + test('caps Map size at default maxTargets (32)', () => { const mgr = new CacheKeepManager({ fetchImpl, @@ -451,7 +582,7 @@ describe('CacheKeepManager.track', () => { expect(status.targets[0]!.sessionKey).toBe('sess-1') }) - test('evicts least-recently-used target instead of oldest inserted target', async () => { + test('sustain leaves least-recently-used eviction active', async () => { const mgr = new CacheKeepManager({ fetchImpl, getMainToken, @@ -461,7 +592,9 @@ describe('CacheKeepManager.track', () => { now: clock.now, ttlMs: TTL_MS, leadMs: LEAD_MS, + maxIdleWarmMs: 1, maxTargets: 2, + getSustain: () => true, }) mgr.track( 'main', @@ -518,6 +651,7 @@ describe('CacheKeepManager subagent pruneStale', () => { now: clock.now, maxIdleWarmMs: 60 * 60 * 1000, // 1h main maxSubagentIdleMs: 30 * 60 * 1000, // 30min subagent + getSustain: () => true, }) mgr.track( 'sub-sess', @@ -1446,6 +1580,7 @@ describe('CacheKeepManager tick/prewarm', () => { ttlMs: longTtl, leadMs: LEAD_MS, maxSubagentIdleMs: 60 * 60 * 1000, // large so the idle prune doesn't fire first + getSustain: () => true, }) mgr.track( 'sub-56', @@ -1486,6 +1621,7 @@ describe('CacheKeepManager tick/prewarm', () => { ttlMs: longTtl, leadMs: LEAD_MS, maxSubagentIdleMs: 30 * 60 * 1000, // same as TTL — would prune pre-change + getSustain: () => true, }) mgr.track( 'sub-56', @@ -1527,6 +1663,7 @@ describe('CacheKeepManager tick/prewarm', () => { ttlMs: longTtl, leadMs: LEAD_MS, maxSubagentIdleMs: 30 * 60 * 1000, + getSustain: () => true, }) mgr.track( 'sub-56-stuck', diff --git a/packages/opencode/src/tests/command-dialogs.test.ts b/packages/opencode/src/tests/command-dialogs.test.ts index 514900f..295181c 100644 --- a/packages/opencode/src/tests/command-dialogs.test.ts +++ b/packages/opencode/src/tests/command-dialogs.test.ts @@ -42,6 +42,25 @@ describe('command dialogs', () => { expect(options.map((option) => option.title)).toContain('Refresh status') }) + test('cachekeep modal exposes a main-only sustain toggle', () => { + const options = buildCachekeepDialogOptions({ + command: 'openai-cachekeep', + text: '', + knobs: { + enabled: true, + running: true, + tracked: 1, + sustain: false, + }, + }) + + expect(options).toContainEqual({ + title: 'Sustain main sessions: off', + value: 'sustain on', + description: expect.stringContaining('main-only'), + }) + }) + // The cachekeep dialog is implemented as JSX over the runtime-provided // `TuiPluginApi`. To exercise its onSelect without spinning the real TUI, // intercept the renderer's render fn and the runtime DialogSelect @@ -50,6 +69,11 @@ describe('command dialogs', () => { function makeCachekeepDialogHarness() { let capturedRenderer: (() => unknown) | null = null let capturedOnSelect: ((option: { value: string }) => void) | null = null + let capturedOptions: Array<{ + title: string + value: string + description?: string + }> = [] const clearCount = { value: 0 } const replaceCount = { value: 0 } @@ -68,8 +92,14 @@ describe('command dialogs', () => { toast: () => {}, DialogSelect: ((props: { onSelect: (option: { value: string }) => void + options?: Array<{ + title: string + value: string + description?: string + }> }) => { capturedOnSelect = props.onSelect + capturedOptions = props.options ?? [] return null }) as unknown as TuiPluginApi['ui']['DialogSelect'], }, @@ -81,11 +111,34 @@ describe('command dialogs', () => { capturedRenderer?.() }, getOnSelect: () => capturedOnSelect, + getOptions: () => capturedOptions, clearCount, replaceCount, } } + test('routing reset dialog exposes sticky-balanced selection and reset', () => { + const { api, renderDialog, getOptions } = makeCachekeepDialogHarness() + + openCommandDialog( + api, + { command: 'openai-routing', text: '', knobs: { mode: 'main-first' } }, + mock(async () => ({ text: '', knobs: {} })), + ) + + renderDialog() + expect(getOptions()).toContainEqual({ + title: 'Sticky balanced', + value: 'sticky-balanced', + description: expect.any(String), + }) + expect(getOptions()).toContainEqual({ + title: "Reset this session's pin", + value: 'reset', + description: expect.any(String), + }) + }) + test('cachekeep dialog "clear_window" applies "window clear" (not the literal option value)', async () => { const { api, renderDialog, getOnSelect } = makeCachekeepDialogHarness() const apply = mock(async () => ({ text: 'window cleared', knobs: {} })) diff --git a/packages/opencode/src/tests/commands.test.ts b/packages/opencode/src/tests/commands.test.ts index 949767b..4d21e6b 100644 --- a/packages/opencode/src/tests/commands.test.ts +++ b/packages/opencode/src/tests/commands.test.ts @@ -8,7 +8,13 @@ import { spyOn, test, } from 'bun:test' -import { existsSync, mkdtempSync, readFileSync, rmSync } from 'node:fs' +import { + existsSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync, +} from 'node:fs' import { tmpdir } from 'node:os' import { join } from 'node:path' import type { CommandContext } from '../commands' @@ -29,6 +35,10 @@ import { loadAccounts, type OAuthAccount, saveAccounts } from '../core/accounts' import { QuotaManager } from '../core/quota-manager' import { createLogger, flushForTest, setLogLevel } from '../logger' import { resetNotificationsForTest } from '../rpc/notifications' +import { + clearSidebarStickyAssignment, + hashSidebarSessionId, +} from '../sidebar-state' import { FLOOR_AUTH_FILE, FLOOR_STATE_FILE } from './setup-env.ts' function makeAccount( @@ -128,6 +138,292 @@ describe('commands', () => { expect(storage?.routing?.mode).toBe('fallback-first') }) + test('routing command persists and reports sticky-balanced', async () => { + const ctx: CommandContext = { + accountStoragePath: configPath, + quotaManager: new QuotaManager({ storage: { version: 1, accounts: [] } }), + loadAccounts, + client: makeClient(), + } + + const payload = await buildDialogPayload( + 'openai-routing', + 'sticky-balanced', + ctx, + ) + + expect(payload.knobs.mode).toBe('sticky-balanced') + expect(payload.text).toContain('sticky-balanced') + expect((await loadAccounts(configPath))?.routing?.mode).toBe( + 'sticky-balanced', + ) + }) + + test('account command lists sticky-balanced routing and session reset', async () => { + await saveAccounts( + { + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [makeAccount('fallback-1')], + }, + configPath, + ) + const ctx: CommandContext = { + accountStoragePath: configPath, + quotaManager: new QuotaManager({ storage: { version: 1, accounts: [] } }), + loadAccounts, + client: makeClient(), + } + + const payload = await buildDialogPayload('openai-account', '', ctx) + + expect(payload.text).toContain( + 'main-first, fallback-first, or sticky-balanced', + ) + expect(payload.text).toContain('/openai-routing reset') + }) + + test('routing reset clears only the current session pin', async () => { + const clearStickyRouting = mock(async () => true) + const ctx: CommandContext = { + accountStoragePath: configPath, + quotaManager: new QuotaManager({ storage: { version: 1, accounts: [] } }), + loadAccounts, + client: makeClient(), + sessionId: 'session-a', + clearStickyRouting, + } + + const payload = await buildDialogPayload('openai-routing', 'reset', ctx) + + expect(clearStickyRouting).toHaveBeenCalledTimes(1) + expect(clearStickyRouting).toHaveBeenCalledWith('session-a') + expect(payload.knobs.mode).toBe('main-first') + expect(payload.text).toContain('pin was cleared') + expect(payload.text).toContain('may choose the same account') + }) + + test('routing status reports a sticky session pin and its reset command', async () => { + await saveAccounts( + { + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [], + routing: { mode: 'sticky-balanced' }, + }, + configPath, + ) + const ctx: CommandContext = { + accountStoragePath: configPath, + quotaManager: new QuotaManager({ storage: { version: 1, accounts: [] } }), + loadAccounts, + client: makeClient(), + sessionId: 'sticky-status-session', + getStickyRouting: async () => 'fallback-1', + } + + const payload = await buildDialogPayload('openai-routing', '', ctx) + + expect(payload.text).toContain('Session pin: `fallback-1`') + expect(payload.text).toContain('/openai-routing reset') + }) + + test('routing reset and cachekeep sustain log their setting changes without a raw session id', async () => { + const logFile = join(tmpDir, 'commands.log') + const savedLogFile = process.env.OPENCODE_OPENAI_AUTH_LOG_FILE + try { + await flushForTest() + process.env.OPENCODE_OPENAI_AUTH_LOG_FILE = logFile + setLogLevel('info') + const ctx: CommandContext = { + accountStoragePath: configPath, + quotaManager: new QuotaManager({ + storage: { version: 1, accounts: [] }, + }), + loadAccounts, + client: makeClient(), + sessionId: 'raw-command-session', + clearStickyRouting: async () => true, + cacheKeepManager: { + status: () => ({ generatedAt: Date.now() }), + } as CommandContext['cacheKeepManager'], + } + + await buildDialogPayload('openai-routing', 'reset', ctx) + await buildDialogPayload('openai-cachekeep', 'sustain on', ctx) + await flushForTest() + + const text = readFileSync(logFile, 'utf8') + expect(text).toContain('routing session pin cleared') + expect(text).toContain('cachekeep sustain enabled') + expect(text).not.toContain('raw-command-session') + } finally { + await flushForTest() + if (savedLogFile === undefined) { + delete process.env.OPENCODE_OPENAI_AUTH_LOG_FILE + } else { + process.env.OPENCODE_OPENAI_AUTH_LOG_FILE = savedLogFile + } + setLogLevel(undefined) + } + }) + + test('routing reset without a current session changes nothing', async () => { + const clearStickyRouting = mock(async () => true) + const ctx: CommandContext = { + accountStoragePath: configPath, + quotaManager: new QuotaManager({ storage: { version: 1, accounts: [] } }), + loadAccounts, + client: makeClient(), + clearStickyRouting, + } + + const payload = await buildDialogPayload('openai-routing', 'reset', ctx) + + expect(clearStickyRouting).not.toHaveBeenCalled() + expect(payload.text).toContain('No current session') + }) + + test('routing reset removes only one sticky pin and leaves health and display state intact', async () => { + const sidebarPath = join(tmpDir, 'sidebar-state.json') + const now = Date.now() + const sessionA = 'session-a' + const sessionB = 'session-b' + const hashA = hashSidebarSessionId(sessionA) + const hashB = hashSidebarSessionId(sessionB) + writeFileSync( + sidebarPath, + JSON.stringify({ + main: { + quota: { + primary: { + usedPercent: 20, + remainingPercent: 80, + checkedAt: now, + resetsAt: new Date(now + 60_000).toISOString(), + }, + }, + killed: true, + quotaBackedOff: true, + quotaBackoffUntil: now + 60_000, + refreshBackedOff: true, + refreshBackoffUntil: now + 120_000, + }, + fallbacks: [ + { + id: 'fallback-1', + label: 'Fallback 1', + quota: null, + killed: false, + enabled: true, + }, + ], + activeId: 'fallback-1', + route: 'sticky-balanced', + activeRouting: { + [sessionA]: { + activeId: 'fallback-1', + route: 'sticky-balanced', + updatedAt: now, + }, + [sessionB]: { + activeId: 'main', + route: 'sticky-balanced', + updatedAt: now, + }, + }, + stickyAssignments: { + [hashA]: { + accountId: 'fallback-1', + assignedAt: now, + lastSeenAt: now, + inputBytes: 100, + }, + [hashB]: { + accountId: 'main', + assignedAt: now, + lastSeenAt: now, + inputBytes: 200, + }, + }, + lastUpdated: now, + }), + ) + await saveAccounts( + { + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [], + routing: { mode: 'sticky-balanced' }, + killswitch: { enabled: true }, + }, + configPath, + ) + const quotaManager = new QuotaManager({ + storage: { version: 1, accounts: [] }, + now: () => now, + }) + quotaManager.markRateLimited('main', now + 60_000) + const cacheKeepManager = { + status: () => ({ running: true, tracked: 2, generatedAt: now }), + } as unknown as CommandContext['cacheKeepManager'] + const before = JSON.parse(readFileSync(sidebarPath, 'utf8')) + const ctx: CommandContext = { + accountStoragePath: configPath, + quotaManager, + loadAccounts, + client: makeClient(), + sessionId: sessionA, + cacheKeepManager, + clearStickyRouting: (sessionId) => + clearSidebarStickyAssignment(sessionId, sidebarPath), + } + + await buildDialogPayload('openai-routing', 'reset', ctx) + + const after = JSON.parse(readFileSync(sidebarPath, 'utf8')) + expect(after.stickyAssignments?.[hashA]).toBeUndefined() + expect(after.stickyAssignments?.[hashB]).toEqual( + before.stickyAssignments[hashB], + ) + const { + stickyAssignments: _beforePins, + lastUpdated: _beforeUpdated, + ...beforeWithoutPins + } = before + const { + stickyAssignments: _afterPins, + lastUpdated: _afterUpdated, + ...afterWithoutPins + } = after + expect(afterWithoutPins).toEqual(beforeWithoutPins) + expect(quotaManager.isRateLimited('main')).toBe(true) + expect(cacheKeepManager?.status().tracked).toBe(2) + expect((await loadAccounts(configPath))?.killswitch).toEqual({ + enabled: true, + }) + }) + + test.each(['always', 'hold'])( + 'routing command rejects obsolete alias %s', + async (alias) => { + const ctx: CommandContext = { + accountStoragePath: configPath, + quotaManager: new QuotaManager({ + storage: { version: 1, accounts: [] }, + }), + loadAccounts, + client: makeClient(), + } + + const payload = await buildDialogPayload('openai-routing', alias, ctx) + + expect(payload.knobs.mode).toBe('main-first') + expect(payload.text).toContain('sticky-balanced') + expect((await loadAccounts(configPath))?.routing?.mode).toBeUndefined() + }, + ) + test('scalar command (routing) with a STALE snapshot does not resurrect a removed account or its secrets', async () => { // Disk authoritatively has only account `a` (e.g. `gone` was just removed by // another session). The scalar command handler, however, loaded a STALE @@ -223,6 +519,7 @@ describe('commands', () => { maxSubagentIdleMs: 30 * 60 * 1000, ttlMs: 5 * 60 * 1000, leadMs: 5000, + sustain: false, targets: [], }), } as unknown as CommandContext['cacheKeepManager'], @@ -233,7 +530,10 @@ describe('commands', () => { expect(payload.command).toBe('openai-cachekeep') expect(payload.text).toContain('Status: **ON**') expect(payload.text).toContain('Timer: **idle**') + expect(payload.text).toContain('Idle policy: **sustain OFF (main only)**') + expect(payload.text).toContain('Window: **always (no window)**') expect(payload.knobs.enabled).toBe(true) + expect(payload.knobs.sustain).toBe(false) expect(payload.knobs.running).toBe(false) expect(payload.knobs.tracked).toBe(0) }) @@ -333,6 +633,88 @@ describe('commands', () => { expect(setCacheKeepSubagents).toHaveBeenCalledWith(false) }) + test('/openai-cachekeep sustain on/off persists and flips the live gate', async () => { + await saveAccounts( + { + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [], + cachekeep: { enabled: true, subagents: true, sustain: false }, + }, + configPath, + ) + const setCacheKeepSustain = mock(() => {}) + const ctx: CommandContext = { + accountStoragePath: configPath, + quotaManager: new QuotaManager({ storage: { version: 1, accounts: [] } }), + loadAccounts, + client: makeClient(), + setCacheKeepSustain, + cacheKeepManager: { + status: () => ({ + running: true, + tracked: 0, + generatedAt: 1700000000000, + startedAt: 1700000000000, + maxIdleWarmMs: 60 * 60 * 1000, + maxSubagentIdleMs: 30 * 60 * 1000, + ttlMs: 5 * 60 * 1000, + leadMs: 5000, + sustain: true, + targets: [], + }), + } as unknown as CommandContext['cacheKeepManager'], + } + + const on = await buildDialogPayload('openai-cachekeep', 'sustain on', ctx) + expect(on.knobs.sustain).toBe(true) + expect(on.text).toContain('non-expiring `cache_ttl`') + expect((await loadAccounts(configPath))?.cachekeep).toEqual({ + enabled: true, + subagents: true, + sustain: true, + }) + expect(setCacheKeepSustain).toHaveBeenCalledWith(true) + + const off = await buildDialogPayload('openai-cachekeep', 'sustain off', ctx) + expect(off.knobs.sustain).toBe(false) + expect((await loadAccounts(configPath))?.cachekeep).toEqual({ + enabled: true, + subagents: true, + sustain: false, + }) + expect(setCacheKeepSustain).toHaveBeenCalledWith(false) + }) + + test('/openai-cachekeep does not accept always or hold as sustain aliases', async () => { + await saveAccounts( + { + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [], + cachekeep: { enabled: true, sustain: false }, + }, + configPath, + ) + const ctx: CommandContext = { + accountStoragePath: configPath, + quotaManager: new QuotaManager({ storage: { version: 1, accounts: [] } }), + loadAccounts, + client: makeClient(), + } + + const always = await buildDialogPayload( + 'openai-cachekeep', + 'always on', + ctx, + ) + const hold = await buildDialogPayload('openai-cachekeep', 'hold on', ctx) + + expect(always.text).toContain('Usage:') + expect(hold.text).toContain('Usage:') + expect((await loadAccounts(configPath))?.cachekeep?.sustain).toBe(false) + }) + test('/openai-cachekeep on creates a store when none exists', async () => { expect(existsSync(configPath)).toBe(false) const start = mock(() => {}) diff --git a/packages/opencode/src/tests/dump.test.ts b/packages/opencode/src/tests/dump.test.ts index ed8938f..33cc607 100644 --- a/packages/opencode/src/tests/dump.test.ts +++ b/packages/opencode/src/tests/dump.test.ts @@ -384,6 +384,37 @@ describe('request dumps', () => { }) }) + test('records the internal serving account without exposing a ChatGPT account id', async () => { + await withDumpEnv(async (dumpDir) => { + await dumpCodexRequest({ + sessionID: 'ses_dump_account', + transport: 'http', + phase: 'http', + accountId: 'work-alt', + bodyText: JSON.stringify({ input: [] }), + headers: { 'chatgpt-account-id': 'chatgpt-account-secret' }, + }) + + const files = await readdir(dumpDir) + const metadata = JSON.parse( + await readFile(join(dumpDir, requireFile(files, '.meta.json')), 'utf8'), + ) + const request = JSON.parse( + await readFile( + join(dumpDir, requireFile(files, '.request.json')), + 'utf8', + ), + ) + + expect(metadata.accountId).toBe('work-alt') + expect(request.accountId).toBe('work-alt') + expect(request.headers['chatgpt-account-id']).toBe('[redacted]') + expect(JSON.stringify({ metadata, request })).not.toContain( + 'chatgpt-account-secret', + ) + }) + }) + test('redacts credentials from JSON dump bodies', async () => { await withDumpEnv(async (dumpDir) => { const bearer = 'Bearer dump-body-token' diff --git a/packages/opencode/src/tests/integration.test.ts b/packages/opencode/src/tests/integration.test.ts index da85ea8..53e6185 100644 --- a/packages/opencode/src/tests/integration.test.ts +++ b/packages/opencode/src/tests/integration.test.ts @@ -13,6 +13,7 @@ import type { Hooks, PluginInput } from '@opencode-ai/plugin' import type { OAuthAccount } from '../core/accounts.ts' import { migrateIfNeeded } from '../core/accounts.ts' import { acquireRefreshFileLock } from '../core/refresh-file-lock.ts' +import { QUOTA_STALENESS_MS } from '../core/sticky-routing.ts' import { AuthPersistError, CodexAuthPlugin, @@ -21,11 +22,13 @@ import { MAIN_REFRESH_LOCK_TTL_MS, resolveSidebarSessionId, } from '../index.ts' +import { flushForTest, setLogLevel } from '../logger.ts' import { resetModelCostsForTest } from '../model-costs.ts' import { ResponseStreamError } from '../response-stream-error' import { drainSidebarWrites, getSidebarStateFile, + hashSidebarSessionId, normalizeSidebarState, resolveSessionSidebarRouting, type SidebarState, @@ -2601,7 +2604,12 @@ describe('integration: active fallback routing', () => { return { hooks, fetchOverride } } - async function runCommand(hooks: Hooks, command: string, args = '') { + async function runCommand( + hooks: Hooks, + command: string, + args = '', + sessionID = 'test-session', + ) { const hook = hooks['command.execute.before'] as | ((input: { command: string @@ -2611,7 +2619,7 @@ describe('integration: active fallback routing', () => { | undefined if (!hook) throw new Error('No command hook') try { - await hook({ command, arguments: args, sessionID: 'test-session' }) + await hook({ command, arguments: args, sessionID }) } catch (error) { if ( !(error instanceof Error) || @@ -2858,714 +2866,932 @@ describe('integration: active fallback routing', () => { writeFileSync(sidebarFile, JSON.stringify(state)) } - function mockAdmissionFetch(seenAuth: string[], status = 200) { - return (async (url: unknown, init?: unknown) => { - if (String(url).includes('responses')) { - seenAuth.push(headerValue(init, 'authorization')) - } - return new Response('{}', { status }) - }) as unknown as typeof globalThis.fetch + function stickyQuota(remainingPercent: number, checkedAt: number) { + return { + primary: { + usedPercent: 100 - remainingPercent, + remainingPercent, + checkedAt, + resetsAt: new Date(checkedAt + 7 * 24 * 3600_000).toISOString(), + windowMinutes: 300, + }, + } } - test.each([ - [ - { - 'x-session-affinity': 'affinity', - 'x-opencode-session': 'opencode', - 'x-session-id': 'x-session', - 'session-id': 'session', - }, - 'affinity', - ], - [ - { 'x-opencode-session': 'opencode', 'x-session-id': 'x-session' }, - 'opencode', - ], - [{ 'x-session-id': 'x-session', 'session-id': 'session' }, 'x-session'], - [{ 'session-id': 'session' }, 'session'], - [{}, undefined], - ])( - 'resolves sidebar session headers by documented precedence', - (raw, expected) => { - expect(resolveSidebarSessionId(new Headers(raw))).toBe(expected) - }, - ) + function seedStickyBalancedAccounts() { + const checkedAt = Date.now() + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + routing: { mode: 'sticky-balanced' }, + refresh: { refreshBeforeExpiryMinutes: 5 }, + accounts: [ + { + id: 'fallback-1', + type: 'oauth', + enabled: true, + access: 'fallback-1-token', + refresh: 'fallback-1-refresh', + expires: Date.now() + 24 * 3600_000, + accountId: 'acc-fallback-1', + }, + { + id: 'fallback-2', + type: 'oauth', + enabled: true, + access: 'fallback-2-token', + refresh: 'fallback-2-refresh', + expires: Date.now() + 24 * 3600_000, + accountId: 'acc-fallback-2', + }, + ], + }), + ) + writeFileSync( + sidebarFile, + JSON.stringify({ + main: { + quota: stickyQuota(20, checkedAt), + mainAccountId: 'acc-main', + killed: false, + }, + fallbacks: [ + { + id: 'fallback-1', + label: 'Fallback 1', + accountId: 'acc-fallback-1', + quota: stickyQuota(90, checkedAt), + killed: false, + enabled: true, + }, + { + id: 'fallback-2', + label: 'Fallback 2', + accountId: 'acc-fallback-2', + quota: stickyQuota(100, checkedAt), + killed: false, + enabled: true, + }, + ], + route: 'sticky-balanced', + lastUpdated: checkedAt, + }), + ) + } - it('keeps different served accounts under different session keys', async () => { - seedStorage({ access: 'fallback-access-token' }) + it('sticky-balanced pins a session, retains it, and balances a new session by pending bytes', async () => { + seedStickyBalancedAccounts() + const seenAuth: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = (async () => - new Response('{}', { status: 200 })) as unknown as typeof globalThis.fetch + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('responses')) { + seenAuth.push(headerValue(init, 'authorization')) + } + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), Date.now() + 3600_000, + false, + false, + 'acc-main', ) hooks = loaded.hooks - await loaded.fetchOverride( + const first = await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - responseRequestInit({ 'x-opencode-session': 'sess-fallback' }), + responseRequestInit({ 'x-session-affinity': 'sticky-session' }), ) + expect(first.status).toBe(200) + expect(seenAuth).toEqual(['Bearer fallback-2-token']) + + await drainSidebarWrites() + const firstState = normalizeSidebarState( + JSON.parse(readFileSync(sidebarFile, 'utf8')), + ) + const firstAssignment = + firstState.stickyAssignments?.[hashSidebarSessionId('sticky-session')] + expect(firstAssignment?.accountId).toBe('fallback-2') + + const changed = JSON.parse(readFileSync(sidebarFile, 'utf8')) + changed.fallbacks[0].quota = stickyQuota(100, Date.now()) + changed.fallbacks[1].quota = stickyQuota(1, Date.now()) + writeFileSync(sidebarFile, JSON.stringify(changed)) - seedStorage({ access: 'fallback-access-token' }, { mode: 'main-first' }) await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - responseRequestInit({ 'x-opencode-session': 'sess-main' }), + responseRequestInit({ 'x-session-affinity': 'sticky-session' }), ) - await drainSidebarWrites() + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'x-session-affinity': 'cold-session' }), + ) + expect(seenAuth).toEqual([ + 'Bearer fallback-2-token', + 'Bearer fallback-2-token', + 'Bearer fallback-1-token', + ]) - const sidebar = normalizeSidebarState( + await drainSidebarWrites() + const finalState = normalizeSidebarState( JSON.parse(readFileSync(sidebarFile, 'utf8')), ) - expect(sidebar.activeRouting?.['sess-fallback']).toMatchObject({ - activeId: 'fallback-1', - route: 'fallback-first', - }) - expect(sidebar.activeRouting?.['sess-main']).toMatchObject({ - activeId: 'main', - route: 'main-first', - }) - expect(sidebar.activeId).toBe('main') - expect(sidebar.route).toBe('main-first') + expect( + finalState.stickyAssignments?.[hashSidebarSessionId('cold-session')] + ?.accountId, + ).toBe('fallback-1') } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('records fallback-served routing on the parent session', async () => { - seedStorage({ access: 'fallback-access-token' }) + it('logs a new sticky placement with its selection fields', async () => { + seedStickyBalancedAccounts() + const sessionId = 'placement-log-session' + const request = responseRequestInit({ 'x-session-affinity': sessionId }) const originalFetch = globalThis.fetch globalThis.fetch = (async () => new Response('{}', { status: 200 })) as unknown as typeof globalThis.fetch - + setLogLevel('debug') let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), Date.now() + 3600_000, + false, + false, + 'acc-main', ) hooks = loaded.hooks - - await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - responseRequestInit({ - 'x-opencode-session': 'child-session', - 'x-parent-session-id': 'parent-session', - }), - ) - await drainSidebarWrites() - - const sidebar = normalizeSidebarState( - JSON.parse(readFileSync(sidebarFile, 'utf8')), - ) - expect(sidebar.activeRouting?.['child-session']).toMatchObject({ - activeId: 'fallback-1', - route: 'fallback-first', - }) - // A fallback (not main) served the child; the parent entry must mirror - // that same fallback so the parent's sidebar highlights the live account. - expect(sidebar.activeRouting?.['parent-session']).toMatchObject({ - activeId: 'fallback-1', - route: 'fallback-first', + await loaded.fetchOverride('https://api.openai.com/v1/responses', request) + await flushForTest() + + const line = readFileSync(logFile, 'utf8') + .split('\n') + .find((entry) => entry.includes('sticky routing: placed session pin')) + if (!line) throw new Error('missing sticky placement log') + const payload = JSON.parse(line.slice(line.indexOf('{'))) + expect(payload).toMatchObject({ + sessionHash: hashSidebarSessionId(sessionId), + accountId: 'fallback-2', + source: 'weighted', + requestBytes: Buffer.byteLength(String(request.body), 'utf8'), + pendingBytes: 0, }) + expect(line).not.toContain(sessionId) } finally { globalThis.fetch = originalFetch + setLogLevel(undefined) await hooks?.dispose?.() } }) - it('uses legacy display routing when the request carries no session headers', async () => { - seedStorage({ access: 'fallback-access-token' }) + it('logs both reachable sticky migration paths with account and reason', async () => { + const sessionId = 'pre-send-migration-session' const originalFetch = globalThis.fetch - globalThis.fetch = (async () => - new Response('{}', { status: 200 })) as unknown as typeof globalThis.fetch - + globalThis.fetch = (async (_url: unknown, init?: unknown) => { + const auth = headerValue(init, 'authorization') + return new Response('{}', { + status: auth.includes('fallback-2') ? 401 : 200, + }) + }) as unknown as typeof globalThis.fetch + setLogLevel('debug') let hooks: Hooks | undefined try { + seedStickyBalancedAccounts() + const preSendState = JSON.parse(readFileSync(sidebarFile, 'utf8')) + preSendState.fallbacks[1].quota = stickyQuota(0, Date.now()) + preSendState.stickyAssignments = { + [hashSidebarSessionId(sessionId)]: { + accountId: 'fallback-2', + assignedAt: Date.now(), + lastSeenAt: Date.now(), + inputBytes: 1, + }, + } + writeFileSync(sidebarFile, JSON.stringify(preSendState)) const loaded = await loadFetchOverride( createMockPluginInput(), Date.now() + 3600_000, + false, + false, + 'acc-main', ) hooks = loaded.hooks - await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - responseRequestInit(), + responseRequestInit({ 'x-session-affinity': sessionId }), + ) + + const postResponseSession = 'post-response-migration-session' + seedStickyBalancedAccounts() + const postResponseState = JSON.parse(readFileSync(sidebarFile, 'utf8')) + postResponseState.stickyAssignments = { + [hashSidebarSessionId(postResponseSession)]: { + accountId: 'fallback-2', + assignedAt: Date.now(), + lastSeenAt: Date.now(), + inputBytes: 1, + }, + } + writeFileSync(sidebarFile, JSON.stringify(postResponseState)) + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'x-session-affinity': postResponseSession }), ) - await drainSidebarWrites() + await flushForTest() - const sidebar = normalizeSidebarState( - JSON.parse(readFileSync(sidebarFile, 'utf8')), + const migrations = readFileSync(logFile, 'utf8') + .split('\n') + .filter((entry) => + entry.includes('sticky routing: migrated session pin'), + ) + .map((entry) => JSON.parse(entry.slice(entry.indexOf('{')))) + expect(migrations).toContainEqual( + expect.objectContaining({ + sessionHash: hashSidebarSessionId(sessionId), + fromAccountId: 'fallback-2', + toAccountId: 'fallback-1', + reason: 'exhausted', + }), + ) + expect(migrations).toContainEqual( + expect.objectContaining({ + sessionHash: hashSidebarSessionId(postResponseSession), + fromAccountId: 'fallback-2', + toAccountId: 'fallback-1', + reason: 'permanent', + }), ) - // Sessionless requests write only the legacy display fields, never a - // per-session entry. - expect(sidebar.activeRouting).toBeUndefined() - expect(sidebar.activeId).toBe('fallback-1') - expect(sidebar.route).toBe('fallback-first') - // Resolving without a session reads those legacy fields and must yield - // defined routing rather than crash or return undefined. - expect(resolveSessionSidebarRouting(sidebar, undefined)).toEqual({ - activeId: 'fallback-1', - route: 'fallback-first', - }) } finally { globalThis.fetch = originalFetch + setLogLevel(undefined) await hooks?.dispose?.() } }) - it('reads the sidebar session from a Request and strips it before the wire', async () => { - seedEmptyAccountStorage() + it('does not log placement or migration when retaining a healthy sticky pin', async () => { + seedStickyBalancedAccounts() + const sessionId = 'healthy-retained-pin' const originalFetch = globalThis.fetch - let wireHeaders = new Headers() - let wireMethod: string | undefined - let wireBody: BodyInit | null | undefined - globalThis.fetch = (async (_url: unknown, init?: RequestInit) => { - wireHeaders = new Headers(init?.headers) - wireMethod = init?.method - wireBody = init?.body - return new Response('{}', { status: 200 }) - }) as typeof globalThis.fetch - + globalThis.fetch = (async () => + new Response('{}', { status: 200 })) as unknown as typeof globalThis.fetch + setLogLevel('debug') let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), Date.now() + 3600_000, + false, + false, + 'acc-main', ) hooks = loaded.hooks - const requestBody = responseRequestInit() - const request = new Request('https://api.openai.com/v1/responses', { - method: 'POST', - headers: { - 'content-type': 'application/json', - 'x-opencode-session': 'request-session', - }, - body: requestBody.body, - }) - await loaded.fetchOverride(request) - await drainSidebarWrites() + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'x-session-affinity': sessionId }), + ) + await flushForTest() + writeFileSync(logFile, '') - const sidebar = normalizeSidebarState( - JSON.parse(readFileSync(sidebarFile, 'utf8')), + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'x-session-affinity': sessionId }), ) - expect(sidebar.activeRouting?.['request-session']).toMatchObject({ - activeId: 'main', - route: 'main-first', - }) - expect(wireHeaders.has('x-opencode-session')).toBe(false) - expect(wireMethod).toBe('POST') - expect(JSON.parse(String(wireBody)).model).toBe('gpt-5.5') + await flushForTest() + + const text = readFileSync(logFile, 'utf8') + expect(text).not.toContain('sticky routing: placed session pin') + expect(text).not.toContain('sticky routing: migrated session pin') } finally { globalThis.fetch = originalFetch + setLogLevel(undefined) await hooks?.dispose?.() } }) - it('does not mutate frozen caller-owned request headers', async () => { - seedEmptyAccountStorage() - const callerHeaders = Object.freeze({ - authorization: 'Bearer caller-token', - 'content-type': 'application/json', - 'x-api-key': 'caller-key', - }) - const callerInit = Object.freeze({ - method: 'POST', - headers: callerHeaders, - body: responseRequestInit().body, - }) as RequestInit - const originalFetch = globalThis.fetch - let wireHeaders = new Headers() - globalThis.fetch = (async (_url: unknown, init?: RequestInit) => { - wireHeaders = new Headers(init?.headers) - return new Response('{}', { status: 200 }) - }) as typeof globalThis.fetch + it('a degraded request roster read preserves existing fallback pins through the main-path sidebar writer', async () => { + seedStickyBalancedAccounts() + const retainedHash = hashSidebarSessionId('retained-after-degraded-read') + const seeded = JSON.parse(readFileSync(sidebarFile, 'utf8')) + seeded.stickyAssignments = { + [retainedHash]: { + accountId: 'fallback-1', + assignedAt: Date.now(), + lastSeenAt: Date.now(), + inputBytes: 1, + }, + } + writeFileSync(sidebarFile, JSON.stringify(seeded)) + const originalFetch = globalThis.fetch + globalThis.fetch = (async () => + new Response('{}', { status: 200 })) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), Date.now() + 3600_000, + false, + false, + 'acc-main', ) hooks = loaded.hooks - await loaded.fetchOverride( + renameSync(configFile, `${configFile}.unavailable`) + + const response = await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - callerInit, + responseRequestInit({ 'x-session-affinity': 'degraded-read-session' }), ) + expect(response.status).toBe(200) + await drainSidebarWrites() - expect(callerHeaders.authorization).toBe('Bearer caller-token') - expect(callerHeaders['x-api-key']).toBe('caller-key') - expect(wireHeaders.get('authorization')).toBe('Bearer main-stale-token') - expect(wireHeaders.has('x-api-key')).toBe(false) + expect( + normalizeSidebarState(JSON.parse(readFileSync(sidebarFile, 'utf8'))) + .stickyAssignments?.[retainedHash]?.accountId, + ).toBe('fallback-1') } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('uses the active fallback token without writing it to the auth slot', async () => { - seedStorage({ access: 'fallback-access-token' }) - const authSetCalls: unknown[] = [] - const seen: Array<{ authorization: string; accountId: string | null }> = [] - const originalFetch = globalThis.fetch - globalThis.fetch = (async (url: unknown, init?: unknown) => { - if (String(url).includes('/oauth/token')) { - throw new Error('refresh unavailable') - } - seen.push({ - authorization: headerValue(init, 'authorization'), - accountId: headerValue(init, 'ChatGPT-Account-Id') || null, - }) - return new Response('{}', { status: 200 }) - }) as unknown as typeof globalThis.fetch + it('a degraded roster read on session deletion removes only the deleted pin', async () => { + seedStickyBalancedAccounts() + const deletedSessionId = 'deleted-after-degraded-read' + const retainedHash = hashSidebarSessionId('retained-after-degraded-delete') + const seeded = JSON.parse(readFileSync(sidebarFile, 'utf8')) + seeded.stickyAssignments = { + [hashSidebarSessionId(deletedSessionId)]: { + accountId: 'fallback-1', + assignedAt: Date.now(), + lastSeenAt: Date.now(), + inputBytes: 1, + }, + [retainedHash]: { + accountId: 'fallback-2', + assignedAt: Date.now(), + lastSeenAt: Date.now(), + inputBytes: 2, + }, + } + writeFileSync(sidebarFile, JSON.stringify(seeded)) let hooks: Hooks | undefined try { - const input = createMockPluginInput({ - client: { - auth: { set: async (payload: unknown) => authSetCalls.push(payload) }, - session: { promptAsync: async () => {} }, - } as unknown as PluginInput['client'], - }) - const loaded = await loadFetchOverride(input, Date.now() + 3600_000) - hooks = loaded.hooks - - const response = await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - requestInit(), + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + false, + false, + 'acc-main', ) + hooks = loaded.hooks + renameSync(configFile, `${configFile}.unavailable`) + const event = ( + hooks as unknown as { + event?: (input: { + event: { + type: 'session.deleted' + properties: { info: { id: string } } + } + }) => Promise + } + ).event + if (!event) throw new Error('No event hook') - expect(response.status).toBe(200) - expect(seen).toEqual([ - { - authorization: 'Bearer fallback-access-token', - accountId: 'acc-fallback-1', + await event({ + event: { + type: 'session.deleted', + properties: { info: { id: deletedSessionId } }, }, - ]) - expect(authSetCalls).toEqual([]) + }) + await drainSidebarWrites() + + const stickyAssignments = normalizeSidebarState( + JSON.parse(readFileSync(sidebarFile, 'utf8')), + ).stickyAssignments + expect( + stickyAssignments?.[hashSidebarSessionId(deletedSessionId)], + ).toBeUndefined() + expect(stickyAssignments?.[retainedHash]?.accountId).toBe('fallback-2') } finally { - globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('waits on a held main refresh file lock and uses the rotated auth token', async () => { - writeFileSync( - configFile, - JSON.stringify({ - version: 1, - main: { type: 'opencode', provider: 'openai' }, - accounts: [], - refresh: { refreshBeforeExpiryMinutes: 5 }, - }), - ) + it('routing reset removes a session pin without forcing a different account', async () => { + seedStickyBalancedAccounts() + const seenAuth: string[] = [] const originalFetch = globalThis.fetch - const seen: string[] = [] - let oauthRefreshCalls = 0 globalThis.fetch = (async (url: unknown, init?: unknown) => { - if (String(url).includes('/oauth/token')) { - oauthRefreshCalls++ - throw new Error('second process must not refresh') + if (String(url).includes('responses')) { + seenAuth.push(headerValue(init, 'authorization')) } - seen.push(headerValue(init, 'authorization')) return new Response('{}', { status: 200 }) }) as unknown as typeof globalThis.fetch - const heldLock = await acquireRefreshFileLock({ - name: 'main-refresh', - ttlMs: 60_000, - path: configFile, - renew: true, - }) - if (!heldLock) throw new Error('failed to acquire test lock') - - let auth = { - type: 'oauth' as const, - provider: 'openai', - access: 'main-stale-token', - refresh: 'main-refresh-token', - expires: Date.now() - 1_000, - } - setTimeout(() => { - auth = { - type: 'oauth' as const, - provider: 'openai', - access: 'main-rotated-token', - refresh: 'main-rotated-refresh', - expires: Date.now() + 3600_000, - } - }, 25) - let hooks: Hooks | undefined try { - const input = createMockPluginInput() - hooks = await CodexAuthPlugin(input, { experimentalWebSockets: false }) - const authHook = hooks.auth - if (!authHook?.loader) throw new Error('No auth loader') - const loaderResult = await authHook.loader(async () => auth, { - id: 'openai', - label: 'OpenAI', - models: [], - } as unknown as Parameters>[1]) - const fetchOverride = (loaderResult as Record).fetch as - | ((url: RequestInfo | URL, init?: RequestInit) => Promise) - | undefined - if (!fetchOverride) throw new Error('No fetch in loader result') + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + false, + false, + 'acc-main', + ) + hooks = loaded.hooks + const sessionId = 'reset-same-account-session' - const response = await fetchOverride( + await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + responseRequestInit({ 'x-session-affinity': sessionId }), + ) + await drainSidebarWrites() + expect( + normalizeSidebarState(JSON.parse(readFileSync(sidebarFile, 'utf8'))) + .stickyAssignments?.[hashSidebarSessionId(sessionId)]?.accountId, + ).toBe('fallback-2') + + const beforeReset = JSON.parse(readFileSync(sidebarFile, 'utf8')) + const selectionNow = Date.now() + beforeReset.main.quota = stickyQuota(1, selectionNow) + beforeReset.fallbacks[0].quota = stickyQuota(1, selectionNow) + beforeReset.fallbacks[1].quota = stickyQuota(100, selectionNow) + writeFileSync(sidebarFile, JSON.stringify(beforeReset)) + + await runCommand(hooks, 'openai-routing', 'reset', sessionId) + await drainSidebarWrites() + expect( + normalizeSidebarState(JSON.parse(readFileSync(sidebarFile, 'utf8'))) + .stickyAssignments?.[hashSidebarSessionId(sessionId)], + ).toBeUndefined() + + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'x-session-affinity': sessionId }), ) - expect(response.status).toBe(200) - expect(oauthRefreshCalls).toBe(0) - expect(seen).toEqual(['Bearer main-rotated-token']) + expect(seenAuth).toEqual([ + 'Bearer fallback-2-token', + 'Bearer fallback-2-token', + ]) } finally { - await heldLock.release() globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('deduplicates concurrent in-process main refreshes and releases the lock on success', async () => { - writeFileSync( - configFile, - JSON.stringify({ - version: 1, - main: { type: 'opencode', provider: 'openai' }, - accounts: [], - refresh: { refreshBeforeExpiryMinutes: 5 }, - }), - ) + it('sticky-balanced keeps the assignment high-water request size', async () => { + seedStickyBalancedAccounts() const originalFetch = globalThis.fetch - const seen: string[] = [] - const authSetCalls: unknown[] = [] - let oauthRefreshCalls = 0 - globalThis.fetch = (async (url: unknown, init?: unknown) => { - if (String(url).includes('/oauth/token')) { - oauthRefreshCalls++ - await new Promise((resolve) => setTimeout(resolve, 25)) - return new Response( - JSON.stringify({ - access_token: 'main-fresh-token', - refresh_token: 'main-fresh-refresh', - expires_in: 3600, - id_token: 'id', - }), - { status: 200, headers: { 'content-type': 'application/json' } }, - ) - } - seen.push(headerValue(init, 'authorization')) - return new Response('{}', { status: 200 }) - }) as unknown as typeof globalThis.fetch + globalThis.fetch = (async () => + new Response('{}', { status: 200 })) as unknown as typeof globalThis.fetch - let auth = { - type: 'oauth' as const, - provider: 'openai', - access: 'main-stale-token', - refresh: 'main-refresh-token', - expires: Date.now() - 1_000, - } let hooks: Hooks | undefined try { - const input = createMockPluginInput({ - client: { - auth: { - set: async (payload: unknown) => { - authSetCalls.push(payload) - const body = (payload as { body: typeof auth }).body - auth = { ...auth, ...body } - }, - }, - session: { promptAsync: async () => {} }, - } as unknown as PluginInput['client'], + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + false, + false, + 'acc-main', + ) + hooks = loaded.hooks + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'x-session-affinity': 'high-water-session' }), + ) + await drainSidebarWrites() + const first = normalizeSidebarState( + JSON.parse(readFileSync(sidebarFile, 'utf8')), + ).stickyAssignments?.[hashSidebarSessionId('high-water-session')] + const longer = responseRequestInit({ + 'x-session-affinity': 'high-water-session', }) - hooks = await CodexAuthPlugin(input, { experimentalWebSockets: false }) - const authHook = hooks.auth - if (!authHook?.loader) throw new Error('No auth loader') - const loaderResult = await authHook.loader(async () => auth, { - id: 'openai', - label: 'OpenAI', - models: [], - } as unknown as Parameters>[1]) - const fetchOverride = (loaderResult as Record).fetch as - | ((url: RequestInfo | URL, init?: RequestInit) => Promise) - | undefined - if (!fetchOverride) throw new Error('No fetch in loader result') - - const [first, second] = await Promise.all([ - fetchOverride('https://api.openai.com/v1/responses', requestInit()), - fetchOverride('https://api.openai.com/v1/responses', requestInit()), - ]) - - expect(first.status).toBe(200) - expect(second.status).toBe(200) - expect(oauthRefreshCalls).toBe(1) - expect(authSetCalls).toHaveLength(1) - expect(seen).toEqual([ - 'Bearer main-fresh-token', - 'Bearer main-fresh-token', - ]) + longer.body = `${longer.body}${'x'.repeat(1_024)}` + await loaded.fetchOverride('https://api.openai.com/v1/responses', longer) + await drainSidebarWrites() + const second = normalizeSidebarState( + JSON.parse(readFileSync(sidebarFile, 'utf8')), + ).stickyAssignments?.[hashSidebarSessionId('high-water-session')] - const releasedLock = await acquireRefreshFileLock({ - name: 'main-refresh', - ttlMs: 60_000, - path: configFile, - }) - expect(releasedLock).not.toBeNull() - await releasedLock?.release() + expect(second?.inputBytes).toBeGreaterThan(first?.inputBytes ?? 0) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('uses a stale main token on refresh failure and releases the lock', async () => { - writeFileSync( - configFile, - JSON.stringify({ - version: 1, - main: { type: 'opencode', provider: 'openai' }, - accounts: [], - refresh: { refreshBeforeExpiryMinutes: 5 }, - }), - ) + it('sticky-balanced migrates only on confirmed exhaustion or permanent auth failure', async () => { + seedStickyBalancedAccounts() + const seenAuth: string[] = [] const originalFetch = globalThis.fetch - const seen: string[] = [] - let oauthRefreshCalls = 0 globalThis.fetch = (async (url: unknown, init?: unknown) => { - if (String(url).includes('/oauth/token')) { - oauthRefreshCalls++ - return new Response('bad refresh', { status: 500 }) - } - seen.push(headerValue(init, 'authorization')) - return new Response('{}', { status: 200 }) + if (!String(url).includes('responses')) + return new Response('{}', { status: 200 }) + const auth = headerValue(init, 'authorization') + seenAuth.push(auth) + return new Response('{}', { + status: auth.includes('fallback-2-token') ? 401 : 200, + }) }) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - Date.now() - 1_000, + Date.now() + 3600_000, + false, + false, + 'acc-main', ) hooks = loaded.hooks - const response = await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + responseRequestInit({ 'x-session-affinity': 'migrate-session' }), ) expect(response.status).toBe(200) - expect(oauthRefreshCalls).toBe(1) - expect(seen).toEqual(['Bearer main-stale-token']) - - const releasedLock = await acquireRefreshFileLock({ - name: 'main-refresh', - ttlMs: 60_000, - path: configFile, - }) - expect(releasedLock).not.toBeNull() - await releasedLock?.release() + expect(seenAuth).toEqual([ + 'Bearer fallback-2-token', + 'Bearer fallback-1-token', + ]) + await drainSidebarWrites() + expect( + normalizeSidebarState(JSON.parse(readFileSync(sidebarFile, 'utf8'))) + .stickyAssignments?.[hashSidebarSessionId('migrate-session')] + ?.accountId, + ).toBe('fallback-1') } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('captures cachekeep fallback targets by storage id, not ChatGPT account id', async () => { - seedStorage({ - access: 'fallback-access-token', - accountId: 'chatgpt-work-alt', - }) - const prompts: string[] = [] + it('sticky-balanced skips a fallback pin marked by the fallback-first path', async () => { + seedStickyBalancedAccounts() + const markingConfig = JSON.parse(readFileSync(configFile, 'utf8')) + markingConfig.routing.mode = 'fallback-first' + markingConfig.accounts[0].enabled = false + writeFileSync(configFile, JSON.stringify(markingConfig)) + + let fallbackTwoSends = 0 + let replacementSends = 0 + let hooks: Hooks | undefined + await withFakeWebSocket( + ({ message, authorization }) => ({ + send() { + if (authorization === 'Bearer fallback-2-token') { + fallbackTwoSends += 1 + if (fallbackTwoSends === 1) { + message( + JSON.stringify({ + type: 'error', + error: { + type: 'usage_limit_reached', + resets_in_seconds: 0.05, + }, + }), + ) + return + } + message( + JSON.stringify({ + type: 'response.completed', + response: { id: `fallback-two-${fallbackTwoSends}` }, + }), + ) + return + } + replacementSends += 1 + message( + JSON.stringify({ + type: 'response.completed', + response: { id: `replacement-${replacementSends}` }, + }), + ) + }, + }), + async () => { + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + true, + false, + 'acc-main', + ) + hooks = loaded.hooks + const request = (sessionId: string) => { + const init = responseRequestInit({ 'session-id': sessionId }) + init.body = JSON.stringify({ + model: 'gpt-5.5', + input: [], + stream: true, + }) + return init + } + + const marked = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + request('marked-fallback-session'), + ) + await expect(marked.text()).rejects.toMatchObject({ + isRetryable: true, + }) + expect(fallbackTwoSends).toBe(1) + + const stickyConfig = JSON.parse(readFileSync(configFile, 'utf8')) + stickyConfig.routing.mode = 'sticky-balanced' + stickyConfig.accounts[0].enabled = true + writeFileSync(configFile, JSON.stringify(stickyConfig)) + await drainSidebarWrites() + const stickyState = JSON.parse(readFileSync(sidebarFile, 'utf8')) + stickyState.stickyAssignments = { + ...(stickyState.stickyAssignments ?? {}), + [hashSidebarSessionId('marked-fallback-session')]: { + accountId: 'fallback-2', + assignedAt: Date.now(), + lastSeenAt: Date.now(), + inputBytes: 1, + }, + } + writeFileSync(sidebarFile, JSON.stringify(stickyState)) + + const replacement = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + request('marked-fallback-session'), + ) + await replacement.text() + expect(fallbackTwoSends).toBe(1) + expect(replacementSends).toBe(1) + + await Bun.sleep(100) + await drainSidebarWrites() + const expiredState = JSON.parse(readFileSync(sidebarFile, 'utf8')) + expiredState.stickyAssignments = { + ...(expiredState.stickyAssignments ?? {}), + [hashSidebarSessionId('eligible-fallback-session')]: { + accountId: 'fallback-2', + assignedAt: Date.now(), + lastSeenAt: Date.now(), + inputBytes: 1, + }, + } + writeFileSync(sidebarFile, JSON.stringify(expiredState)) + + const afterExpiry = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + request('eligible-fallback-session'), + ) + await afterExpiry.text() + expect(fallbackTwoSends).toBe(2) + } finally { + await drainSidebarWrites() + await hooks?.dispose?.() + } + }, + ) + }) + + it('sticky-balanced skips a freshly exhausted pin before sending and excludes it from migration', async () => { + seedStickyBalancedAccounts() + const seenAuth: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = (async (_url: unknown, _init?: unknown) => { + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('responses')) { + seenAuth.push(headerValue(init, 'authorization')) + } return new Response('{}', { status: 200 }) }) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( - createMockPluginInput({ - client: { - auth: { set: async () => {} }, - session: { - promptAsync: async (request: unknown) => { - const body = ( - request as { body?: { parts?: Array<{ text?: string }> } } - ).body - const text = body?.parts?.[0]?.text - if (text) prompts.push(text) - }, - }, - } as unknown as PluginInput['client'], - }), + createMockPluginInput(), Date.now() + 3600_000, + false, + false, + 'acc-main', ) hooks = loaded.hooks + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'x-session-affinity': 'pre-break-session' }), + ) + await drainSidebarWrites() + const changed = JSON.parse(readFileSync(sidebarFile, 'utf8')) + changed.fallbacks[1].quota = stickyQuota(0, Date.now() + 1_000) + writeFileSync(sidebarFile, JSON.stringify(changed)) - await runCommand(hooks, 'openai-cachekeep', 'on') await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - responseRequestInit({ 'session-id': 'main-session' }), + responseRequestInit({ 'x-session-affinity': 'pre-break-session' }), ) - await runCommand(hooks, 'openai-cachekeep', 'status') - const status = prompts.at(-1) ?? '' - expect(status).toContain('Tracked sessions: **1**') - expect(status).toContain('(fallback-1)') - expect(status).not.toContain('(chatgpt-work-alt)') + expect(seenAuth).toEqual([ + 'Bearer fallback-2-token', + 'Bearer fallback-1-token', + ]) + await drainSidebarWrites() + expect( + normalizeSidebarState(JSON.parse(readFileSync(sidebarFile, 'utf8'))) + .stickyAssignments?.[hashSidebarSessionId('pre-break-session')] + ?.accountId, + ).toBe('fallback-1') } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('keeps capture enabled across loader reconstruction so the new manager can self-arm', async () => { - seedStorage({ access: 'fallback-access-token' }) - const prompts: string[] = [] - const client = { - auth: { set: async () => {} }, - session: { - promptAsync: async (request: unknown) => { - const body = ( - request as { body?: { parts?: Array<{ text?: string }> } } - ).body - const text = body?.parts?.[0]?.text - if (text) prompts.push(text) - }, - }, - } as unknown as PluginInput['client'] + it('sticky-balanced retries a 429 only after its fresh quota headers confirm exhaustion', async () => { + seedStickyBalancedAccounts() + const seenAuth: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = (async (_url: unknown, _init?: unknown) => { + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (!String(url).includes('responses')) + return new Response('{}', { status: 200 }) + const auth = headerValue(init, 'authorization') + seenAuth.push(auth) + if (auth.includes('fallback-2-token')) { + return new Response('{}', { + status: 429, + headers: { + 'x-codex-primary-used-percent': '100', + 'x-codex-primary-window-minutes': '300', + 'x-codex-primary-reset-at': String( + Math.floor((Date.now() + 3600_000) / 1000), + ), + }, + }) + } return new Response('{}', { status: 200 }) }) as unknown as typeof globalThis.fetch - let firstHooks: Hooks | undefined - let secondHooks: Hooks | undefined + let hooks: Hooks | undefined try { - const first = await loadFetchOverride( - createMockPluginInput({ client }), + const loaded = await loadFetchOverride( + createMockPluginInput(), Date.now() + 3600_000, + false, + false, + 'acc-main', + ) + hooks = loaded.hooks + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'x-session-affinity': 'rate-limit-session' }), ) - firstHooks = first.hooks - await runCommand(firstHooks, 'openai-cachekeep', 'on') - const second = await loadFetchOverride( - createMockPluginInput({ client }), + expect(response.status).toBe(200) + expect(seenAuth).toEqual([ + 'Bearer fallback-2-token', + 'Bearer fallback-1-token', + ]) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('sticky-balanced does not replay a successful exhausted response and migrates on the next request', async () => { + seedStickyBalancedAccounts() + const seenAuth: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (!String(url).includes('responses')) + return new Response('{}', { status: 200 }) + const auth = headerValue(init, 'authorization') + seenAuth.push(auth) + if (auth.includes('fallback-2-token')) { + return new Response('served', { + status: 200, + headers: { + 'x-codex-primary-used-percent': '100', + 'x-codex-primary-window-minutes': '300', + 'x-codex-primary-reset-at': String( + Math.floor((Date.now() + 3600_000) / 1000), + ), + }, + }) + } + return new Response('replacement', { status: 200 }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), Date.now() + 3600_000, + false, + false, + 'acc-main', ) - secondHooks = second.hooks - - await second.fetchOverride( + hooks = loaded.hooks + const first = await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - responseRequestInit({ 'session-id': 'main-session' }), + responseRequestInit({ 'x-session-affinity': 'successful-exhaustion' }), ) - await runCommand(secondHooks, 'openai-cachekeep', 'status') - const status = prompts.at(-1) ?? '' - expect(status).toContain('Timer: **armed**') - expect(status).toContain('Tracked sessions: **1**') + expect(first.status).toBe(200) + expect(await first.text()).toBe('served') + expect(seenAuth).toEqual(['Bearer fallback-2-token']) + await drainSidebarWrites() + expect( + normalizeSidebarState( + JSON.parse(readFileSync(sidebarFile, 'utf8')), + ).fallbacks.find((account) => account.id === 'fallback-2')?.quota + ?.primary?.remainingPercent, + ).toBe(0) + + const second = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'x-session-affinity': 'successful-exhaustion' }), + ) + expect(second.status).toBe(200) + expect(await second.text()).toBe('replacement') + expect(seenAuth).toHaveLength(2) + expect(seenAuth[0]).toBe('Bearer fallback-2-token') + expect(seenAuth[1]).not.toBe('Bearer fallback-2-token') + await drainSidebarWrites() + expect( + normalizeSidebarState(JSON.parse(readFileSync(sidebarFile, 'utf8'))) + .stickyAssignments?.[hashSidebarSessionId('successful-exhaustion')] + ?.accountId, + ).not.toBe('fallback-2') } finally { globalThis.fetch = originalFetch - await secondHooks?.dispose?.() - await firstHooks?.dispose?.() + await hooks?.dispose?.() } }) - it('persists cachekeep enabled on and off', async () => { - seedStorage({ access: 'fallback-access-token' }) - const client = { - auth: { set: async () => {} }, - session: { promptAsync: async () => {} }, - } as unknown as PluginInput['client'] + it('sticky-balanced retains a pin after a transient server failure', async () => { + seedStickyBalancedAccounts() + const seenAuth: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('responses')) { + seenAuth.push(headerValue(init, 'authorization')) + return new Response('{}', { status: 500 }) + } + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( - createMockPluginInput({ client }), + createMockPluginInput(), Date.now() + 3600_000, + false, + false, + 'acc-main', ) hooks = loaded.hooks + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'x-session-affinity': 'transient-session' }), + ) - await runCommand(hooks, 'openai-cachekeep', 'on') - expect(JSON.parse(readFileSync(configFile, 'utf8')).cachekeep).toEqual({ - enabled: true, - }) - - await runCommand(hooks, 'openai-cachekeep', 'off') - expect(JSON.parse(readFileSync(configFile, 'utf8')).cachekeep).toEqual({ - enabled: false, - }) + expect(response.status).toBe(500) + expect(seenAuth).toEqual(['Bearer fallback-2-token']) + await drainSidebarWrites() + expect( + normalizeSidebarState(JSON.parse(readFileSync(sidebarFile, 'utf8'))) + .stickyAssignments?.[hashSidebarSessionId('transient-session')] + ?.accountId, + ).toBe('fallback-2') } finally { + globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('resolves cachekeep fallback accounts by storage id or ChatGPT account id', () => { - const accounts: OAuthAccount[] = [ - { - id: 'work-alt', - type: 'oauth', - label: 'Work Alt', - enabled: true, - access: 'fallback-access-token', - refresh: 'fallback-refresh-token', - expires: Date.now() + 3600_000, - accountId: '8c97f046-7e21-409b-9829-0488897e475b', - }, - ] - - expect(findCachekeepFallbackAccount(accounts, 'work-alt')?.id).toBe( - 'work-alt', - ) - expect( - findCachekeepFallbackAccount( - accounts, - '8c97f046-7e21-409b-9829-0488897e475b', - )?.id, - ).toBe('work-alt') - }) - - it('fallback-first attributes served-fallback quota to the fallback and marks it active in the sidebar', async () => { - writeFileSync( - configFile, - JSON.stringify({ - version: 1, - main: { type: 'opencode', provider: 'openai' }, - accounts: [ - { - id: 'work-alt', - type: 'oauth', - label: 'Work Alt', - enabled: true, - access: 'work-alt-token', - refresh: 'work-alt-refresh', - expires: Date.now() + 3600_000 * 24, - accountId: 'chatgpt-work-alt', - }, - ], - refresh: { refreshBeforeExpiryMinutes: 5 }, - // fallback-first: the fallback is tried before main and serves. - routing: { mode: 'fallback-first' }, - }), - ) - + it('sticky-balanced fails open in configured order when every quota is stale and logs the fallback', async () => { + seedStickyBalancedAccounts() + await drainSidebarWrites() + const stale = JSON.parse(readFileSync(sidebarFile, 'utf8')) + stale.main.quota = stickyQuota(100, 0) + for (const fallback of stale.fallbacks) { + fallback.quota = stickyQuota(100, 0) + } + writeFileSync(sidebarFile, JSON.stringify(stale)) + const seenAuth: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = (async (_url: unknown, _init?: unknown) => { - return new Response('{}', { - status: 200, - headers: { - 'content-type': 'application/json', - 'x-codex-primary-used-percent': '63', - 'x-codex-primary-window-minutes': '300', - 'x-codex-primary-reset-at': '1781729038', - }, - }) + const originalLevel = process.env.OPENCODE_OPENAI_AUTH_LOG_LEVEL + process.env.OPENCODE_OPENAI_AUTH_LOG_LEVEL = 'debug' + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('responses')) { + seenAuth.push(headerValue(init, 'authorization')) + } + return new Response('{}', { status: 200 }) }) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined @@ -3573,256 +3799,310 @@ describe('integration: active fallback routing', () => { const loaded = await loadFetchOverride( createMockPluginInput(), Date.now() + 3600_000, + false, + false, + 'acc-main', ) hooks = loaded.hooks - - const response = await loaded.fetchOverride( + await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + responseRequestInit({ 'x-session-affinity': 'stale-session' }), ) - expect(response.status).toBe(200) - await response.body?.cancel() - const sidebar = await waitForSidebarState( - sidebarFile, - (s) => - s.activeId === 'work-alt' && - s.main.quota === null && - s.fallbacks.find((a) => a.id === 'work-alt')?.quota?.primary - ?.usedPercent === 63, + expect(seenAuth).toEqual(['Bearer main-stale-token']) + await flushForTest() + expect(readFileSync(logFile, 'utf8')).toContain( + 'sticky routing: no fresh weighted candidates; using configured order', ) - expect(sidebar.activeId).toBe('work-alt') - expect(sidebar.activeRouting).toBeUndefined() - expect(sidebar.main.quota).toBeNull() - expect( - sidebar.fallbacks.find((a) => a.id === 'work-alt')?.quota?.primary - ?.usedPercent, - ).toBe(63) } finally { + if (originalLevel === undefined) { + delete process.env.OPENCODE_OPENAI_AUTH_LOG_LEVEL + } else { + process.env.OPENCODE_OPENAI_AUTH_LOG_LEVEL = originalLevel + } globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('admission quota skips an exhausted first fallback from the shared sidebar state', async () => { - const now = Date.now() - const reset = new Date(now + 7 * 24 * 3600_000).toISOString() - seedAdmissionAccounts(['work-alt', 'client-alt']) - const seenAuth: string[] = [] + it('sticky-balanced captures cachekeep only for the account that serves', async () => { + seedStickyBalancedAccounts() + const prompts: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = mockAdmissionFetch(seenAuth) + globalThis.fetch = (async () => + new Response('{}', { status: 200 })) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( - createMockPluginInput(), - now + 3600_000, + createMockPluginInput({ + client: { + auth: { set: async () => {} }, + session: { + promptAsync: async (request: unknown) => { + const text = ( + request as { body?: { parts?: Array<{ text?: string }> } } + ).body?.parts?.[0]?.text + if (text) prompts.push(text) + }, + }, + } as unknown as PluginInput['client'], + }), + Date.now() + 3600_000, + false, + false, + 'acc-main', ) hooks = loaded.hooks - writeAdmissionSidebarState({ - fallbackIds: ['work-alt', 'client-alt'], - fallbackQuotas: { - 'work-alt': admissionQuota(100, reset, now), - 'client-alt': admissionQuota(20, reset, now), - }, - fallbackAccountIds: { - 'work-alt': 'chatgpt-work-alt', - 'client-alt': 'chatgpt-client-alt', - }, - activeId: 'work-alt', - }) - - const response = await loaded.fetchOverride( + await runCommand(hooks, 'openai-cachekeep', 'on') + await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + responseRequestInit({ 'x-session-affinity': 'capture-session' }), ) + await runCommand(hooks, 'openai-cachekeep', 'status') - expect(response.status).toBe(200) - expect(seenAuth).toEqual(['Bearer client-alt-token']) + const status = prompts.at(-1) ?? '' + expect(status).toContain('Tracked sessions: **1**') + expect(status).toContain('(fallback-2)') + expect(status).not.toContain('(fallback-1)') + expect(status).not.toContain('(main)') } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('admission quota skips a file-exhausted fallback with an empty process quota cache', async () => { - const now = Date.now() - const reset = new Date(now + 7 * 24 * 3600_000).toISOString() - seedAdmissionAccounts(['work-alt']) - const seenAuth: string[] = [] + it('sticky-balanced does not pin sessionless or non-replayable requests', async () => { + seedStickyBalancedAccounts() const originalFetch = globalThis.fetch - globalThis.fetch = mockAdmissionFetch(seenAuth) + globalThis.fetch = (async () => + new Response('{}', { status: 200 })) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - now + 3600_000, + Date.now() + 3600_000, + false, + false, + 'acc-main', ) hooks = loaded.hooks - writeAdmissionSidebarState({ - fallbackIds: ['work-alt'], - fallbackQuotas: { - 'work-alt': admissionQuota(100, reset, now), - }, - fallbackAccountIds: { 'work-alt': 'chatgpt-work-alt' }, - }) - - const response = await loaded.fetchOverride( + await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + responseRequestInit(), + ) + await loaded.fetchOverride( + 'https://api.openai.com/v1/models', + responseRequestInit({ 'x-session-affinity': 'non-replayable-session' }), ) - expect(response.status).toBe(200) - expect(seenAuth).toEqual(['Bearer main-stale-token']) + await drainSidebarWrites() + const sidebar = normalizeSidebarState( + JSON.parse(readFileSync(sidebarFile, 'utf8')), + ) + expect(sidebar.stickyAssignments).toBeUndefined() } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('admission quota uses fresher healthy memory instead of a stale exhausted file row', async () => { - const now = Date.now() - const reset = new Date(now + 7 * 24 * 3600_000).toISOString() - seedAdmissionAccounts(['work-alt'], 'fallback-first', { - 'work-alt': admissionQuota(20, reset, now), - }) - const seenAuth: string[] = [] + it('sticky-balanced retains its pin and wire response when no replacement exists', async () => { + const checkedAt = Date.now() + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + routing: { mode: 'sticky-balanced' }, + accounts: [], + }), + ) + writeFileSync( + sidebarFile, + JSON.stringify({ + main: { + quota: stickyQuota(100, checkedAt), + mainAccountId: 'acc-main', + killed: false, + }, + fallbacks: [], + route: 'sticky-balanced', + lastUpdated: checkedAt, + }), + ) const originalFetch = globalThis.fetch - globalThis.fetch = mockAdmissionFetch(seenAuth) + globalThis.fetch = (async (url: unknown) => + new Response('{}', { + status: String(url).includes('responses') ? 401 : 200, + })) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - now + 3600_000, + Date.now() + 3600_000, + false, + false, + 'acc-main', ) hooks = loaded.hooks - writeAdmissionSidebarState({ - fallbackIds: ['work-alt'], - fallbackQuotas: { - 'work-alt': admissionQuota(100, reset, now - 60_000), - }, - }) - - await loaded.fetchOverride( + const response = await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + responseRequestInit({ 'x-session-affinity': 'main-only-session' }), ) - expect(seenAuth).toEqual(['Bearer work-alt-token']) + expect(response.status).toBe(401) + await drainSidebarWrites() + expect( + normalizeSidebarState(JSON.parse(readFileSync(sidebarFile, 'utf8'))) + .stickyAssignments?.[hashSidebarSessionId('main-only-session')] + ?.accountId, + ).toBe('main') } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('admission quota uses a fresher exhausted file row instead of stale healthy memory', async () => { - const now = Date.now() - const reset = new Date(now + 7 * 24 * 3600_000).toISOString() - seedAdmissionAccounts(['work-alt', 'client-alt'], 'fallback-first', { - 'work-alt': admissionQuota(20, reset, now - 60_000), - }) - const seenAuth: string[] = [] + it('sticky-balanced preserves the parent display pin instead of mirroring a child account', async () => { + seedStickyBalancedAccounts() const originalFetch = globalThis.fetch - globalThis.fetch = mockAdmissionFetch(seenAuth) + globalThis.fetch = (async () => + new Response('{}', { status: 200 })) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - now + 3600_000, + Date.now() + 3600_000, + false, + false, + 'acc-main', ) hooks = loaded.hooks - writeAdmissionSidebarState({ - fallbackIds: ['work-alt', 'client-alt'], - fallbackQuotas: { - 'work-alt': admissionQuota(100, reset, now), - 'client-alt': admissionQuota(20, reset, now), - }, - fallbackAccountIds: { - 'work-alt': 'chatgpt-work-alt', - 'client-alt': 'chatgpt-client-alt', - }, - }) - await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + responseRequestInit({ 'x-session-affinity': 'parent-session' }), + ) + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ + 'x-session-affinity': 'child-session', + 'x-parent-session-id': 'parent-session', + }), ) + await drainSidebarWrites() - expect(seenAuth).toEqual(['Bearer client-alt-token']) + const sidebar = normalizeSidebarState( + JSON.parse(readFileSync(sidebarFile, 'utf8')), + ) + expect(sidebar.activeRouting?.['parent-session']?.activeId).toBe( + sidebar.stickyAssignments?.[hashSidebarSessionId('parent-session')] + ?.accountId, + ) + expect(sidebar.activeRouting?.['parent-session']?.activeId).not.toBe( + sidebar.activeRouting?.['child-session']?.activeId, + ) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('admission quota skips from fresher exhausted memory instead of a stale healthy file row', async () => { - const now = Date.now() - const reset = new Date(now + 7 * 24 * 3600_000).toISOString() - seedAdmissionAccounts(['work-alt', 'client-alt'], 'fallback-first', { - 'work-alt': admissionQuota(100, reset, now), - }) - const seenAuth: string[] = [] + it('sticky-balanced does not overwrite a parent display that has no pin', async () => { + seedStickyBalancedAccounts() + await drainSidebarWrites() + const seeded = JSON.parse(readFileSync(sidebarFile, 'utf8')) + seeded.activeRouting = { + 'parent-no-pin': { + activeId: 'main', + route: 'sticky-balanced', + updatedAt: Date.now(), + }, + } + writeFileSync(sidebarFile, JSON.stringify(seeded)) const originalFetch = globalThis.fetch - globalThis.fetch = mockAdmissionFetch(seenAuth) + globalThis.fetch = (async () => + new Response('{}', { status: 200 })) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - now + 3600_000, + Date.now() + 3600_000, + false, + false, + 'acc-main', ) hooks = loaded.hooks - writeAdmissionSidebarState({ - fallbackIds: ['work-alt', 'client-alt'], - fallbackQuotas: { - 'work-alt': admissionQuota(20, reset, now - 60_000), - 'client-alt': admissionQuota(20, reset, now), - }, - }) - await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + responseRequestInit({ + 'x-session-affinity': 'child-no-pin', + 'x-parent-session-id': 'parent-no-pin', + }), ) + await drainSidebarWrites() - expect(seenAuth).toEqual(['Bearer client-alt-token']) + const sidebar = normalizeSidebarState( + JSON.parse(readFileSync(sidebarFile, 'utf8')), + ) + expect(sidebar.activeRouting?.['child-no-pin']?.activeId).toBe( + 'fallback-2', + ) + expect(sidebar.activeRouting?.['parent-no-pin']).toMatchObject({ + activeId: 'main', + route: 'sticky-balanced', + }) + expect( + sidebar.stickyAssignments?.[hashSidebarSessionId('parent-no-pin')], + ).toBeUndefined() } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) + function mockAdmissionFetch(seenAuth: string[], status = 200) { + return (async (url: unknown, init?: unknown) => { + if (String(url).includes('responses')) { + seenAuth.push(headerValue(init, 'authorization')) + } + return new Response('{}', { status }) + }) as unknown as typeof globalThis.fetch + } + test.each([ - ['missing quota', null], - ['missing reset', { primary: { usedPercent: 100, remainingPercent: 0 } }], [ - 'malformed reset', { - primary: { - usedPercent: 100, - remainingPercent: 0, - resetsAt: 'not-a-date', - }, + 'x-session-affinity': 'affinity', + 'x-opencode-session': 'opencode', + 'x-session-id': 'x-session', + 'session-id': 'session', }, + 'affinity', ], [ - 'malformed usage', - { - primary: { - usedPercent: '100', - remainingPercent: 0, - resetsAt: new Date(Date.now() + 3600_000).toISOString(), - }, - }, + { 'x-opencode-session': 'opencode', 'x-session-id': 'x-session' }, + 'opencode', ], - ])('admission quota retains a fallback with %s', async (_label, quota) => { - seedAdmissionAccounts(['work-alt']) - const seenAuth: string[] = [] + [{ 'x-session-id': 'x-session', 'session-id': 'session' }, 'x-session'], + [{ 'session-id': 'session' }, 'session'], + [{}, undefined], + ])( + 'resolves sidebar session headers by documented precedence', + (raw, expected) => { + expect(resolveSidebarSessionId(new Headers(raw))).toBe(expected) + }, + ) + + it('keeps different served accounts under different session keys', async () => { + seedStorage({ access: 'fallback-access-token' }) const originalFetch = globalThis.fetch - globalThis.fetch = mockAdmissionFetch(seenAuth) + globalThis.fetch = (async () => + new Response('{}', { status: 200 })) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { @@ -3831,216 +4111,235 @@ describe('integration: active fallback routing', () => { Date.now() + 3600_000, ) hooks = loaded.hooks - writeAdmissionSidebarState({ - fallbackIds: ['work-alt'], - fallbackQuotas: { - 'work-alt': quota as SidebarState['main']['quota'], - }, - }) await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + responseRequestInit({ 'x-opencode-session': 'sess-fallback' }), ) - expect(seenAuth).toEqual(['Bearer work-alt-token']) + seedStorage({ access: 'fallback-access-token' }, { mode: 'main-first' }) + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'x-opencode-session': 'sess-main' }), + ) + await drainSidebarWrites() + + const sidebar = normalizeSidebarState( + JSON.parse(readFileSync(sidebarFile, 'utf8')), + ) + expect(sidebar.activeRouting?.['sess-fallback']).toMatchObject({ + activeId: 'fallback-1', + route: 'fallback-first', + }) + expect(sidebar.activeRouting?.['sess-main']).toMatchObject({ + activeId: 'main', + route: 'main-first', + }) + expect(sidebar.activeId).toBe('main') + expect(sidebar.route).toBe('main-first') } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('admission quota preserves probe order when every account is exhausted', async () => { - const now = Date.now() - const reset = new Date(now + 7 * 24 * 3600_000).toISOString() - seedAdmissionAccounts(['work-alt', 'client-alt']) - const seenAuth: string[] = [] + it('records fallback-served routing on the parent session', async () => { + seedStorage({ access: 'fallback-access-token' }) const originalFetch = globalThis.fetch - globalThis.fetch = mockAdmissionFetch(seenAuth, 429) + globalThis.fetch = (async () => + new Response('{}', { status: 200 })) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - now + 3600_000, + Date.now() + 3600_000, ) hooks = loaded.hooks - writeAdmissionSidebarState({ - fallbackIds: ['work-alt', 'client-alt'], - fallbackQuotas: { - 'work-alt': admissionQuota(100, reset, now), - 'client-alt': admissionQuota(100, reset, now), - }, - fallbackAccountIds: { - 'work-alt': 'chatgpt-work-alt', - 'client-alt': 'chatgpt-client-alt', - }, - mainQuota: admissionQuota(100, reset, now), - }) - const response = await loaded.fetchOverride( + await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + responseRequestInit({ + 'x-opencode-session': 'child-session', + 'x-parent-session-id': 'parent-session', + }), ) + await drainSidebarWrites() - expect(response.status).toBe(429) - expect(seenAuth).toEqual([ - 'Bearer work-alt-token', - 'Bearer client-alt-token', - 'Bearer main-stale-token', - ]) + const sidebar = normalizeSidebarState( + JSON.parse(readFileSync(sidebarFile, 'utf8')), + ) + expect(sidebar.activeRouting?.['child-session']).toMatchObject({ + activeId: 'fallback-1', + route: 'fallback-first', + }) + // A fallback (not main) served the child; the parent entry must mirror + // that same fallback so the parent's sidebar highlights the live account. + expect(sidebar.activeRouting?.['parent-session']).toMatchObject({ + activeId: 'fallback-1', + route: 'fallback-first', + }) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('admission quota reroutes a file-exhausted main without probing it', async () => { - const now = Date.now() - const reset = new Date(now + 7 * 24 * 3600_000).toISOString() - seedAdmissionAccounts(['client-alt'], 'main-first') - const seenAuth: string[] = [] + it('uses legacy display routing when the request carries no session headers', async () => { + seedStorage({ access: 'fallback-access-token' }) const originalFetch = globalThis.fetch - globalThis.fetch = mockAdmissionFetch(seenAuth) + globalThis.fetch = (async () => + new Response('{}', { status: 200 })) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - now + 3600_000, + Date.now() + 3600_000, ) hooks = loaded.hooks - await new Promise((resolve) => setTimeout(resolve, 2)) - const checkedAt = Date.now() - writeAdmissionSidebarState({ - fallbackIds: ['client-alt'], - fallbackQuotas: { - 'client-alt': admissionQuota(20, reset, checkedAt), - }, - mainQuota: admissionQuota(100, reset, checkedAt), - route: 'main-first', - }) - const response = await loaded.fetchOverride( + await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + responseRequestInit(), ) + await drainSidebarWrites() - expect(response.status).toBe(200) - expect(seenAuth).toEqual(['Bearer client-alt-token']) + const sidebar = normalizeSidebarState( + JSON.parse(readFileSync(sidebarFile, 'utf8')), + ) + // Sessionless requests write only the legacy display fields, never a + // per-session entry. + expect(sidebar.activeRouting).toBeUndefined() + expect(sidebar.activeId).toBe('fallback-1') + expect(sidebar.route).toBe('fallback-first') + // Resolving without a session reads those legacy fields and must yield + // defined routing rather than crash or return undefined. + expect(resolveSessionSidebarRouting(sidebar, undefined)).toEqual({ + activeId: 'fallback-1', + route: 'fallback-first', + }) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('admission quota ignores an exhausted main row from a different account', async () => { - const now = Date.now() - const reset = new Date(now + 7 * 24 * 3600_000).toISOString() - seedAdmissionAccounts(['client-alt'], 'main-first') - const seenAuth: string[] = [] + it('reads the sidebar session from a Request and strips it before the wire', async () => { + seedEmptyAccountStorage() const originalFetch = globalThis.fetch - globalThis.fetch = mockAdmissionFetch(seenAuth) + let wireHeaders = new Headers() + let wireMethod: string | undefined + let wireBody: BodyInit | null | undefined + globalThis.fetch = (async (_url: unknown, init?: RequestInit) => { + wireHeaders = new Headers(init?.headers) + wireMethod = init?.method + wireBody = init?.body + return new Response('{}', { status: 200 }) + }) as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - now + 3600_000, - false, - false, - 'new-account', + Date.now() + 3600_000, ) hooks = loaded.hooks - await new Promise((resolve) => setTimeout(resolve, 2)) - const checkedAt = Date.now() - writeAdmissionSidebarState({ - fallbackIds: ['client-alt'], - fallbackQuotas: { - 'client-alt': admissionQuota(20, reset, checkedAt), + const requestBody = responseRequestInit() + const request = new Request('https://api.openai.com/v1/responses', { + method: 'POST', + headers: { + 'content-type': 'application/json', + 'x-opencode-session': 'request-session', }, - mainQuota: admissionQuota(100, reset, checkedAt), - mainAccountId: 'old-account', - route: 'main-first', + body: requestBody.body, }) + await loaded.fetchOverride(request) + await drainSidebarWrites() - const response = await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - requestInit(), + const sidebar = normalizeSidebarState( + JSON.parse(readFileSync(sidebarFile, 'utf8')), ) - - expect(response.status).toBe(200) - expect(seenAuth).toEqual(['Bearer main-stale-token']) - } finally { + expect(sidebar.activeRouting?.['request-session']).toMatchObject({ + activeId: 'main', + route: 'main-first', + }) + expect(wireHeaders.has('x-opencode-session')).toBe(false) + expect(wireMethod).toBe('POST') + expect(JSON.parse(String(wireBody)).model).toBe('gpt-5.5') + } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('admission quota ignores an exhausted fallback row stamped with a different account identity', async () => { - const now = Date.now() - const reset = new Date(now + 7 * 24 * 3600_000).toISOString() - // Live account identity is chatgpt-work-alt (see seedAdmissionAccounts). - seedAdmissionAccounts(['work-alt']) - const seenAuth: string[] = [] + it('does not mutate frozen caller-owned request headers', async () => { + seedEmptyAccountStorage() + const callerHeaders = Object.freeze({ + authorization: 'Bearer caller-token', + 'content-type': 'application/json', + 'x-api-key': 'caller-key', + }) + const callerInit = Object.freeze({ + method: 'POST', + headers: callerHeaders, + body: responseRequestInit().body, + }) as RequestInit const originalFetch = globalThis.fetch - globalThis.fetch = mockAdmissionFetch(seenAuth) + let wireHeaders = new Headers() + globalThis.fetch = (async (_url: unknown, init?: RequestInit) => { + wireHeaders = new Headers(init?.headers) + return new Response('{}', { status: 200 }) + }) as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - now + 3600_000, + Date.now() + 3600_000, ) hooks = loaded.hooks - writeAdmissionSidebarState({ - fallbackIds: ['work-alt'], - fallbackQuotas: { - 'work-alt': admissionQuota(100, reset, now), - }, - // The file row belongs to a previous login of this stable id; the live - // account is a different ChatGPT identity, so the exhausted row must be - // treated as absent (fail-open) rather than blocking the replacement. - fallbackAccountIds: { 'work-alt': 'chatgpt-stale' }, - }) - - const response = await loaded.fetchOverride( + await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + callerInit, ) - expect(response.status).toBe(200) - expect(seenAuth).toEqual(['Bearer work-alt-token']) + expect(callerHeaders.authorization).toBe('Bearer caller-token') + expect(callerHeaders['x-api-key']).toBe('caller-key') + expect(wireHeaders.get('authorization')).toBe('Bearer main-stale-token') + expect(wireHeaders.has('x-api-key')).toBe(false) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('admission quota honors an exhausted fallback row matching the live account identity', async () => { - const now = Date.now() - const reset = new Date(now + 7 * 24 * 3600_000).toISOString() - seedAdmissionAccounts(['work-alt']) - const seenAuth: string[] = [] + it('uses the active fallback token without writing it to the auth slot', async () => { + seedStorage({ access: 'fallback-access-token' }) + const authSetCalls: unknown[] = [] + const seen: Array<{ authorization: string; accountId: string | null }> = [] const originalFetch = globalThis.fetch - globalThis.fetch = mockAdmissionFetch(seenAuth) + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('/oauth/token')) { + throw new Error('refresh unavailable') + } + seen.push({ + authorization: headerValue(init, 'authorization'), + accountId: headerValue(init, 'ChatGPT-Account-Id') || null, + }) + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { - const loaded = await loadFetchOverride( - createMockPluginInput(), - now + 3600_000, - ) - hooks = loaded.hooks - writeAdmissionSidebarState({ - fallbackIds: ['work-alt'], - fallbackQuotas: { - 'work-alt': admissionQuota(100, reset, now), - }, - // Identity matches the live account, so the exhausted row is honored - // and the fallback is skipped in favor of main. - fallbackAccountIds: { 'work-alt': 'chatgpt-work-alt' }, + const input = createMockPluginInput({ + client: { + auth: { set: async (payload: unknown) => authSetCalls.push(payload) }, + session: { promptAsync: async () => {} }, + } as unknown as PluginInput['client'], }) + const loaded = await loadFetchOverride(input, Date.now() + 3600_000) + hooks = loaded.hooks const response = await loaded.fetchOverride( 'https://api.openai.com/v1/responses', @@ -4048,280 +4347,477 @@ describe('integration: active fallback routing', () => { ) expect(response.status).toBe(200) - expect(seenAuth).toEqual(['Bearer main-stale-token']) + expect(seen).toEqual([ + { + authorization: 'Bearer fallback-access-token', + accountId: 'acc-fallback-1', + }, + ]) + expect(authSetCalls).toEqual([]) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('admission quota skips a fallback exhausted only on its secondary window', async () => { - const now = Date.now() - const reset = new Date(now + 7 * 24 * 3600_000).toISOString() - seedAdmissionAccounts(['work-alt']) - const seenAuth: string[] = [] + it('waits on a held main refresh file lock and uses the rotated auth token', async () => { + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [], + refresh: { refreshBeforeExpiryMinutes: 5 }, + }), + ) const originalFetch = globalThis.fetch - globalThis.fetch = mockAdmissionFetch(seenAuth) + const seen: string[] = [] + let oauthRefreshCalls = 0 + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('/oauth/token')) { + oauthRefreshCalls++ + throw new Error('second process must not refresh') + } + seen.push(headerValue(init, 'authorization')) + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch + + const heldLock = await acquireRefreshFileLock({ + name: 'main-refresh', + ttlMs: 60_000, + path: configFile, + renew: true, + }) + if (!heldLock) throw new Error('failed to acquire test lock') + + let auth = { + type: 'oauth' as const, + provider: 'openai', + access: 'main-stale-token', + refresh: 'main-refresh-token', + expires: Date.now() - 1_000, + } + setTimeout(() => { + auth = { + type: 'oauth' as const, + provider: 'openai', + access: 'main-rotated-token', + refresh: 'main-rotated-refresh', + expires: Date.now() + 3600_000, + } + }, 25) let hooks: Hooks | undefined try { - const loaded = await loadFetchOverride( - createMockPluginInput(), - now + 3600_000, - ) - hooks = loaded.hooks - writeAdmissionSidebarState({ - fallbackIds: ['work-alt'], - fallbackQuotas: { - // No primary window; the secondary window alone is exhausted. - 'work-alt': { - secondary: { - usedPercent: 100, - remainingPercent: 0, - resetsAt: reset, - checkedAt: now, - windowMinutes: 10_080, - }, - }, - }, - fallbackAccountIds: { 'work-alt': 'chatgpt-work-alt' }, - }) + const input = createMockPluginInput() + hooks = await CodexAuthPlugin(input, { experimentalWebSockets: false }) + const authHook = hooks.auth + if (!authHook?.loader) throw new Error('No auth loader') + const loaderResult = await authHook.loader(async () => auth, { + id: 'openai', + label: 'OpenAI', + models: [], + } as unknown as Parameters>[1]) + const fetchOverride = (loaderResult as Record).fetch as + | ((url: RequestInfo | URL, init?: RequestInit) => Promise) + | undefined + if (!fetchOverride) throw new Error('No fetch in loader result') - const response = await loaded.fetchOverride( + const response = await fetchOverride( 'https://api.openai.com/v1/responses', requestInit(), ) expect(response.status).toBe(200) - expect(seenAuth).toEqual(['Bearer main-stale-token']) + expect(oauthRefreshCalls).toBe(0) + expect(seen).toEqual(['Bearer main-rotated-token']) } finally { + await heldLock.release() globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('admission quota retains an exhausted-looking fallback after its reset passes', async () => { - const now = Date.now() - seedAdmissionAccounts(['work-alt']) - const seenAuth: string[] = [] + it('deduplicates concurrent in-process main refreshes and releases the lock on success', async () => { + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [], + refresh: { refreshBeforeExpiryMinutes: 5 }, + }), + ) const originalFetch = globalThis.fetch - globalThis.fetch = mockAdmissionFetch(seenAuth) + const seen: string[] = [] + const authSetCalls: unknown[] = [] + let oauthRefreshCalls = 0 + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('/oauth/token')) { + oauthRefreshCalls++ + await new Promise((resolve) => setTimeout(resolve, 25)) + return new Response( + JSON.stringify({ + access_token: 'main-fresh-token', + refresh_token: 'main-fresh-refresh', + expires_in: 3600, + id_token: 'id', + }), + { status: 200, headers: { 'content-type': 'application/json' } }, + ) + } + seen.push(headerValue(init, 'authorization')) + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch + let auth = { + type: 'oauth' as const, + provider: 'openai', + access: 'main-stale-token', + refresh: 'main-refresh-token', + expires: Date.now() - 1_000, + } let hooks: Hooks | undefined try { - const loaded = await loadFetchOverride( - createMockPluginInput(), - now + 3600_000, - ) - hooks = loaded.hooks - writeAdmissionSidebarState({ - fallbackIds: ['work-alt'], - fallbackQuotas: { - 'work-alt': admissionQuota( - 100, - new Date(now - 60_000).toISOString(), - now, - ), - }, - }) - - await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - requestInit(), - ) + const input = createMockPluginInput({ + client: { + auth: { + set: async (payload: unknown) => { + authSetCalls.push(payload) + const body = (payload as { body: typeof auth }).body + auth = { ...auth, ...body } + }, + }, + session: { promptAsync: async () => {} }, + } as unknown as PluginInput['client'], + }) + hooks = await CodexAuthPlugin(input, { experimentalWebSockets: false }) + const authHook = hooks.auth + if (!authHook?.loader) throw new Error('No auth loader') + const loaderResult = await authHook.loader(async () => auth, { + id: 'openai', + label: 'OpenAI', + models: [], + } as unknown as Parameters>[1]) + const fetchOverride = (loaderResult as Record).fetch as + | ((url: RequestInfo | URL, init?: RequestInit) => Promise) + | undefined + if (!fetchOverride) throw new Error('No fetch in loader result') - expect(seenAuth).toEqual(['Bearer work-alt-token']) + const [first, second] = await Promise.all([ + fetchOverride('https://api.openai.com/v1/responses', requestInit()), + fetchOverride('https://api.openai.com/v1/responses', requestInit()), + ]) + + expect(first.status).toBe(200) + expect(second.status).toBe(200) + expect(oauthRefreshCalls).toBe(1) + expect(authSetCalls).toHaveLength(1) + expect(seen).toEqual([ + 'Bearer main-fresh-token', + 'Bearer main-fresh-token', + ]) + + const releasedLock = await acquireRefreshFileLock({ + name: 'main-refresh', + ttlMs: 60_000, + path: configFile, + }) + expect(releasedLock).not.toBeNull() + await releasedLock?.release() } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('admission quota ignores an unstamped (no accountId) exhausted fallback row against a known identity', async () => { - const now = Date.now() - const reset = new Date(now + 7 * 24 * 3600_000).toISOString() - // Live account identity is chatgpt-work-alt (see seedAdmissionAccounts). - seedAdmissionAccounts(['work-alt']) - const seenAuth: string[] = [] + it('uses a stale main token on refresh failure and releases the lock', async () => { + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [], + refresh: { refreshBeforeExpiryMinutes: 5 }, + }), + ) const originalFetch = globalThis.fetch - globalThis.fetch = mockAdmissionFetch(seenAuth) + const seen: string[] = [] + let oauthRefreshCalls = 0 + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('/oauth/token')) { + oauthRefreshCalls++ + return new Response('bad refresh', { status: 500 }) + } + seen.push(headerValue(init, 'authorization')) + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - now + 3600_000, + Date.now() - 1_000, ) hooks = loaded.hooks - await new Promise((resolve) => setTimeout(resolve, 2)) - const checkedAt = Date.now() - writeAdmissionSidebarState({ - fallbackIds: ['work-alt'], - fallbackQuotas: { - 'work-alt': admissionQuota(100, reset, checkedAt), - }, - // No fallbackAccountIds — the file row carries no accountId stamp. - // The live identity is known (chatgpt-work-alt), so this exhausted - // unstamped row must not be trusted: the fallback is still probed. - }) const response = await loaded.fetchOverride( 'https://api.openai.com/v1/responses', requestInit(), ) - // Fallback is probed — not skipped based on an unstamped file row. expect(response.status).toBe(200) - expect(seenAuth).toEqual(['Bearer work-alt-token']) + expect(oauthRefreshCalls).toBe(1) + expect(seen).toEqual(['Bearer main-stale-token']) + + const releasedLock = await acquireRefreshFileLock({ + name: 'main-refresh', + ttlMs: 60_000, + path: configFile, + }) + expect(releasedLock).not.toBeNull() + await releasedLock?.release() } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('admission quota probes main for a non-replayable request even when the file says exhausted and a fallback is retained', async () => { - const now = Date.now() - const reset = new Date(now + 7 * 24 * 3600_000).toISOString() - seedAdmissionAccounts(['work-alt'], 'main-first') - const seenAuth: string[] = [] + it('captures cachekeep fallback targets by storage id, not ChatGPT account id', async () => { + seedStorage({ + access: 'fallback-access-token', + accountId: 'chatgpt-work-alt', + }) + const prompts: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = mockAdmissionFetch(seenAuth) + globalThis.fetch = (async (_url: unknown, _init?: unknown) => { + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( - createMockPluginInput(), - now + 3600_000, + createMockPluginInput({ + client: { + auth: { set: async () => {} }, + session: { + promptAsync: async (request: unknown) => { + const body = ( + request as { body?: { parts?: Array<{ text?: string }> } } + ).body + const text = body?.parts?.[0]?.text + if (text) prompts.push(text) + }, + }, + } as unknown as PluginInput['client'], + }), + Date.now() + 3600_000, ) hooks = loaded.hooks - await new Promise((resolve) => setTimeout(resolve, 2)) - const checkedAt = Date.now() - writeAdmissionSidebarState({ - fallbackIds: ['work-alt'], - fallbackQuotas: { - 'work-alt': admissionQuota(20, reset, checkedAt), - }, - mainQuota: admissionQuota(100, reset, checkedAt), - route: 'main-first', - }) - // A non-replayable GET request: main is file-exhausted and a healthy - // fallback is retained, but without the replayability guard the - // quotaBlocksMain check would produce a synthetic 429 without ever - // probing main. With the fix, main IS probed. - const response = await loaded.fetchOverride( + await runCommand(hooks, 'openai-cachekeep', 'on') + await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - { method: 'GET', headers: { 'content-type': 'application/json' } }, + responseRequestInit({ 'session-id': 'main-session' }), ) + await runCommand(hooks, 'openai-cachekeep', 'status') - // Main was probed — not skipped by a synthetic 429. - expect(response.status).toBe(200) - expect(seenAuth).toEqual(['Bearer main-stale-token']) + const status = prompts.at(-1) ?? '' + expect(status).toContain('Tracked sessions: **1**') + expect(status).toContain('(fallback-1)') + expect(status).not.toContain('(chatgpt-work-alt)') } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('refreshes an expired active fallback without writing the auth slot', async () => { - seedStorage({ - access: 'fallback-stale-token', - expires: Date.now() - 60_000, - }) - const authSetCalls: unknown[] = [] - const seenAuth: string[] = [] + it('keeps capture enabled across loader reconstruction so the new manager can self-arm', async () => { + seedStorage({ access: 'fallback-access-token' }) + const prompts: string[] = [] + const client = { + auth: { set: async () => {} }, + session: { + promptAsync: async (request: unknown) => { + const body = ( + request as { body?: { parts?: Array<{ text?: string }> } } + ).body + const text = body?.parts?.[0]?.text + if (text) prompts.push(text) + }, + }, + } as unknown as PluginInput['client'] const originalFetch = globalThis.fetch - globalThis.fetch = (async (url: unknown, init?: unknown) => { - if (String(url).includes('/oauth/token')) { - return new Response( - JSON.stringify({ - access_token: 'fallback-refreshed-token', - refresh_token: 'fallback-refresh-new', - expires_in: 3600, - }), - { status: 200, headers: { 'content-type': 'application/json' } }, - ) - } - seenAuth.push(headerValue(init, 'authorization')) + globalThis.fetch = (async (_url: unknown, _init?: unknown) => { return new Response('{}', { status: 200 }) }) as unknown as typeof globalThis.fetch - let hooks: Hooks | undefined + let firstHooks: Hooks | undefined + let secondHooks: Hooks | undefined try { - const input = createMockPluginInput({ - client: { - auth: { set: async (payload: unknown) => authSetCalls.push(payload) }, - session: { promptAsync: async () => {} }, - } as unknown as PluginInput['client'], - }) - const loaded = await loadFetchOverride(input, Date.now() + 3600_000) - hooks = loaded.hooks + const first = await loadFetchOverride( + createMockPluginInput({ client }), + Date.now() + 3600_000, + ) + firstHooks = first.hooks + await runCommand(firstHooks, 'openai-cachekeep', 'on') - const response = await loaded.fetchOverride( + const second = await loadFetchOverride( + createMockPluginInput({ client }), + Date.now() + 3600_000, + ) + secondHooks = second.hooks + + await second.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + responseRequestInit({ 'session-id': 'main-session' }), ) + await runCommand(secondHooks, 'openai-cachekeep', 'status') - expect(response.status).toBe(200) - expect(seenAuth).toEqual(['Bearer fallback-refreshed-token']) - expect(authSetCalls).toEqual([]) + const status = prompts.at(-1) ?? '' + expect(status).toContain('Timer: **armed**') + expect(status).toContain('Tracked sessions: **1**') } finally { globalThis.fetch = originalFetch - await hooks?.dispose?.() + await secondHooks?.dispose?.() + await firstHooks?.dispose?.() } }) - it('fallback-first uses a still-valid fallback token when its refresh fails', async () => { - // Token is inside the refresh window (needs refresh) but NOT expired, so a - // failed refresh must not drop it — the still-valid token is used. - seedStorage({ - access: 'fallback-stale-token', - expires: Date.now() + 2 * 60_000, - }) - const seenAuth: string[] = [] - const originalFetch = globalThis.fetch - globalThis.fetch = (async (url: unknown, init?: unknown) => { - if (String(url).includes('/oauth/token')) { - throw new Error('refresh unavailable') - } - seenAuth.push(headerValue(init, 'authorization')) - return new Response('{}', { status: 200 }) - }) as unknown as typeof globalThis.fetch + it('persists cachekeep enabled on and off', async () => { + seedStorage({ access: 'fallback-access-token' }) + const client = { + auth: { set: async () => {} }, + session: { promptAsync: async () => {} }, + } as unknown as PluginInput['client'] let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( - createMockPluginInput(), + createMockPluginInput({ client }), Date.now() + 3600_000, ) hooks = loaded.hooks - const response = await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - requestInit(), + await runCommand(hooks, 'openai-cachekeep', 'on') + expect(JSON.parse(readFileSync(configFile, 'utf8')).cachekeep).toEqual({ + enabled: true, + }) + + await runCommand(hooks, 'openai-cachekeep', 'off') + expect(JSON.parse(readFileSync(configFile, 'utf8')).cachekeep).toEqual({ + enabled: false, + }) + } finally { + await hooks?.dispose?.() + } + }) + + it('sustain command flips the loader live gate and bypasses main idle pruning', async () => { + seedEmptyAccountStorage() + const originalFetch = globalThis.fetch + const originalNow = Date.now + let now = originalNow() + let hooks: Hooks | undefined + Date.now = () => now + try { + globalThis.fetch = (async () => + new Response('{}', { + status: 200, + })) as unknown as typeof globalThis.fetch + const loaded = await loadFetchOverride( + createMockPluginInput(), + now + 3600_000, ) - expect(response.status).toBe(200) - expect(seenAuth).toEqual(['Bearer fallback-stale-token']) + hooks = loaded.hooks + + await runCommand(hooks, 'openai-cachekeep', 'on') + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'session-id': 'main-session' }), + ) + + now += 60 * 60_000 + 1 + await runCommand(hooks, 'openai-cachekeep', 'sustain on') + const manager = ( + globalThis as typeof globalThis & { + __openaiAuthCacheKeepManager?: { + tick(): Promise + status(): { tracked: number; sustain: boolean } + } + } + ).__openaiAuthCacheKeepManager + if (!manager) throw new Error('missing cachekeep manager') + + await manager.tick() + expect(manager.status()).toMatchObject({ tracked: 1, sustain: true }) } finally { + Date.now = originalNow globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('fallback-first does not re-try fallbacks reactively after main also fails (no double-spend)', async () => { - // fallback-first with one fallback that 429s: the proactive gate tries it, - // falls through to main, and main also 429s. The reactive path must NOT - // re-try the already-tried fallback — so the fallback is hit exactly once. - seedStorage({ access: 'fallback-access-token' }) - const seenAuth: string[] = [] + it('resolves cachekeep fallback accounts by storage id or ChatGPT account id', () => { + const accounts: OAuthAccount[] = [ + { + id: 'work-alt', + type: 'oauth', + label: 'Work Alt', + enabled: true, + access: 'fallback-access-token', + refresh: 'fallback-refresh-token', + expires: Date.now() + 3600_000, + accountId: '8c97f046-7e21-409b-9829-0488897e475b', + }, + ] + + expect(findCachekeepFallbackAccount(accounts, 'work-alt')?.id).toBe( + 'work-alt', + ) + expect( + findCachekeepFallbackAccount( + accounts, + '8c97f046-7e21-409b-9829-0488897e475b', + )?.id, + ).toBe('work-alt') + }) + + it('fallback-first attributes served-fallback quota to the fallback and marks it active in the sidebar', async () => { + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [ + { + id: 'work-alt', + type: 'oauth', + label: 'Work Alt', + enabled: true, + access: 'work-alt-token', + refresh: 'work-alt-refresh', + expires: Date.now() + 3600_000 * 24, + accountId: 'chatgpt-work-alt', + }, + ], + refresh: { refreshBeforeExpiryMinutes: 5 }, + // fallback-first: the fallback is tried before main and serves. + routing: { mode: 'fallback-first' }, + }), + ) + const originalFetch = globalThis.fetch - globalThis.fetch = (async (url: unknown, init?: unknown) => { - if (String(url).includes('/oauth/token')) { - throw new Error('refresh unavailable') - } - seenAuth.push(headerValue(init, 'authorization')) - // Everything is rate-limited. - return new Response('{}', { status: 429 }) + globalThis.fetch = (async (_url: unknown, _init?: unknown) => { + return new Response('{}', { + status: 200, + headers: { + 'content-type': 'application/json', + 'x-codex-primary-used-percent': '63', + 'x-codex-primary-window-minutes': '300', + 'x-codex-primary-reset-at': '1781729038', + }, + }) }) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined @@ -4336,359 +4832,314 @@ describe('integration: active fallback routing', () => { 'https://api.openai.com/v1/responses', requestInit(), ) - expect(response.status).toBe(429) + expect(response.status).toBe(200) await response.body?.cancel() - // Fallback tried once (proactive), then main once — no reactive re-try. - expect(seenAuth).toEqual([ - 'Bearer fallback-access-token', - 'Bearer main-stale-token', - ]) + + const sidebar = await waitForSidebarState( + sidebarFile, + (s) => + s.activeId === 'work-alt' && + s.main.quota === null && + s.fallbacks.find((a) => a.id === 'work-alt')?.quota?.primary + ?.usedPercent === 63, + ) + expect(sidebar.activeId).toBe('work-alt') + expect(sidebar.activeRouting).toBeUndefined() + expect(sidebar.main.quota).toBeNull() + expect( + sidebar.fallbacks.find((a) => a.id === 'work-alt')?.quota?.primary + ?.usedPercent, + ).toBe(63) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('fallback-first propagates a transport error without replaying on main', async () => { - // The fallback send may already have generated or billed before the - // transport error surfaced, so routing must stop instead of replaying. - seedStorage({ access: 'fallback-access-token' }) + it('admission quota skips an exhausted first fallback from the shared sidebar state', async () => { + const now = Date.now() + const reset = new Date(now + 7 * 24 * 3600_000).toISOString() + seedAdmissionAccounts(['work-alt', 'client-alt']) const seenAuth: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = (async (url: unknown, init?: unknown) => { - if (String(url).includes('/oauth/token')) { - throw new Error('refresh unavailable') - } - const auth = headerValue(init, 'authorization') - seenAuth.push(auth) - if (auth.includes('fallback-access-token')) { - throw new Error('ECONNRESET') - } - return new Response('main must not be called', { status: 200 }) - }) as unknown as typeof globalThis.fetch + globalThis.fetch = mockAdmissionFetch(seenAuth) let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - Date.now() + 3600_000, + now + 3600_000, ) hooks = loaded.hooks + writeAdmissionSidebarState({ + fallbackIds: ['work-alt', 'client-alt'], + fallbackQuotas: { + 'work-alt': admissionQuota(100, reset, now), + 'client-alt': admissionQuota(20, reset, now), + }, + fallbackAccountIds: { + 'work-alt': 'chatgpt-work-alt', + 'client-alt': 'chatgpt-client-alt', + }, + activeId: 'work-alt', + }) - let caught: unknown - try { - await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - requestInit(), - ) - } catch (error) { - caught = error - } - expect(caught).toBeInstanceOf(Error) - expect((caught as Error).message).toBe('ECONNRESET') - expect(seenAuth).toEqual(['Bearer fallback-access-token']) + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + + expect(response.status).toBe(200) + expect(seenAuth).toEqual(['Bearer client-alt-token']) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('fallback-first propagates caller aborts without trying main', async () => { - seedStorage({ access: 'fallback-access-token' }) + it('admission quota skips a file-exhausted fallback with an empty process quota cache', async () => { + const now = Date.now() + const reset = new Date(now + 7 * 24 * 3600_000).toISOString() + seedAdmissionAccounts(['work-alt']) const seenAuth: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = (async (_url: unknown, init?: unknown) => { - const auth = headerValue(init, 'authorization') - seenAuth.push(auth) - throw new DOMException('request aborted', 'AbortError') - }) as unknown as typeof globalThis.fetch + globalThis.fetch = mockAdmissionFetch(seenAuth) let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - Date.now() + 3600_000, + now + 3600_000, ) hooks = loaded.hooks + writeAdmissionSidebarState({ + fallbackIds: ['work-alt'], + fallbackQuotas: { + 'work-alt': admissionQuota(100, reset, now), + }, + fallbackAccountIds: { 'work-alt': 'chatgpt-work-alt' }, + }) - let caught: unknown - try { - await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - requestInit(), - ) - } catch (error) { - caught = error - } - expect(caught).toBeInstanceOf(DOMException) - expect((caught as DOMException).name).toBe('AbortError') - expect(seenAuth).toEqual(['Bearer fallback-access-token']) + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + + expect(response.status).toBe(200) + expect(seenAuth).toEqual(['Bearer main-stale-token']) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('captures cachekeep bodies before the WebSocket transport early return', async () => { - writeFileSync( - configFile, - JSON.stringify({ - version: 1, - main: { type: 'opencode', provider: 'openai' }, - accounts: [], - refresh: { refreshBeforeExpiryMinutes: 5 }, - routing: { mode: 'main-first' }, - }), - ) - - const prompts: string[] = [] + it('admission quota uses fresher healthy memory instead of a stale exhausted file row', async () => { + const now = Date.now() + const reset = new Date(now + 7 * 24 * 3600_000).toISOString() + seedAdmissionAccounts(['work-alt'], 'fallback-first', { + 'work-alt': admissionQuota(20, reset, now), + }) + const seenAuth: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = (async (_url: unknown, _init?: unknown) => { - return new Response('{}', { status: 200 }) - }) as unknown as typeof globalThis.fetch + globalThis.fetch = mockAdmissionFetch(seenAuth) let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( - createMockPluginInput({ - client: { - auth: { set: async () => {} }, - session: { - promptAsync: async (request: unknown) => { - const body = ( - request as { body?: { parts?: Array<{ text?: string }> } } - ).body - const text = body?.parts?.[0]?.text - if (text) prompts.push(text) - }, - }, - } as unknown as PluginInput['client'], - }), - Date.now() + 3600_000, - true, + createMockPluginInput(), + now + 3600_000, ) hooks = loaded.hooks + writeAdmissionSidebarState({ + fallbackIds: ['work-alt'], + fallbackQuotas: { + 'work-alt': admissionQuota(100, reset, now - 60_000), + }, + }) - await runCommand(hooks, 'openai-cachekeep', 'on') await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - responseRequestInit({ 'session-id': 'main-session' }), + requestInit(), ) - await runCommand(hooks, 'openai-cachekeep', 'status') - expect(prompts.at(-1)).toContain('Tracked sessions: **1**') + expect(seenAuth).toEqual(['Bearer work-alt-token']) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('does not capture subagent cachekeep bodies with x-parent-session-id', async () => { - writeFileSync( - configFile, - JSON.stringify({ - version: 1, - main: { type: 'opencode', provider: 'openai' }, - accounts: [], - refresh: { refreshBeforeExpiryMinutes: 5 }, - routing: { mode: 'main-first' }, - }), - ) - - const prompts: string[] = [] + it('admission quota uses a fresher exhausted file row instead of stale healthy memory', async () => { + const now = Date.now() + const reset = new Date(now + 7 * 24 * 3600_000).toISOString() + seedAdmissionAccounts(['work-alt', 'client-alt'], 'fallback-first', { + 'work-alt': admissionQuota(20, reset, now - 60_000), + }) + const seenAuth: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = (async (_url: unknown, _init?: unknown) => { - return new Response('{}', { status: 200 }) - }) as unknown as typeof globalThis.fetch + globalThis.fetch = mockAdmissionFetch(seenAuth) let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( - createMockPluginInput({ - client: { - auth: { set: async () => {} }, - session: { - promptAsync: async (request: unknown) => { - const body = ( - request as { body?: { parts?: Array<{ text?: string }> } } - ).body - const text = body?.parts?.[0]?.text - if (text) prompts.push(text) - }, - }, - } as unknown as PluginInput['client'], - }), - Date.now() + 3600_000, - false, + createMockPluginInput(), + now + 3600_000, ) hooks = loaded.hooks + writeAdmissionSidebarState({ + fallbackIds: ['work-alt', 'client-alt'], + fallbackQuotas: { + 'work-alt': admissionQuota(100, reset, now), + 'client-alt': admissionQuota(20, reset, now), + }, + fallbackAccountIds: { + 'work-alt': 'chatgpt-work-alt', + 'client-alt': 'chatgpt-client-alt', + }, + }) - await runCommand(hooks, 'openai-cachekeep', 'on') await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - responseRequestInit({ - 'session-id': 'main-session', - 'x-parent-session-id': 'parent-session', - }), + requestInit(), ) - await runCommand(hooks, 'openai-cachekeep', 'status') - expect(prompts.at(-1)).toContain('Tracked sessions: **0**') + expect(seenAuth).toEqual(['Bearer client-alt-token']) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('demotes to main when active fallback has no usable access token', async () => { - seedStorage({ access: undefined, expires: Date.now() - 60_000 }) - const seen: Array<{ authorization: string; accountId: string | null }> = [] + it('admission quota skips from fresher exhausted memory instead of a stale healthy file row', async () => { + const now = Date.now() + const reset = new Date(now + 7 * 24 * 3600_000).toISOString() + seedAdmissionAccounts(['work-alt', 'client-alt'], 'fallback-first', { + 'work-alt': admissionQuota(100, reset, now), + }) + const seenAuth: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = (async (_url: unknown, init?: unknown) => { - seen.push({ - authorization: headerValue(init, 'authorization'), - accountId: headerValue(init, 'ChatGPT-Account-Id') || null, - }) - return new Response('{}', { status: 200 }) - }) as unknown as typeof globalThis.fetch + globalThis.fetch = mockAdmissionFetch(seenAuth) let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - Date.now() + 3600_000, + now + 3600_000, ) hooks = loaded.hooks + writeAdmissionSidebarState({ + fallbackIds: ['work-alt', 'client-alt'], + fallbackQuotas: { + 'work-alt': admissionQuota(20, reset, now - 60_000), + 'client-alt': admissionQuota(20, reset, now), + }, + }) - const response = await loaded.fetchOverride( + await loaded.fetchOverride( 'https://api.openai.com/v1/responses', requestInit(), ) - expect(response.status).toBe(200) - expect(seen.filter((entry) => entry.authorization)).toEqual([ - { authorization: 'Bearer main-stale-token', accountId: null }, - ]) + expect(seenAuth).toEqual(['Bearer client-alt-token']) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('sends the refreshed main token when main primary starts expired', async () => { - writeFileSync( - configFile, - JSON.stringify({ - version: 1, - main: { type: 'opencode', provider: 'openai' }, - accounts: [], - refresh: { refreshBeforeExpiryMinutes: 5 }, - routing: { mode: 'main-first' }, - }), - ) - const authSetCalls: unknown[] = [] + test.each([ + ['missing quota', null], + ['missing reset', { primary: { usedPercent: 100, remainingPercent: 0 } }], + [ + 'malformed reset', + { + primary: { + usedPercent: 100, + remainingPercent: 0, + resetsAt: 'not-a-date', + }, + }, + ], + [ + 'malformed usage', + { + primary: { + usedPercent: '100', + remainingPercent: 0, + resetsAt: new Date(Date.now() + 3600_000).toISOString(), + }, + }, + ], + ])('admission quota retains a fallback with %s', async (_label, quota) => { + seedAdmissionAccounts(['work-alt']) const seenAuth: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = (async (url: unknown, init?: unknown) => { - if (String(url).includes('/oauth/token')) { - return new Response( - JSON.stringify({ - access_token: 'main-refreshed-token', - refresh_token: 'main-refresh-new', - expires_in: 3600, - }), - { status: 200, headers: { 'content-type': 'application/json' } }, - ) - } - seenAuth.push(headerValue(init, 'authorization')) - return new Response('{}', { status: 200 }) - }) as unknown as typeof globalThis.fetch + globalThis.fetch = mockAdmissionFetch(seenAuth) let hooks: Hooks | undefined try { - const input = createMockPluginInput({ - client: { - auth: { set: async (payload: unknown) => authSetCalls.push(payload) }, - session: { promptAsync: async () => {} }, - } as unknown as PluginInput['client'], - }) - const loaded = await loadFetchOverride(input, Date.now() - 60_000) + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + ) hooks = loaded.hooks + writeAdmissionSidebarState({ + fallbackIds: ['work-alt'], + fallbackQuotas: { + 'work-alt': quota as SidebarState['main']['quota'], + }, + }) - const response = await loaded.fetchOverride( + await loaded.fetchOverride( 'https://api.openai.com/v1/responses', requestInit(), ) - expect(response.status).toBe(200) - expect(seenAuth).toEqual(['Bearer main-refreshed-token']) - expect(authSetCalls.length).toBe(1) + expect(seenAuth).toEqual(['Bearer work-alt-token']) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('main-first (default): tries main first, then reactively falls back on 429', async () => { - writeFileSync( - configFile, - JSON.stringify({ - version: 1, - main: { type: 'opencode', provider: 'openai' }, - accounts: [ - { - id: 'fallback-1', - type: 'oauth', - enabled: true, - access: 'fallback-primary-token', - refresh: 'fallback-primary-refresh', - expires: Date.now() + 3600_000 * 24, - accountId: 'acc-fallback-primary', - }, - { - id: 'fallback-2', - type: 'oauth', - enabled: true, - access: 'fallback-secondary-token', - refresh: 'fallback-secondary-refresh', - expires: Date.now() + 3600_000 * 24, - accountId: 'acc-fallback-secondary', - }, - ], - refresh: { refreshBeforeExpiryMinutes: 5 }, - // Default (main-first): main is the primary; no per-account pin. - routing: { mode: 'main-first' }, - }), - ) + it('admission quota preserves probe order when every account is exhausted', async () => { + const now = Date.now() + const reset = new Date(now + 7 * 24 * 3600_000).toISOString() + seedAdmissionAccounts(['work-alt', 'client-alt']) const seenAuth: string[] = [] const originalFetch = globalThis.fetch - // Main (main-stale-token) is rate-limited → reactive fallback to fallback-1. - globalThis.fetch = (async (_url: unknown, init?: unknown) => { - const auth = headerValue(init, 'authorization') - seenAuth.push(auth) - return new Response('{}', { - status: auth.includes('main-stale-token') ? 429 : 200, - }) - }) as unknown as typeof globalThis.fetch + globalThis.fetch = mockAdmissionFetch(seenAuth, 429) let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - Date.now() + 3600_000, + now + 3600_000, ) hooks = loaded.hooks + writeAdmissionSidebarState({ + fallbackIds: ['work-alt', 'client-alt'], + fallbackQuotas: { + 'work-alt': admissionQuota(100, reset, now), + 'client-alt': admissionQuota(100, reset, now), + }, + fallbackAccountIds: { + 'work-alt': 'chatgpt-work-alt', + 'client-alt': 'chatgpt-client-alt', + }, + mainQuota: admissionQuota(100, reset, now), + }) const response = await loaded.fetchOverride( 'https://api.openai.com/v1/responses', requestInit(), ) - expect(response.status).toBe(200) - // Main tried first, then the first usable fallback served. + expect(response.status).toBe(429) expect(seenAuth).toEqual([ + 'Bearer work-alt-token', + 'Bearer client-alt-token', 'Bearer main-stale-token', - 'Bearer fallback-primary-token', ]) } finally { globalThis.fetch = originalFetch @@ -4696,438 +5147,1239 @@ describe('integration: active fallback routing', () => { } }) - it('records quota from a failed fallback so the killswitch skips it next turn', async () => { - writeFileSync( - configFile, - JSON.stringify({ - version: 1, - main: { type: 'opencode', provider: 'openai' }, - accounts: [ - { - id: 'fallback-1', - type: 'oauth', - label: 'Fallback', - enabled: true, - access: 'fallback-access-token', - refresh: 'fallback-refresh-token', - expires: Date.now() + 3600_000 * 24, - accountId: 'acc-fallback-1', - }, - ], - routing: { mode: 'fallback-first' }, - killswitch: { - enabled: true, - accounts: { 'fallback-1': { primary: 50, secondary: 50 } }, - }, - }), - ) - + it('admission quota reroutes a file-exhausted main without probing it', async () => { + const now = Date.now() + const reset = new Date(now + 7 * 24 * 3600_000).toISOString() + seedAdmissionAccounts(['client-alt'], 'main-first') const seenAuth: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = (async (_url: unknown, init?: unknown) => { - const auth = headerValue(init, 'authorization') - seenAuth.push(auth) - if (auth.includes('fallback-access-token')) { - return new Response('{}', { - status: 429, - headers: { - 'x-codex-primary-used-percent': '95', - 'x-codex-secondary-used-percent': '95', - }, - }) - } - return new Response('{}', { status: 200 }) - }) as unknown as typeof globalThis.fetch + globalThis.fetch = mockAdmissionFetch(seenAuth) let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - Date.now() + 3600_000, + now + 3600_000, ) hooks = loaded.hooks + await new Promise((resolve) => setTimeout(resolve, 2)) + const checkedAt = Date.now() + writeAdmissionSidebarState({ + fallbackIds: ['client-alt'], + fallbackQuotas: { + 'client-alt': admissionQuota(20, reset, checkedAt), + }, + mainQuota: admissionQuota(100, reset, checkedAt), + route: 'main-first', + }) - const first = await loaded.fetchOverride( + const response = await loaded.fetchOverride( 'https://api.openai.com/v1/responses', requestInit(), ) - expect(first.status).toBe(200) - await first.body?.cancel() - const second = await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - requestInit(), - ) - expect(second.status).toBe(200) - await second.body?.cancel() - expect(seenAuth).toEqual([ - 'Bearer fallback-access-token', - 'Bearer main-stale-token', - 'Bearer main-stale-token', - ]) + expect(response.status).toBe(200) + expect(seenAuth).toEqual(['Bearer client-alt-token']) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('a later single-primary snapshot fully replaces an earlier two-window snapshot, not merges with it', async () => { - // Header/WS pushes are always complete snapshots of every live window - // (never a partial subset), so a later push that omits a window means - // the wire genuinely dropped it — the cached value must not survive. - seedStorage({ access: 'fallback-access-token' }) + it('admission quota ignores an exhausted main row from a different account', async () => { + const now = Date.now() + const reset = new Date(now + 7 * 24 * 3600_000).toISOString() + seedAdmissionAccounts(['client-alt'], 'main-first') + const seenAuth: string[] = [] const originalFetch = globalThis.fetch - let fallbackCalls = 0 - globalThis.fetch = (async (_url: unknown, init?: unknown) => { - const auth = headerValue(init, 'authorization') - if (auth.includes('fallback-access-token')) { - fallbackCalls++ - return new Response('{}', { - status: 429, - headers: - fallbackCalls === 1 - ? { - 'x-codex-primary-used-percent': '10', - 'x-codex-secondary-used-percent': '95', - } - : { 'x-codex-primary-used-percent': '20' }, - }) - } - return new Response('{}', { status: 200 }) - }) as unknown as typeof globalThis.fetch + globalThis.fetch = mockAdmissionFetch(seenAuth) let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - Date.now() + 3600_000, + now + 3600_000, + false, + false, + 'new-account', ) hooks = loaded.hooks + await new Promise((resolve) => setTimeout(resolve, 2)) + const checkedAt = Date.now() + writeAdmissionSidebarState({ + fallbackIds: ['client-alt'], + fallbackQuotas: { + 'client-alt': admissionQuota(20, reset, checkedAt), + }, + mainQuota: admissionQuota(100, reset, checkedAt), + mainAccountId: 'old-account', + route: 'main-first', + }) - const first = await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - requestInit(), - ) - expect(first.status).toBe(200) - await first.body?.cancel() - - const second = await loaded.fetchOverride( + const response = await loaded.fetchOverride( 'https://api.openai.com/v1/responses', requestInit(), ) - expect(second.status).toBe(200) - await second.body?.cancel() - const sidebar = await waitForSidebarState( - sidebarFile, - (s) => - s.fallbacks.find((a) => a.id === 'fallback-1')?.quota?.primary - ?.usedPercent === 20, - ) - const quota = sidebar.fallbacks.find((a) => a.id === 'fallback-1')?.quota - expect(quota?.primary?.usedPercent).toBe(20) - expect(quota?.secondary).toBeUndefined() + expect(response.status).toBe(200) + expect(seenAuth).toEqual(['Bearer main-stale-token']) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('does not replay non-responses POSTs or GET requests through fallbacks', async () => { - writeFileSync( - configFile, - JSON.stringify({ - version: 1, - main: { type: 'opencode', provider: 'openai' }, - accounts: [ - { - id: 'fallback-1', - type: 'oauth', - enabled: true, - access: 'fallback-access-token', - refresh: 'fallback-refresh-token', - expires: Date.now() + 3600_000 * 24, - accountId: 'acc-fallback-1', - }, - ], - routing: { mode: 'main-first' }, - }), - ) - + it('admission quota ignores an exhausted fallback row stamped with a different account identity', async () => { + const now = Date.now() + const reset = new Date(now + 7 * 24 * 3600_000).toISOString() + // Live account identity is chatgpt-work-alt (see seedAdmissionAccounts). + seedAdmissionAccounts(['work-alt']) const seenAuth: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = (async (_url: unknown, init?: unknown) => { - seenAuth.push(headerValue(init, 'authorization')) - return new Response('main limited', { status: 429 }) - }) as unknown as typeof globalThis.fetch + globalThis.fetch = mockAdmissionFetch(seenAuth) let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - Date.now() + 3600_000, + now + 3600_000, ) hooks = loaded.hooks + writeAdmissionSidebarState({ + fallbackIds: ['work-alt'], + fallbackQuotas: { + 'work-alt': admissionQuota(100, reset, now), + }, + // The file row belongs to a previous login of this stable id; the live + // account is a different ChatGPT identity, so the exhausted row must be + // treated as absent (fail-open) rather than blocking the replacement. + fallbackAccountIds: { 'work-alt': 'chatgpt-stale' }, + }) - const chat = await loaded.fetchOverride( - 'https://api.openai.com/v1/chat/completions', + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', requestInit(), ) - expect(chat.status).toBe(429) - expect(await chat.text()).toBe('main limited') - const get = await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - { method: 'GET' }, - ) - expect(get.status).toBe(429) - expect(await get.text()).toBe('main limited') - expect(seenAuth).toEqual([ - 'Bearer main-stale-token', - 'Bearer main-stale-token', - ]) + expect(response.status).toBe(200) + expect(seenAuth).toEqual(['Bearer work-alt-token']) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('stops reactive fallback on an indeterminate transport throw and returns the primary response', async () => { - writeFileSync( - configFile, - JSON.stringify({ - version: 1, - main: { type: 'opencode', provider: 'openai' }, - accounts: [ - { - id: 'fallback-1', - type: 'oauth', - enabled: true, - access: 'fallback-throw-token', - refresh: 'fallback-throw-refresh', - expires: Date.now() + 3600_000 * 24, - }, - { - id: 'fallback-2', - type: 'oauth', - enabled: true, - access: 'fallback-never-token', - refresh: 'fallback-never-refresh', - expires: Date.now() + 3600_000 * 24, - }, - ], - routing: { mode: 'main-first' }, - }), - ) - + it('admission quota honors an exhausted fallback row matching the live account identity', async () => { + const now = Date.now() + const reset = new Date(now + 7 * 24 * 3600_000).toISOString() + seedAdmissionAccounts(['work-alt']) const seenAuth: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = (async (_url: unknown, init?: unknown) => { - const auth = headerValue(init, 'authorization') - seenAuth.push(auth) - if (auth.includes('fallback-throw-token')) throw new Error('ECONNRESET') - if (auth.includes('fallback-never-token')) { - return new Response('should not be called', { status: 200 }) - } - return new Response('primary body stays readable', { status: 429 }) - }) as unknown as typeof globalThis.fetch + globalThis.fetch = mockAdmissionFetch(seenAuth) let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - Date.now() + 3600_000, + now + 3600_000, ) hooks = loaded.hooks - - const response = await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - requestInit(), - ) - expect(response.status).toBe(429) - expect(await response.text()).toBe('primary body stays readable') - expect(seenAuth).toEqual([ - 'Bearer main-stale-token', - 'Bearer fallback-throw-token', - ]) + writeAdmissionSidebarState({ + fallbackIds: ['work-alt'], + fallbackQuotas: { + 'work-alt': admissionQuota(100, reset, now), + }, + // Identity matches the live account, so the exhausted row is honored + // and the fallback is skipped in favor of main. + fallbackAccountIds: { 'work-alt': 'chatgpt-work-alt' }, + }) + + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + + expect(response.status).toBe(200) + expect(seenAuth).toEqual(['Bearer main-stale-token']) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('propagates caller aborts from reactive fallback attempts', async () => { - seedStorage({ access: 'fallback-access-token' }, { mode: 'main-first' }) + it('admission quota skips a fallback exhausted only on its secondary window', async () => { + const now = Date.now() + const reset = new Date(now + 7 * 24 * 3600_000).toISOString() + seedAdmissionAccounts(['work-alt']) + const seenAuth: string[] = [] const originalFetch = globalThis.fetch - globalThis.fetch = (async (_url: unknown, init?: unknown) => { - const auth = headerValue(init, 'authorization') - if (auth.includes('fallback-access-token')) { - throw new DOMException('request aborted', 'AbortError') - } - return new Response('main limited', { status: 429 }) - }) as unknown as typeof globalThis.fetch + globalThis.fetch = mockAdmissionFetch(seenAuth) let hooks: Hooks | undefined try { const loaded = await loadFetchOverride( createMockPluginInput(), - Date.now() + 3600_000, + now + 3600_000, ) hooks = loaded.hooks + writeAdmissionSidebarState({ + fallbackIds: ['work-alt'], + fallbackQuotas: { + // No primary window; the secondary window alone is exhausted. + 'work-alt': { + secondary: { + usedPercent: 100, + remainingPercent: 0, + resetsAt: reset, + checkedAt: now, + windowMinutes: 10_080, + }, + }, + }, + fallbackAccountIds: { 'work-alt': 'chatgpt-work-alt' }, + }) - let caught: unknown - try { - await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - requestInit(), - ) - } catch (error) { - caught = error - } - expect(caught).toBeInstanceOf(DOMException) - expect((caught as DOMException).name).toBe('AbortError') + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + + expect(response.status).toBe(200) + expect(seenAuth).toEqual(['Bearer main-stale-token']) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('orders HTTP Codex bodies the same way as WebSocket bodies', async () => { - writeFileSync( - configFile, - JSON.stringify({ - version: 1, - main: { type: 'opencode', provider: 'openai' }, - accounts: [], - routing: { mode: 'main-first' }, - }), - ) - const request = (): RequestInit => ({ - method: 'POST', - headers: { - 'content-type': 'application/json', - 'session-id': 'body-order-session', - }, - body: JSON.stringify({ - stream: true, - client_metadata: { existing: 'yes' }, - input: [{ role: 'user', content: 'hi' }], - previous_response_id: 'resp_prev', - model: 'gpt-5.5', - type: 'response.create', - reasoning: { effort: 'max', summary: 'auto' }, - tools: [], - store: false, - }), - }) - const expectedKeys = [ - 'type', - 'model', - 'previous_response_id', - 'input', - 'tools', - 'parallel_tool_calls', - 'reasoning', - 'store', - 'stream', - 'prompt_cache_key', - 'client_metadata', - ] - + it('admission quota retains an exhausted-looking fallback after its reset passes', async () => { + const now = Date.now() + seedAdmissionAccounts(['work-alt']) + const seenAuth: string[] = [] const originalFetch = globalThis.fetch - let httpBody = '' - let httpHooks: Hooks | undefined + globalThis.fetch = mockAdmissionFetch(seenAuth) + + let hooks: Hooks | undefined try { - globalThis.fetch = (async (_url: unknown, init?: unknown) => { - httpBody = String((init as { body?: unknown } | undefined)?.body ?? '') - return new Response('{}', { status: 200 }) - }) as unknown as typeof globalThis.fetch const loaded = await loadFetchOverride( createMockPluginInput(), - Date.now() + 3600_000, - false, + now + 3600_000, ) - httpHooks = loaded.hooks - const response = await loaded.fetchOverride( + hooks = loaded.hooks + writeAdmissionSidebarState({ + fallbackIds: ['work-alt'], + fallbackQuotas: { + 'work-alt': admissionQuota( + 100, + new Date(now - 60_000).toISOString(), + now, + ), + }, + }) + + await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - request(), + requestInit(), ) - expect(response.status).toBe(200) - await response.body?.cancel() + + expect(seenAuth).toEqual(['Bearer work-alt-token']) } finally { globalThis.fetch = originalFetch - await httpHooks?.dispose?.() + await hooks?.dispose?.() } + }) - let wsBody = '' - let wsHooks: Hooks | undefined - await withFakeWebSocket( - ({ message }) => ({ - send(data) { - wsBody = data - message( - JSON.stringify({ - type: 'response.completed', - response: { id: 'resp_order' }, - }), - ) - }, - }), - async () => { - try { - globalThis.fetch = (async () => - new Response('{}', { - status: 200, - })) as unknown as typeof globalThis.fetch - const loaded = await loadFetchOverride( - createMockPluginInput(), - Date.now() + 3600_000, - true, - ) - wsHooks = loaded.hooks - const response = await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - request(), - ) - expect(response.status).toBe(200) - await response.text() - } finally { - globalThis.fetch = originalFetch - await wsHooks?.dispose?.() - } - }, - ) + it('admission quota ignores an unstamped (no accountId) exhausted fallback row against a known identity', async () => { + const now = Date.now() + const reset = new Date(now + 7 * 24 * 3600_000).toISOString() + // Live account identity is chatgpt-work-alt (see seedAdmissionAccounts). + seedAdmissionAccounts(['work-alt']) + const seenAuth: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = mockAdmissionFetch(seenAuth) - const parsedHttpBody = JSON.parse(httpBody) - const parsedWsBody = JSON.parse(wsBody) - expect(Object.keys(parsedHttpBody)).toEqual(expectedKeys) - expect(Object.keys(parsedWsBody)).toEqual(expectedKeys) - expect(parsedHttpBody.reasoning).toEqual({ effort: 'max', summary: 'auto' }) - expect(parsedWsBody.reasoning).toEqual({ effort: 'max', summary: 'auto' }) - }) + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + now + 3600_000, + ) + hooks = loaded.hooks + await new Promise((resolve) => setTimeout(resolve, 2)) + const checkedAt = Date.now() + writeAdmissionSidebarState({ + fallbackIds: ['work-alt'], + fallbackQuotas: { + 'work-alt': admissionQuota(100, reset, checkedAt), + }, + // No fallbackAccountIds — the file row carries no accountId stamp. + // The live identity is known (chatgpt-work-alt), so this exhausted + // unstamped row must not be trusted: the fallback is still probed. + }) - it('gates Responses Lite by setting and exact HTTP model', async () => { - const cases = [ - { name: 'sol', model: 'gpt-5.6-sol', enabled: true, marked: true }, - { - name: 'legacy-pro', - model: 'gpt-5.6-sol-pro', - enabled: true, - marked: false, - }, - { name: 'disabled', model: 'gpt-5.6-sol', enabled: false, marked: false }, - ] - for (const testCase of cases) { - const captured = await captureResponsesLiteHttpRequest( - testCase.model, - testCase.enabled, - `responses-lite-${testCase.name}`, + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), ) - expect( - new Headers(captured.headers).get( - 'x-openai-internal-codex-responses-lite', - ), - ).toBe(testCase.marked ? 'true' : null) + + // Fallback is probed — not skipped based on an unstamped file row. + expect(response.status).toBe(200) + expect(seenAuth).toEqual(['Bearer work-alt-token']) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('admission quota probes main for a non-replayable request even when the file says exhausted and a fallback is retained', async () => { + const now = Date.now() + const reset = new Date(now + 7 * 24 * 3600_000).toISOString() + seedAdmissionAccounts(['work-alt'], 'main-first') + const seenAuth: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = mockAdmissionFetch(seenAuth) + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + now + 3600_000, + ) + hooks = loaded.hooks + await new Promise((resolve) => setTimeout(resolve, 2)) + const checkedAt = Date.now() + writeAdmissionSidebarState({ + fallbackIds: ['work-alt'], + fallbackQuotas: { + 'work-alt': admissionQuota(20, reset, checkedAt), + }, + mainQuota: admissionQuota(100, reset, checkedAt), + route: 'main-first', + }) + + // A non-replayable GET request: main is file-exhausted and a healthy + // fallback is retained, but without the replayability guard the + // quotaBlocksMain check would produce a synthetic 429 without ever + // probing main. With the fix, main IS probed. + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + { method: 'GET', headers: { 'content-type': 'application/json' } }, + ) + + // Main was probed — not skipped by a synthetic 429. + expect(response.status).toBe(200) + expect(seenAuth).toEqual(['Bearer main-stale-token']) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('refreshes an expired active fallback without writing the auth slot', async () => { + seedStorage({ + access: 'fallback-stale-token', + expires: Date.now() - 60_000, + }) + const authSetCalls: unknown[] = [] + const seenAuth: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('/oauth/token')) { + return new Response( + JSON.stringify({ + access_token: 'fallback-refreshed-token', + refresh_token: 'fallback-refresh-new', + expires_in: 3600, + }), + { status: 200, headers: { 'content-type': 'application/json' } }, + ) + } + seenAuth.push(headerValue(init, 'authorization')) + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const input = createMockPluginInput({ + client: { + auth: { set: async (payload: unknown) => authSetCalls.push(payload) }, + session: { promptAsync: async () => {} }, + } as unknown as PluginInput['client'], + }) + const loaded = await loadFetchOverride(input, Date.now() + 3600_000) + hooks = loaded.hooks + + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + + expect(response.status).toBe(200) + expect(seenAuth).toEqual(['Bearer fallback-refreshed-token']) + expect(authSetCalls).toEqual([]) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('fallback-first uses a still-valid fallback token when its refresh fails', async () => { + // Token is inside the refresh window (needs refresh) but NOT expired, so a + // failed refresh must not drop it — the still-valid token is used. + seedStorage({ + access: 'fallback-stale-token', + expires: Date.now() + 2 * 60_000, + }) + const seenAuth: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('/oauth/token')) { + throw new Error('refresh unavailable') + } + seenAuth.push(headerValue(init, 'authorization')) + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + ) + hooks = loaded.hooks + + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + expect(response.status).toBe(200) + expect(seenAuth).toEqual(['Bearer fallback-stale-token']) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('fallback-first does not re-try fallbacks reactively after main also fails (no double-spend)', async () => { + // fallback-first with one fallback that 429s: the proactive gate tries it, + // falls through to main, and main also 429s. The reactive path must NOT + // re-try the already-tried fallback — so the fallback is hit exactly once. + seedStorage({ access: 'fallback-access-token' }) + const seenAuth: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('/oauth/token')) { + throw new Error('refresh unavailable') + } + seenAuth.push(headerValue(init, 'authorization')) + // Everything is rate-limited. + return new Response('{}', { status: 429 }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + ) + hooks = loaded.hooks + + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + expect(response.status).toBe(429) + await response.body?.cancel() + // Fallback tried once (proactive), then main once — no reactive re-try. + expect(seenAuth).toEqual([ + 'Bearer fallback-access-token', + 'Bearer main-stale-token', + ]) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('fallback-first propagates a transport error without replaying on main', async () => { + // The fallback send may already have generated or billed before the + // transport error surfaced, so routing must stop instead of replaying. + seedStorage({ access: 'fallback-access-token' }) + const seenAuth: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('/oauth/token')) { + throw new Error('refresh unavailable') + } + const auth = headerValue(init, 'authorization') + seenAuth.push(auth) + if (auth.includes('fallback-access-token')) { + throw new Error('ECONNRESET') + } + return new Response('main must not be called', { status: 200 }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + ) + hooks = loaded.hooks + + let caught: unknown + try { + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + } catch (error) { + caught = error + } + expect(caught).toBeInstanceOf(Error) + expect((caught as Error).message).toBe('ECONNRESET') + expect(seenAuth).toEqual(['Bearer fallback-access-token']) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('fallback-first propagates caller aborts without trying main', async () => { + seedStorage({ access: 'fallback-access-token' }) + const seenAuth: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = (async (_url: unknown, init?: unknown) => { + const auth = headerValue(init, 'authorization') + seenAuth.push(auth) + throw new DOMException('request aborted', 'AbortError') + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + ) + hooks = loaded.hooks + + let caught: unknown + try { + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + } catch (error) { + caught = error + } + expect(caught).toBeInstanceOf(DOMException) + expect((caught as DOMException).name).toBe('AbortError') + expect(seenAuth).toEqual(['Bearer fallback-access-token']) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('captures cachekeep bodies before the WebSocket transport early return', async () => { + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [], + refresh: { refreshBeforeExpiryMinutes: 5 }, + routing: { mode: 'main-first' }, + }), + ) + + const prompts: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = (async (_url: unknown, _init?: unknown) => { + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput({ + client: { + auth: { set: async () => {} }, + session: { + promptAsync: async (request: unknown) => { + const body = ( + request as { body?: { parts?: Array<{ text?: string }> } } + ).body + const text = body?.parts?.[0]?.text + if (text) prompts.push(text) + }, + }, + } as unknown as PluginInput['client'], + }), + Date.now() + 3600_000, + true, + ) + hooks = loaded.hooks + + await runCommand(hooks, 'openai-cachekeep', 'on') + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'session-id': 'main-session' }), + ) + await runCommand(hooks, 'openai-cachekeep', 'status') + + expect(prompts.at(-1)).toContain('Tracked sessions: **1**') + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('does not capture subagent cachekeep bodies with x-parent-session-id', async () => { + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [], + refresh: { refreshBeforeExpiryMinutes: 5 }, + routing: { mode: 'main-first' }, + }), + ) + + const prompts: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = (async (_url: unknown, _init?: unknown) => { + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput({ + client: { + auth: { set: async () => {} }, + session: { + promptAsync: async (request: unknown) => { + const body = ( + request as { body?: { parts?: Array<{ text?: string }> } } + ).body + const text = body?.parts?.[0]?.text + if (text) prompts.push(text) + }, + }, + } as unknown as PluginInput['client'], + }), + Date.now() + 3600_000, + false, + ) + hooks = loaded.hooks + + await runCommand(hooks, 'openai-cachekeep', 'on') + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ + 'session-id': 'main-session', + 'x-parent-session-id': 'parent-session', + }), + ) + await runCommand(hooks, 'openai-cachekeep', 'status') + + expect(prompts.at(-1)).toContain('Tracked sessions: **0**') + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('demotes to main when active fallback has no usable access token', async () => { + seedStorage({ access: undefined, expires: Date.now() - 60_000 }) + const seen: Array<{ authorization: string; accountId: string | null }> = [] + const originalFetch = globalThis.fetch + globalThis.fetch = (async (_url: unknown, init?: unknown) => { + seen.push({ + authorization: headerValue(init, 'authorization'), + accountId: headerValue(init, 'ChatGPT-Account-Id') || null, + }) + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + ) + hooks = loaded.hooks + + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + + expect(response.status).toBe(200) + expect(seen.filter((entry) => entry.authorization)).toEqual([ + { authorization: 'Bearer main-stale-token', accountId: null }, + ]) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('sends the refreshed main token when main primary starts expired', async () => { + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [], + refresh: { refreshBeforeExpiryMinutes: 5 }, + routing: { mode: 'main-first' }, + }), + ) + const authSetCalls: unknown[] = [] + const seenAuth: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('/oauth/token')) { + return new Response( + JSON.stringify({ + access_token: 'main-refreshed-token', + refresh_token: 'main-refresh-new', + expires_in: 3600, + }), + { status: 200, headers: { 'content-type': 'application/json' } }, + ) + } + seenAuth.push(headerValue(init, 'authorization')) + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const input = createMockPluginInput({ + client: { + auth: { set: async (payload: unknown) => authSetCalls.push(payload) }, + session: { promptAsync: async () => {} }, + } as unknown as PluginInput['client'], + }) + const loaded = await loadFetchOverride(input, Date.now() - 60_000) + hooks = loaded.hooks + + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + + expect(response.status).toBe(200) + expect(seenAuth).toEqual(['Bearer main-refreshed-token']) + expect(authSetCalls.length).toBe(1) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('main-first (default): tries main first, then reactively falls back on 429', async () => { + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [ + { + id: 'fallback-1', + type: 'oauth', + enabled: true, + access: 'fallback-primary-token', + refresh: 'fallback-primary-refresh', + expires: Date.now() + 3600_000 * 24, + accountId: 'acc-fallback-primary', + }, + { + id: 'fallback-2', + type: 'oauth', + enabled: true, + access: 'fallback-secondary-token', + refresh: 'fallback-secondary-refresh', + expires: Date.now() + 3600_000 * 24, + accountId: 'acc-fallback-secondary', + }, + ], + refresh: { refreshBeforeExpiryMinutes: 5 }, + // Default (main-first): main is the primary; no per-account pin. + routing: { mode: 'main-first' }, + }), + ) + const seenAuth: string[] = [] + const originalFetch = globalThis.fetch + // Main (main-stale-token) is rate-limited → reactive fallback to fallback-1. + globalThis.fetch = (async (_url: unknown, init?: unknown) => { + const auth = headerValue(init, 'authorization') + seenAuth.push(auth) + return new Response('{}', { + status: auth.includes('main-stale-token') ? 429 : 200, + }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + ) + hooks = loaded.hooks + + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + + expect(response.status).toBe(200) + // Main tried first, then the first usable fallback served. + expect(seenAuth).toEqual([ + 'Bearer main-stale-token', + 'Bearer fallback-primary-token', + ]) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('records quota from a failed fallback so the killswitch skips it next turn', async () => { + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [ + { + id: 'fallback-1', + type: 'oauth', + label: 'Fallback', + enabled: true, + access: 'fallback-access-token', + refresh: 'fallback-refresh-token', + expires: Date.now() + 3600_000 * 24, + accountId: 'acc-fallback-1', + }, + ], + routing: { mode: 'fallback-first' }, + killswitch: { + enabled: true, + accounts: { 'fallback-1': { primary: 50, secondary: 50 } }, + }, + }), + ) + + const seenAuth: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = (async (_url: unknown, init?: unknown) => { + const auth = headerValue(init, 'authorization') + seenAuth.push(auth) + if (auth.includes('fallback-access-token')) { + return new Response('{}', { + status: 429, + headers: { + 'x-codex-primary-used-percent': '95', + 'x-codex-secondary-used-percent': '95', + }, + }) + } + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + ) + hooks = loaded.hooks + + const first = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + expect(first.status).toBe(200) + await first.body?.cancel() + + const second = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + expect(second.status).toBe(200) + await second.body?.cancel() + expect(seenAuth).toEqual([ + 'Bearer fallback-access-token', + 'Bearer main-stale-token', + 'Bearer main-stale-token', + ]) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('a later single-primary snapshot fully replaces an earlier two-window snapshot, not merges with it', async () => { + // Header/WS pushes are always complete snapshots of every live window + // (never a partial subset), so a later push that omits a window means + // the wire genuinely dropped it — the cached value must not survive. + seedStorage({ access: 'fallback-access-token' }) + const originalFetch = globalThis.fetch + let fallbackCalls = 0 + globalThis.fetch = (async (_url: unknown, init?: unknown) => { + const auth = headerValue(init, 'authorization') + if (auth.includes('fallback-access-token')) { + fallbackCalls++ + return new Response('{}', { + status: 429, + headers: + fallbackCalls === 1 + ? { + 'x-codex-primary-used-percent': '10', + 'x-codex-secondary-used-percent': '95', + } + : { 'x-codex-primary-used-percent': '20' }, + }) + } + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + ) + hooks = loaded.hooks + + const first = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + expect(first.status).toBe(200) + await first.body?.cancel() + + const second = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + expect(second.status).toBe(200) + await second.body?.cancel() + + const sidebar = await waitForSidebarState( + sidebarFile, + (s) => + s.fallbacks.find((a) => a.id === 'fallback-1')?.quota?.primary + ?.usedPercent === 20, + ) + const quota = sidebar.fallbacks.find((a) => a.id === 'fallback-1')?.quota + expect(quota?.primary?.usedPercent).toBe(20) + expect(quota?.secondary).toBeUndefined() + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('does not replay non-responses POSTs or GET requests through fallbacks', async () => { + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [ + { + id: 'fallback-1', + type: 'oauth', + enabled: true, + access: 'fallback-access-token', + refresh: 'fallback-refresh-token', + expires: Date.now() + 3600_000 * 24, + accountId: 'acc-fallback-1', + }, + ], + routing: { mode: 'main-first' }, + }), + ) + + const seenAuth: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = (async (_url: unknown, init?: unknown) => { + seenAuth.push(headerValue(init, 'authorization')) + return new Response('main limited', { status: 429 }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + ) + hooks = loaded.hooks + + const chat = await loaded.fetchOverride( + 'https://api.openai.com/v1/chat/completions', + requestInit(), + ) + expect(chat.status).toBe(429) + expect(await chat.text()).toBe('main limited') + + const get = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + { method: 'GET' }, + ) + expect(get.status).toBe(429) + expect(await get.text()).toBe('main limited') + expect(seenAuth).toEqual([ + 'Bearer main-stale-token', + 'Bearer main-stale-token', + ]) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('stops reactive fallback on an indeterminate transport throw and returns the primary response', async () => { + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [ + { + id: 'fallback-1', + type: 'oauth', + enabled: true, + access: 'fallback-throw-token', + refresh: 'fallback-throw-refresh', + expires: Date.now() + 3600_000 * 24, + }, + { + id: 'fallback-2', + type: 'oauth', + enabled: true, + access: 'fallback-never-token', + refresh: 'fallback-never-refresh', + expires: Date.now() + 3600_000 * 24, + }, + ], + routing: { mode: 'main-first' }, + }), + ) + + const seenAuth: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = (async (_url: unknown, init?: unknown) => { + const auth = headerValue(init, 'authorization') + seenAuth.push(auth) + if (auth.includes('fallback-throw-token')) throw new Error('ECONNRESET') + if (auth.includes('fallback-never-token')) { + return new Response('should not be called', { status: 200 }) + } + return new Response('primary body stays readable', { status: 429 }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + ) + hooks = loaded.hooks + + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + expect(response.status).toBe(429) + expect(await response.text()).toBe('primary body stays readable') + expect(seenAuth).toEqual([ + 'Bearer main-stale-token', + 'Bearer fallback-throw-token', + ]) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('propagates caller aborts from reactive fallback attempts', async () => { + seedStorage({ access: 'fallback-access-token' }, { mode: 'main-first' }) + const originalFetch = globalThis.fetch + globalThis.fetch = (async (_url: unknown, init?: unknown) => { + const auth = headerValue(init, 'authorization') + if (auth.includes('fallback-access-token')) { + throw new DOMException('request aborted', 'AbortError') + } + return new Response('main limited', { status: 429 }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + ) + hooks = loaded.hooks + + let caught: unknown + try { + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + } catch (error) { + caught = error + } + expect(caught).toBeInstanceOf(DOMException) + expect((caught as DOMException).name).toBe('AbortError') + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('orders HTTP Codex bodies the same way as WebSocket bodies', async () => { + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [], + routing: { mode: 'main-first' }, + }), + ) + const request = (): RequestInit => ({ + method: 'POST', + headers: { + 'content-type': 'application/json', + 'session-id': 'body-order-session', + }, + body: JSON.stringify({ + stream: true, + client_metadata: { existing: 'yes' }, + input: [{ role: 'user', content: 'hi' }], + previous_response_id: 'resp_prev', + model: 'gpt-5.5', + type: 'response.create', + reasoning: { effort: 'max', summary: 'auto' }, + tools: [], + store: false, + }), + }) + const expectedKeys = [ + 'type', + 'model', + 'previous_response_id', + 'input', + 'tools', + 'parallel_tool_calls', + 'reasoning', + 'store', + 'stream', + 'prompt_cache_key', + 'client_metadata', + ] + + const originalFetch = globalThis.fetch + let httpBody = '' + let httpHooks: Hooks | undefined + try { + globalThis.fetch = (async (_url: unknown, init?: unknown) => { + httpBody = String((init as { body?: unknown } | undefined)?.body ?? '') + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + false, + ) + httpHooks = loaded.hooks + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + request(), + ) + expect(response.status).toBe(200) + await response.body?.cancel() + } finally { + globalThis.fetch = originalFetch + await httpHooks?.dispose?.() + } + + let wsBody = '' + let wsHooks: Hooks | undefined + await withFakeWebSocket( + ({ message }) => ({ + send(data) { + wsBody = data + message( + JSON.stringify({ + type: 'response.completed', + response: { id: 'resp_order' }, + }), + ) + }, + }), + async () => { + try { + globalThis.fetch = (async () => + new Response('{}', { + status: 200, + })) as unknown as typeof globalThis.fetch + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + true, + ) + wsHooks = loaded.hooks + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + request(), + ) + expect(response.status).toBe(200) + await response.text() + } finally { + globalThis.fetch = originalFetch + await wsHooks?.dispose?.() + } + }, + ) + + const parsedHttpBody = JSON.parse(httpBody) + const parsedWsBody = JSON.parse(wsBody) + expect(Object.keys(parsedHttpBody)).toEqual(expectedKeys) + expect(Object.keys(parsedWsBody)).toEqual(expectedKeys) + expect(parsedHttpBody.reasoning).toEqual({ effort: 'max', summary: 'auto' }) + expect(parsedWsBody.reasoning).toEqual({ effort: 'max', summary: 'auto' }) + }) + + it('gates Responses Lite by setting and exact HTTP model', async () => { + const cases = [ + { name: 'sol', model: 'gpt-5.6-sol', enabled: true, marked: true }, + { + name: 'legacy-pro', + model: 'gpt-5.6-sol-pro', + enabled: true, + marked: false, + }, + { name: 'disabled', model: 'gpt-5.6-sol', enabled: false, marked: false }, + ] + for (const testCase of cases) { + const captured = await captureResponsesLiteHttpRequest( + testCase.model, + testCase.enabled, + `responses-lite-${testCase.name}`, + ) + expect( + new Headers(captured.headers).get( + 'x-openai-internal-codex-responses-lite', + ), + ).toBe(testCase.marked ? 'true' : null) } }) @@ -5144,351 +6396,820 @@ describe('integration: active fallback routing', () => { expect('instructions' in body).toBe(false) expect('tools' in body).toBe(false) - const input = body.input as Array> - expect(input[0]).toMatchObject({ - type: 'additional_tools', - role: 'developer', - }) - expect(input[0]?.tools).toContainEqual({ - type: 'function', - name: 'read', - strict: false, - parameters: {}, - }) + const input = body.input as Array> + expect(input[0]).toMatchObject({ + type: 'additional_tools', + role: 'developer', + }) + expect(input[0]?.tools).toContainEqual({ + type: 'function', + name: 'read', + strict: false, + parameters: {}, + }) + expect( + (input[0]?.tools as Array> | undefined)?.some( + (tool) => tool.type === 'web_search', + ) ?? false, + ).toBe(false) + expect(input[1]).toEqual({ + type: 'message', + role: 'developer', + content: [{ type: 'input_text', text: 'Be concise' }], + }) + const sourceInput = input.slice(2) + expect([ + ( + sourceInput[0]?.content as Array> | undefined + )?.[1]?.detail, + ( + sourceInput[1]?.output as Array> | undefined + )?.[0]?.detail, + ( + sourceInput[2]?.output as Array> | undefined + )?.[0]?.detail, + ]).toEqual([undefined, undefined, undefined]) + }) + + it('preserves the standard body when Responses Lite is disabled', async () => { + const captured = await captureResponsesLiteHttpRequest( + 'gpt-5.6-sol', + false, + 'responses-lite-disabled', + ) + const body = JSON.parse(String(captured.body)) as Record + const input = body.input as Array> + expect([ + (input[0]?.content as Array> | undefined)?.[1] + ?.detail, + (input[1]?.output as Array> | undefined)?.[0] + ?.detail, + (input[2]?.output as Array> | undefined)?.[0] + ?.detail, + ]).toEqual(['high', 'low', 'auto']) + expect(body.parallel_tool_calls).toBe(true) + expect(body.instructions).toBe('Be concise') + expect(body.tools).toBeDefined() + }) + + it('marks Responses Lite in WS metadata and still prewarms', async () => { + seedEmptyAccountStorage() + const sent: Array> = [] + let seenUpgradeHeaders: Record = {} + let hooks: Hooks | undefined + await withFakeWebSocket( + ({ message, upgradeHeaders }) => ({ + send(data) { + seenUpgradeHeaders = upgradeHeaders + const body = JSON.parse(data) as Record + sent.push(body) + message( + JSON.stringify({ + type: 'response.completed', + response: { + id: body.generate === false ? 'resp_prewarm' : 'resp_main', + }, + }), + ) + }, + }), + async () => { + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + true, + true, + ) + hooks = loaded.hooks + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responsesLiteRequestInit('gpt-5.6-sol', 'responses-lite-ws', { + stream: true, + }), + ) + await response.text() + } finally { + await hooks?.dispose?.() + } + }, + ) + + expect(sent).toHaveLength(2) + expect(sent[0]?.generate).toBe(false) + const main = sent[1]! + expect( + (main.client_metadata as Record) + .ws_request_header_x_openai_internal_codex_responses_lite, + ).toBe('true') + expect( + seenUpgradeHeaders['x-openai-internal-codex-responses-lite'], + ).toBeUndefined() + }) + + it('converts a WS Responses Lite capture into a sanitized HTTP cachekeep request', async () => { + seedEmptyAccountStorage() + const originalFetch = globalThis.fetch + const originalNow = Date.now + let now = originalNow() + const warmRequests: RequestInit[] = [] + let hooks: Hooks | undefined + Date.now = () => now + try { + globalThis.fetch = (async (_url: unknown, init?: RequestInit) => { + if (init) warmRequests.push(init) + return new Response('{}', { status: 200 }) + }) as typeof globalThis.fetch + await withFakeWebSocket( + ({ message }) => ({ + send(data) { + const body = JSON.parse(data) as Record + message( + JSON.stringify({ + type: 'response.completed', + response: { + id: body.generate === false ? 'resp_prewarm' : 'resp_main', + }, + }), + ) + }, + }), + async () => { + const loaded = await loadFetchOverride( + createMockPluginInput(), + now + 3600_000, + true, + true, + ) + hooks = loaded.hooks + await runCommand(hooks, 'openai-cachekeep', 'on') + const request = responsesLiteRequestInit( + 'gpt-5.6-sol', + 'responses-lite-keepwarm', + { stream: true }, + ) + const headers = new Headers(request.headers) + headers.set('x-opencode-session', 'internal-session') + request.headers = headers + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + request, + ) + await response.text() + now += 30 * 60_000 + const manager = ( + globalThis as typeof globalThis & { + __openaiAuthCacheKeepManager?: { tick(): Promise } + } + ).__openaiAuthCacheKeepManager + if (!manager) throw new Error('missing cachekeep manager') + await manager.tick() + }, + ) + } finally { + Date.now = originalNow + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + + expect(warmRequests).toHaveLength(1) + const warm = warmRequests[0]! + const warmHeaders = new Headers(warm.headers) + expect(warmHeaders.get('x-openai-internal-codex-responses-lite')).toBe( + 'true', + ) + expect(warmHeaders.has('x-opencode-session')).toBe(false) + const body = JSON.parse(String(warm.body)) as Record + const metadata = body.client_metadata as Record + expect('x-codex-turn-metadata' in metadata).toBe(false) + expect('x-codex-ws-stream-request-start-ms' in metadata).toBe(false) expect( - (input[0]?.tools as Array> | undefined)?.some( - (tool) => tool.type === 'web_search', - ) ?? false, + 'ws_request_header_x_openai_internal_codex_responses_lite' in metadata, ).toBe(false) - expect(input[1]).toEqual({ - type: 'message', - role: 'developer', - content: [{ type: 'input_text', text: 'Be concise' }], - }) - const sourceInput = input.slice(2) - expect([ - ( - sourceInput[0]?.content as Array> | undefined - )?.[1]?.detail, - ( - sourceInput[1]?.output as Array> | undefined - )?.[0]?.detail, - ( - sourceInput[2]?.output as Array> | undefined - )?.[0]?.detail, - ]).toEqual([undefined, undefined, undefined]) }) - it('preserves the standard body when Responses Lite is disabled', async () => { - const captured = await captureResponsesLiteHttpRequest( - 'gpt-5.6-sol', - false, - 'responses-lite-disabled', - ) - const body = JSON.parse(String(captured.body)) as Record - const input = body.input as Array> - expect([ - (input[0]?.content as Array> | undefined)?.[1] - ?.detail, - (input[1]?.output as Array> | undefined)?.[0] - ?.detail, - (input[2]?.output as Array> | undefined)?.[0] - ?.detail, - ]).toEqual(['high', 'low', 'auto']) - expect(body.parallel_tool_calls).toBe(true) - expect(body.instructions).toBe('Be concise') - expect(body.tools).toBeDefined() + it('keeps the main refresh advisory lease shorter than the file lock TTL', () => { + expect(MAIN_REFRESH_LEASE_TTL_MS).toBe(90_000) + expect(MAIN_REFRESH_LEASE_TTL_MS).toBeLessThan(MAIN_REFRESH_LOCK_TTL_MS) + }) + + it('retries persisting rotated main tokens without refreshing twice', async () => { + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [], + refresh: { refreshBeforeExpiryMinutes: 5 }, + routing: { mode: 'main-first' }, + }), + ) + const originalFetch = globalThis.fetch + const seenAuth: string[] = [] + let oauthRefreshCalls = 0 + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('/oauth/token')) { + oauthRefreshCalls++ + return new Response( + JSON.stringify({ + access_token: 'main-refreshed-token', + refresh_token: 'main-refresh-new', + expires_in: 3600, + }), + { status: 200, headers: { 'content-type': 'application/json' } }, + ) + } + seenAuth.push(headerValue(init, 'authorization')) + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch + + let authSetCalls = 0 + let hooks: Hooks | undefined + try { + const input = createMockPluginInput({ + client: { + auth: { + set: async () => { + authSetCalls++ + if (authSetCalls < 3) throw new Error('temporary auth write') + }, + }, + session: { promptAsync: async () => {} }, + } as unknown as PluginInput['client'], + }) + const loaded = await loadFetchOverride(input, Date.now() - 60_000) + hooks = loaded.hooks + + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + expect(response.status).toBe(200) + await response.body?.cancel() + expect(oauthRefreshCalls).toBe(1) + expect(authSetCalls).toBe(3) + expect(seenAuth).toEqual(['Bearer main-refreshed-token']) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('surfaces a distinct auth persistence error after rotated tokens cannot be saved', async () => { + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + accounts: [], + refresh: { refreshBeforeExpiryMinutes: 5 }, + routing: { mode: 'main-first' }, + }), + ) + const originalFetch = globalThis.fetch + const seenAuth: string[] = [] + let oauthRefreshCalls = 0 + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('/oauth/token')) { + oauthRefreshCalls++ + return new Response( + JSON.stringify({ + access_token: 'main-refreshed-token', + refresh_token: 'main-refresh-new', + expires_in: 3600, + }), + { status: 200, headers: { 'content-type': 'application/json' } }, + ) + } + seenAuth.push(headerValue(init, 'authorization')) + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch + + let authSetCalls = 0 + let hooks: Hooks | undefined + try { + const input = createMockPluginInput({ + client: { + auth: { + set: async () => { + authSetCalls++ + throw new Error('auth write failed') + }, + }, + session: { promptAsync: async () => {} }, + } as unknown as PluginInput['client'], + }) + const loaded = await loadFetchOverride(input, Date.now() - 60_000) + hooks = loaded.hooks + + let caught: unknown + try { + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + } catch (error) { + caught = error + } + expect(caught).toBeInstanceOf(AuthPersistError) + expect((caught as AuthPersistError).code).toBe( + 'OPENAI_AUTH_PERSIST_FAILED', + ) + expect(oauthRefreshCalls).toBe(1) + expect(authSetCalls).toBe(3) + expect(seenAuth).toEqual([]) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) + + it('a fallback that stops sending a secondary window does not keep resurrecting it from cache', async () => { + seedStorage({ access: 'fallback-access-token' }) + const originalFetch = globalThis.fetch + const farFuture = Math.floor((Date.now() + 7 * 24 * 3600_000) / 1000) + let responseHeaders: Record = { + 'x-codex-primary-used-percent': '10', + 'x-codex-primary-window-minutes': '300', + 'x-codex-primary-reset-at': String(farFuture), + 'x-codex-secondary-used-percent': '20', + 'x-codex-secondary-window-minutes': '10080', + 'x-codex-secondary-reset-at': String(farFuture), + } + globalThis.fetch = (async () => { + return new Response('{}', { + status: 200, + headers: { 'content-type': 'application/json', ...responseHeaders }, + }) + }) as unknown as typeof globalThis.fetch + + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + ) + hooks = loaded.hooks + + // First push: real two-window frame — cache now holds both windows. + let response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + expect(response.status).toBe(200) + await response.body?.cancel() + + await waitForSidebarState( + sidebarFile, + (s) => + s.fallbacks.find((a) => a.id === 'fallback-1')?.quota?.secondary + ?.usedPercent === 20, + ) + + // The backend stops sending a secondary window entirely (the current + // live wire shape) — every subsequent push is single-primary. + responseHeaders = { + 'x-codex-primary-used-percent': '15', + 'x-codex-primary-window-minutes': '300', + 'x-codex-primary-reset-at': String(farFuture), + } + response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + expect(response.status).toBe(200) + await response.body?.cancel() + + response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + requestInit(), + ) + expect(response.status).toBe(200) + await response.body?.cancel() + + const sidebar = await waitForSidebarState( + sidebarFile, + (s) => + s.fallbacks.find((a) => a.id === 'fallback-1')?.quota?.primary + ?.usedPercent === 15, + ) + // The removed secondary must not be perpetuated from the stale cache. + expect( + sidebar.fallbacks.find((a) => a.id === 'fallback-1')?.quota?.secondary, + ).toBeUndefined() + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } }) - it('marks Responses Lite in WS metadata and still prewarms', async () => { - seedEmptyAccountStorage() - const sent: Array> = [] - let seenUpgradeHeaders: Record = {} - let hooks: Hooks | undefined - await withFakeWebSocket( - ({ message, upgradeHeaders }) => ({ - send(data) { - seenUpgradeHeaders = upgradeHeaders - const body = JSON.parse(data) as Record - sent.push(body) - message( - JSON.stringify({ - type: 'response.completed', - response: { - id: body.generate === false ? 'resp_prewarm' : 'resp_main', - }, - }), - ) + // --------------------------------------------------------------------------- + // Sticky-balanced + killswitch (the maintainer's blocker) + // --------------------------------------------------------------------------- + + // Seeds a sticky-balanced config with killswitch enabled. Every account's + // quota is parked near the floor so the killswitch rejects them. + function seedStickyBalancedKillswitchAllBelowFloor() { + const checkedAt = Date.now() + writeFileSync( + configFile, + JSON.stringify({ + version: 1, + main: { type: 'opencode', provider: 'openai' }, + routing: { mode: 'sticky-balanced' }, + refresh: { refreshBeforeExpiryMinutes: 5 }, + accounts: [ + { + id: 'fallback-1', + type: 'oauth', + enabled: true, + access: 'fallback-1-token', + refresh: 'fallback-1-refresh', + expires: Date.now() + 24 * 3600_000, + accountId: 'acc-fallback-1', + }, + { + id: 'fallback-2', + type: 'oauth', + enabled: true, + access: 'fallback-2-token', + refresh: 'fallback-2-refresh', + expires: Date.now() + 24 * 3600_000, + accountId: 'acc-fallback-2', + }, + ], + killswitch: { enabled: true, main: { primary: 50, secondary: 50 } }, + }), + ) + // All accounts sit between 0% and the 50% floor — the band the killswitch + // exists to protect. They are NOT exhausted (remainingPercent > 0), so + // today's break decision would retain them. + const belowFloor = (window: 'primary' | 'secondary', base: number) => ({ + usedPercent: 100 - base, + remainingPercent: base, + checkedAt, + resetsAt: new Date(checkedAt + 7 * 24 * 3600_000).toISOString(), + windowMinutes: window === 'primary' ? 300 : 10_080, + }) + writeFileSync( + sidebarFile, + JSON.stringify({ + main: { + quota: { + primary: belowFloor('primary', 40), + secondary: belowFloor('secondary', 40), + }, + mainAccountId: 'acc-main', + killed: false, }, + fallbacks: [ + { + id: 'fallback-1', + label: 'Fallback 1', + accountId: 'acc-fallback-1', + quota: { + primary: belowFloor('primary', 30), + secondary: belowFloor('secondary', 30), + }, + killed: false, + enabled: true, + }, + { + id: 'fallback-2', + label: 'Fallback 2', + accountId: 'acc-fallback-2', + quota: { + primary: belowFloor('primary', 35), + secondary: belowFloor('secondary', 35), + }, + killed: false, + enabled: true, + }, + ], + route: 'sticky-balanced', + lastUpdated: checkedAt, }), - async () => { - try { - const loaded = await loadFetchOverride( - createMockPluginInput(), - Date.now() + 3600_000, - true, - true, - ) - hooks = loaded.hooks - const response = await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - responsesLiteRequestInit('gpt-5.6-sol', 'responses-lite-ws', { - stream: true, - }), - ) - await response.text() - } finally { - await hooks?.dispose?.() - } - }, ) + } - expect(sent).toHaveLength(2) - expect(sent[0]?.generate).toBe(false) - const main = sent[1]! - expect( - (main.client_metadata as Record) - .ws_request_header_x_openai_internal_codex_responses_lite, - ).toBe('true') - expect( - seenUpgradeHeaders['x-openai-internal-codex-responses-lite'], - ).toBeUndefined() - }) - - it('converts a WS Responses Lite capture into a sanitized HTTP cachekeep request', async () => { - seedEmptyAccountStorage() + it('sticky-balanced + killswitch: every account below the floor returns the shared 429 and never reaches the wire', async () => { + seedStickyBalancedKillswitchAllBelowFloor() + // The kill check is peek-based (memory only). The response headers push + // below-floor quota for the account that served, so each request can only + // populate ONE account's memory. To verify the 429 on the test request + // we walk every account once, then send the test request — its roster + // is empty (all-killed) and the main path's killswitch block fires. + let fetchCalls = 0 const originalFetch = globalThis.fetch - const originalNow = Date.now - let now = originalNow() - const warmRequests: RequestInit[] = [] + globalThis.fetch = (async () => { + fetchCalls++ + return new Response('{}', { + status: 200, + headers: { + 'x-codex-primary-used-percent': '95', + 'x-codex-primary-window-minutes': '300', + 'x-codex-primary-reset-at': String( + Math.floor((Date.now() + 5 * 3600_000) / 1000), + ), + 'x-codex-secondary-used-percent': '95', + 'x-codex-secondary-window-minutes': '10080', + 'x-codex-secondary-reset-at': String( + Math.floor((Date.now() + 7 * 24 * 3600_000) / 1000), + ), + }, + }) + }) as unknown as typeof globalThis.fetch + let hooks: Hooks | undefined - Date.now = () => now try { - globalThis.fetch = (async (_url: unknown, init?: RequestInit) => { - if (init) warmRequests.push(init) - return new Response('{}', { status: 200 }) - }) as typeof globalThis.fetch - await withFakeWebSocket( - ({ message }) => ({ - send(data) { - const body = JSON.parse(data) as Record - message( - JSON.stringify({ - type: 'response.completed', - response: { - id: body.generate === false ? 'resp_prewarm' : 'resp_main', - }, - }), - ) - }, - }), - async () => { - const loaded = await loadFetchOverride( - createMockPluginInput(), - now + 3600_000, - true, - true, - ) - hooks = loaded.hooks - await runCommand(hooks, 'openai-cachekeep', 'on') - const request = responsesLiteRequestInit( - 'gpt-5.6-sol', - 'responses-lite-keepwarm', - { stream: true }, - ) - const headers = new Headers(request.headers) - headers.set('x-opencode-session', 'internal-session') - request.headers = headers - const response = await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - request, - ) - await response.text() - now += 30 * 60_000 - const manager = ( - globalThis as typeof globalThis & { - __openaiAuthCacheKeepManager?: { tick(): Promise } - } - ).__openaiAuthCacheKeepManager - if (!manager) throw new Error('missing cachekeep manager') - await manager.tick() - }, + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + false, + false, + 'acc-main', + ) + hooks = loaded.hooks + // Setup: each request cycles through the remaining accounts (the + // killswitch filter excludes the most-recently-killed one). After all + // three, every account has below-floor quota in memory. + for (let i = 0; i < 3; i++) { + const response = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ + 'x-session-affinity': `all-killed-spend-${i}`, + }), + ) + expect(response.status).toBe(200) + } + const setupCalls = fetchCalls + expect(setupCalls).toBe(3) + + // Test request: the roster is empty (all-killed), the sticky path + // returns undefined, and the main path returns the shared 429. + const test = await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'x-session-affinity': 'all-killed-block' }), ) + expect(test.status).toBe(429) + expect(test.headers.get('retry-after')).toBeTruthy() + const body = (await test.json()) as { + error?: { type?: string; message?: string } + } + expect(body.error?.type).toBe('rate_limit_exceeded') + expect(body.error?.message).toContain('Killswitch') + + // The blocked request did NOT reach upstream. + expect(fetchCalls).toBe(setupCalls) } finally { - Date.now = originalNow globalThis.fetch = originalFetch await hooks?.dispose?.() } - - expect(warmRequests).toHaveLength(1) - const warm = warmRequests[0]! - const warmHeaders = new Headers(warm.headers) - expect(warmHeaders.get('x-openai-internal-codex-responses-lite')).toBe( - 'true', - ) - expect(warmHeaders.has('x-opencode-session')).toBe(false) - const body = JSON.parse(String(warm.body)) as Record - const metadata = body.client_metadata as Record - expect('x-codex-turn-metadata' in metadata).toBe(false) - expect('x-codex-ws-stream-request-start-ms' in metadata).toBe(false) - expect( - 'ws_request_header_x_openai_internal_codex_responses_lite' in metadata, - ).toBe(false) - }) - - it('keeps the main refresh advisory lease shorter than the file lock TTL', () => { - expect(MAIN_REFRESH_LEASE_TTL_MS).toBe(90_000) - expect(MAIN_REFRESH_LEASE_TTL_MS).toBeLessThan(MAIN_REFRESH_LOCK_TTL_MS) }) - it('retries persisting rotated main tokens without refreshing twice', async () => { + it('sticky-balanced + killswitch: a retained pin migrates off an account that drops below the floor', async () => { + // Healthy at pin time, then drops below the floor — the band the killswitch + // exists to protect. The break decision must migrate the pin. + seedStickyBalancedAccounts() writeFileSync( configFile, JSON.stringify({ version: 1, main: { type: 'opencode', provider: 'openai' }, - accounts: [], + routing: { mode: 'sticky-balanced' }, refresh: { refreshBeforeExpiryMinutes: 5 }, - routing: { mode: 'main-first' }, + accounts: [ + { + id: 'fallback-1', + type: 'oauth', + enabled: true, + access: 'fallback-1-token', + refresh: 'fallback-1-refresh', + expires: Date.now() + 24 * 3600_000, + accountId: 'acc-fallback-1', + }, + { + id: 'fallback-2', + type: 'oauth', + enabled: true, + access: 'fallback-2-token', + refresh: 'fallback-2-refresh', + expires: Date.now() + 24 * 3600_000, + accountId: 'acc-fallback-2', + }, + ], + killswitch: { enabled: true, main: { primary: 50, secondary: 50 } }, }), ) - const originalFetch = globalThis.fetch const seenAuth: string[] = [] - let oauthRefreshCalls = 0 + const originalFetch = globalThis.fetch globalThis.fetch = (async (url: unknown, init?: unknown) => { - if (String(url).includes('/oauth/token')) { - oauthRefreshCalls++ - return new Response( - JSON.stringify({ - access_token: 'main-refreshed-token', - refresh_token: 'main-refresh-new', - expires_in: 3600, - }), - { status: 200, headers: { 'content-type': 'application/json' } }, - ) + if (String(url).includes('responses')) { + seenAuth.push(headerValue(init, 'authorization')) + const auth = headerValue(init, 'authorization') + // Push below-floor quota ONLY for the pinned account (fallback-2) so + // the peek-based kill check trips on the next request. + if (auth.includes('fallback-2-token')) { + return new Response('{}', { + status: 200, + headers: { + 'x-codex-primary-used-percent': '95', + 'x-codex-primary-window-minutes': '300', + 'x-codex-primary-reset-at': String( + Math.floor((Date.now() + 5 * 3600_000) / 1000), + ), + 'x-codex-secondary-used-percent': '95', + 'x-codex-secondary-window-minutes': '10080', + 'x-codex-secondary-reset-at': String( + Math.floor((Date.now() + 7 * 24 * 3600_000) / 1000), + ), + }, + }) + } } - seenAuth.push(headerValue(init, 'authorization')) return new Response('{}', { status: 200 }) }) as unknown as typeof globalThis.fetch - let authSetCalls = 0 let hooks: Hooks | undefined try { - const input = createMockPluginInput({ - client: { - auth: { - set: async () => { - authSetCalls++ - if (authSetCalls < 3) throw new Error('temporary auth write') - }, - }, - session: { promptAsync: async () => {} }, - } as unknown as PluginInput['client'], - }) - const loaded = await loadFetchOverride(input, Date.now() - 60_000) + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + false, + false, + 'acc-main', + ) hooks = loaded.hooks + // First request: sticky places the session on fallback-2 (roomiest + // account). The response headers push below-floor quota for fallback-2. + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'x-session-affinity': 'pin-migrate-session' }), + ) + await drainSidebarWrites() + const initialPin = normalizeSidebarState( + JSON.parse(readFileSync(sidebarFile, 'utf8')), + ).stickyAssignments?.[hashSidebarSessionId('pin-migrate-session')] + ?.accountId + expect(initialPin).toBe('fallback-2') - const response = await loaded.fetchOverride( + // Second request: kill filter excludes fallback-2 (memory has below-floor + // quota). The pin migrates to fallback-1. + const next = await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + responseRequestInit({ 'x-session-affinity': 'pin-migrate-session' }), ) - expect(response.status).toBe(200) - await response.body?.cancel() - expect(oauthRefreshCalls).toBe(1) - expect(authSetCalls).toBe(3) - expect(seenAuth).toEqual(['Bearer main-refreshed-token']) + expect(next.status).toBe(200) + expect(seenAuth).toEqual([ + 'Bearer fallback-2-token', + 'Bearer fallback-1-token', + ]) + await drainSidebarWrites() + expect( + normalizeSidebarState(JSON.parse(readFileSync(sidebarFile, 'utf8'))) + .stickyAssignments?.[hashSidebarSessionId('pin-migrate-session')] + ?.accountId, + ).toBe('fallback-1') } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('surfaces a distinct auth persistence error after rotated tokens cannot be saved', async () => { + it('sticky-balanced + killswitch: a killed account is never picked by the mode-fallback fail-open branch', async () => { + // The "subtle half" from the brief: with every quota stale the mode-fallback + // branch is the only branch that runs. It must never spend on a killed + // account. Pin to the killed account first (so its memory has below-floor + // quota), then verify the next request does NOT pick it (the kill filter + // excludes it from the mode-fallback fail-open). + seedStickyBalancedAccounts() writeFileSync( configFile, JSON.stringify({ version: 1, main: { type: 'opencode', provider: 'openai' }, - accounts: [], + routing: { mode: 'sticky-balanced' }, refresh: { refreshBeforeExpiryMinutes: 5 }, - routing: { mode: 'main-first' }, + accounts: [ + { + id: 'fallback-1', + type: 'oauth', + enabled: true, + access: 'fallback-1-token', + refresh: 'fallback-1-refresh', + expires: Date.now() + 24 * 3600_000, + accountId: 'acc-fallback-1', + }, + { + id: 'fallback-2', + type: 'oauth', + enabled: true, + access: 'fallback-2-token', + refresh: 'fallback-2-refresh', + expires: Date.now() + 24 * 3600_000, + accountId: 'acc-fallback-2', + }, + ], + killswitch: { enabled: true, main: { primary: 50, secondary: 50 } }, }), ) - const originalFetch = globalThis.fetch const seenAuth: string[] = [] - let oauthRefreshCalls = 0 + const originalFetch = globalThis.fetch globalThis.fetch = (async (url: unknown, init?: unknown) => { - if (String(url).includes('/oauth/token')) { - oauthRefreshCalls++ - return new Response( - JSON.stringify({ - access_token: 'main-refreshed-token', - refresh_token: 'main-refresh-new', - expires_in: 3600, - }), - { status: 200, headers: { 'content-type': 'application/json' } }, - ) + if (String(url).includes('responses')) { + seenAuth.push(headerValue(init, 'authorization')) } - seenAuth.push(headerValue(init, 'authorization')) - return new Response('{}', { status: 200 }) + // First request: get below-floor quota into memory for the chosen + // account. Subsequent requests: return whatever quota the caller pushes. + return new Response('{}', { + status: 200, + headers: { + 'x-codex-primary-used-percent': '95', + 'x-codex-primary-window-minutes': '300', + 'x-codex-primary-reset-at': String( + Math.floor((Date.now() + 5 * 3600_000) / 1000), + ), + 'x-codex-secondary-used-percent': '95', + 'x-codex-secondary-window-minutes': '10080', + 'x-codex-secondary-reset-at': String( + Math.floor((Date.now() + 7 * 24 * 3600_000) / 1000), + ), + }, + }) }) as unknown as typeof globalThis.fetch - let authSetCalls = 0 let hooks: Hooks | undefined try { - const input = createMockPluginInput({ - client: { - auth: { - set: async () => { - authSetCalls++ - throw new Error('auth write failed') - }, - }, - session: { promptAsync: async () => {} }, - } as unknown as PluginInput['client'], - }) - const loaded = await loadFetchOverride(input, Date.now() - 60_000) + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + false, + false, + 'acc-main', + ) hooks = loaded.hooks - - let caught: unknown - try { - await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - requestInit(), - ) - } catch (error) { - caught = error - } - expect(caught).toBeInstanceOf(AuthPersistError) - expect((caught as AuthPersistError).code).toBe( - 'OPENAI_AUTH_PERSIST_FAILED', + // Setup: pin a session to whichever account the sticky path picks. + // The response headers push below-floor quota into memory for that + // account. + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ 'x-session-affinity': 'killswitch-pin-setup' }), ) - expect(oauthRefreshCalls).toBe(1) - expect(authSetCalls).toBe(3) - expect(seenAuth).toEqual([]) + const pinnedId = (seenAuth[0] ?? '') + .replace('Bearer ', '') + .replace('-token', '') + await drainSidebarWrites() + // Now make all quotas stale so the mode-fallback is the only branch. + const stale = JSON.parse(readFileSync(sidebarFile, 'utf8')) + stale.main.quota = stickyQuota(100, Date.now() - QUOTA_STALENESS_MS - 1) + stale.fallbacks[0].quota = stickyQuota( + 100, + Date.now() - QUOTA_STALENESS_MS - 1, + ) + stale.fallbacks[1].quota = stickyQuota( + 100, + Date.now() - QUOTA_STALENESS_MS - 1, + ) + writeFileSync(sidebarFile, JSON.stringify(stale)) + + // Test: the pinned account's memory has below-floor quota. The kill + // filter MUST exclude it. Without the filter, mode-fallback would pick + // the same account (the pin is honored when the account is still in + // the candidates list). + await loaded.fetchOverride( + 'https://api.openai.com/v1/responses', + responseRequestInit({ + 'x-session-affinity': 'killswitch-mode-fallback', + }), + ) + + // Either the pin migrates (kill filter excludes the pinned account) OR + // the mode-fallback picks a different account. The key assertion is + // that the killed account is NOT picked. + expect(seenAuth[1]).not.toBe(`Bearer ${pinnedId}-token`) } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() } }) - it('a fallback that stops sending a secondary window does not keep resurrecting it from cache', async () => { - seedStorage({ access: 'fallback-access-token' }) + it('sticky-balanced + killswitch DISABLED: placement and retention are byte-identical (no-op)', async () => { + // Load-bearing negative case: the dominant path with killswitch off must + // be unchanged. Even with one account near zero, the request goes through. + seedStickyBalancedAccounts() + await drainSidebarWrites() + const state = JSON.parse(readFileSync(sidebarFile, 'utf8')) + // fallback-1 below the floor; killswitch is OFF so it must still be served. + state.fallbacks[0].quota = stickyQuota(20, Date.now()) + writeFileSync(sidebarFile, JSON.stringify(state)) + + const seenAuth: string[] = [] const originalFetch = globalThis.fetch - const farFuture = Math.floor((Date.now() + 7 * 24 * 3600_000) / 1000) - let responseHeaders: Record = { - 'x-codex-primary-used-percent': '10', - 'x-codex-primary-window-minutes': '300', - 'x-codex-primary-reset-at': String(farFuture), - 'x-codex-secondary-used-percent': '20', - 'x-codex-secondary-window-minutes': '10080', - 'x-codex-secondary-reset-at': String(farFuture), - } - globalThis.fetch = (async () => { - return new Response('{}', { - status: 200, - headers: { 'content-type': 'application/json', ...responseHeaders }, - }) + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('responses')) { + seenAuth.push(headerValue(init, 'authorization')) + } + return new Response('{}', { status: 200 }) }) as unknown as typeof globalThis.fetch let hooks: Hooks | undefined @@ -5496,55 +7217,106 @@ describe('integration: active fallback routing', () => { const loaded = await loadFetchOverride( createMockPluginInput(), Date.now() + 3600_000, + false, + false, + 'acc-main', ) hooks = loaded.hooks - - // First push: real two-window frame — cache now holds both windows. - let response = await loaded.fetchOverride( + await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + responseRequestInit({ + 'x-session-affinity': 'killswitch-disabled-session', + }), ) - expect(response.status).toBe(200) - await response.body?.cancel() - await waitForSidebarState( - sidebarFile, - (s) => - s.fallbacks.find((a) => a.id === 'fallback-1')?.quota?.secondary - ?.usedPercent === 20, - ) + // Killswitch disabled — the dominant path. Weighted placement behaves + // identically to the pre-killswitch implementation. The selection picks + // the roomiest account; this is the load-bearing negative case. + expect(seenAuth).toHaveLength(1) + expect(seenAuth[0]).toMatch(/Bearer fallback-[12]-token/) + } finally { + globalThis.fetch = originalFetch + await hooks?.dispose?.() + } + }) - // The backend stops sending a secondary window entirely (the current - // live wire shape) — every subsequent push is single-primary. - responseHeaders = { - 'x-codex-primary-used-percent': '15', - 'x-codex-primary-window-minutes': '300', - 'x-codex-primary-reset-at': String(farFuture), + // --------------------------------------------------------------------------- + // resetCreditsAvailable → resetCreditsApplicable wiring (the real path) + // --------------------------------------------------------------------------- + // The shotgun unit test (`prefers a positive optional reset-credit count in + // empty-set fallback` in sticky-routing.test.ts) bypasses the extractor at + // index.ts:1962 by setting `resetCreditsApplicable` directly on the candidate. + // The extractor was reading the WRONG key — `.resetCreditsApplicable` instead + // of the real field `.resetCreditsAvailable` — so the wiring was dead in + // production. This test goes through the sidebar file → roster builder → sort + // pipeline and would fail until the extractor reads the right key. + + it('sticky-balanced: mode-fallback prefers the credit-bearing account via the REAL wiring (resetCreditsAvailable)', async () => { + seedStickyBalancedAccounts() + await drainSidebarWrites() + const state = JSON.parse(readFileSync(sidebarFile, 'utf8')) + // All credentials stale → mode-fallback is the only branch that runs. + // configuredOrder says fallback-1 wins (added first → configuredOrder 1). + // resetCreditsAvailable says fallback-2 wins (higher credit priority). + // The fix that makes this test green is the extractor reading + // `resetCreditsAvailable` from the quota — the sort then picks fallback-2. + state.main.quota = stickyQuota(100, Date.now() - QUOTA_STALENESS_MS - 1) + state.fallbacks[0].quota = { + primary: { + usedPercent: 0, + remainingPercent: 100, + checkedAt: Date.now() - QUOTA_STALENESS_MS - 1, + windowMinutes: 300, + }, + resetCreditsAvailable: 0, + } + state.fallbacks[1].quota = { + primary: { + usedPercent: 0, + remainingPercent: 100, + checkedAt: Date.now() - QUOTA_STALENESS_MS - 1, + windowMinutes: 300, + }, + resetCreditsAvailable: 1, + } + writeFileSync(sidebarFile, JSON.stringify(state)) + + const seenAuth: string[] = [] + const originalFetch = globalThis.fetch + globalThis.fetch = (async (url: unknown, init?: unknown) => { + if (String(url).includes('responses')) { + seenAuth.push(headerValue(init, 'authorization')) } - response = await loaded.fetchOverride( - 'https://api.openai.com/v1/responses', - requestInit(), - ) - expect(response.status).toBe(200) - await response.body?.cancel() + return new Response('{}', { status: 200 }) + }) as unknown as typeof globalThis.fetch - response = await loaded.fetchOverride( + let hooks: Hooks | undefined + try { + const loaded = await loadFetchOverride( + createMockPluginInput(), + Date.now() + 3600_000, + false, + false, + 'acc-main', + ) + hooks = loaded.hooks + await loaded.fetchOverride( 'https://api.openai.com/v1/responses', - requestInit(), + responseRequestInit({ + 'x-session-affinity': 'credit-priority-session', + }), ) - expect(response.status).toBe(200) - await response.body?.cancel() - const sidebar = await waitForSidebarState( - sidebarFile, - (s) => - s.fallbacks.find((a) => a.id === 'fallback-1')?.quota?.primary - ?.usedPercent === 15, - ) - // The removed secondary must not be perpetuated from the stale cache. - expect( - sidebar.fallbacks.find((a) => a.id === 'fallback-1')?.quota?.secondary, - ).toBeUndefined() + // The mode-fallback sort is + // resetCreditsApplicable DESC → configuredOrder ASC → id ASC. + // Without the fix: the extractor returns undefined for both accounts + // (it reads the wrong key). The sort falls through to configuredOrder, + // and fallback-1 wins by its lower configuredOrder. + // With the fix: the extractor reads `resetCreditsAvailable=1` from + // fallback-2's quota and sets `resetCreditsApplicable=1` on its + // candidate. fallback-1 has `resetCreditsAvailable=0`. The sort picks + // fallback-2. + expect(seenAuth[0]).toBe('Bearer fallback-2-token') } finally { globalThis.fetch = originalFetch await hooks?.dispose?.() diff --git a/packages/opencode/src/tests/rpc-client.test.ts b/packages/opencode/src/tests/rpc-client.test.ts new file mode 100644 index 0000000..d345a11 --- /dev/null +++ b/packages/opencode/src/tests/rpc-client.test.ts @@ -0,0 +1,53 @@ +import { afterEach, expect, test } from 'bun:test' +import { mkdtemp, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { drainNotifications } from '../rpc/notifications' +import { createRpcClient } from '../rpc/rpc-client' +import { type RpcServerHandle, startRpcServer } from '../rpc/rpc-server' + +let dir: string | undefined +let server: RpcServerHandle | undefined + +afterEach(async () => { + if (server) { + await server.stop() + server = undefined + } + if (dir) { + await rm(dir, { recursive: true, force: true }) + dir = undefined + } +}) + +test('RPC client preserves sessionId through the server apply callback', async () => { + dir = await mkdtemp(join(tmpdir(), 'oa-rpcclient-')) + const received: Array<{ sessionId?: string }> = [] + server = await startRpcServer({ + dir, + drain: drainNotifications, + apply: async (request) => { + received.push(request) + return { text: 'ok', knobs: {} } + }, + }) + + const client = createRpcClient(dir, process.pid) + await Promise.all([ + client.apply({ + command: 'openai-routing', + arguments: 'reset', + sessionId: 'session-a', + }), + client.apply({ + command: 'openai-routing', + arguments: 'reset', + sessionId: 'session-b', + }), + ]) + + expect(received.map((request) => request.sessionId).sort()).toEqual([ + 'session-a', + 'session-b', + ]) +}) diff --git a/packages/opencode/src/tests/rpc-server.test.ts b/packages/opencode/src/tests/rpc-server.test.ts index feaa387..3448a07 100644 --- a/packages/opencode/src/tests/rpc-server.test.ts +++ b/packages/opencode/src/tests/rpc-server.test.ts @@ -21,13 +21,17 @@ afterEach(async () => { }) describe('rpc-server', () => { - test('health is open; pending-notifications requires bearer and drains', async () => { + test('apply callback receives sessionId unchanged; health is open and pending-notifications drains', async () => { resetNotificationsForTest() dir = await mkdtemp(join(tmpdir(), 'oa-rpcsrv-')) + let receivedApply: unknown const server = await startRpcServer({ dir, drain: drainNotifications, - apply: async () => ({ text: 'ok', knobs: {} }), + apply: async (request) => { + receivedApply = request + return { text: 'ok', knobs: {} } + }, }) stop = server.stop const base = `http://127.0.0.1:${server.port}` @@ -69,10 +73,19 @@ describe('rpc-server', () => { 'content-type': 'application/json', authorization: `Bearer ${server.token}`, }, - body: JSON.stringify({ command: 'openai-quota', arguments: '' }), + body: JSON.stringify({ + command: 'openai-routing', + arguments: 'reset', + sessionId: 'session-a', + }), }) expect(applyOk.status).toBe(200) expect(await applyOk.json()).toEqual({ text: 'ok', knobs: {} }) + expect(receivedApply).toEqual({ + command: 'openai-routing', + arguments: 'reset', + sessionId: 'session-a', + }) }) test('rejects body exceeding 1 MB byte limit', async () => { diff --git a/packages/opencode/src/tests/sidebar-state.test.ts b/packages/opencode/src/tests/sidebar-state.test.ts index 9576ca9..f3a2205 100644 --- a/packages/opencode/src/tests/sidebar-state.test.ts +++ b/packages/opencode/src/tests/sidebar-state.test.ts @@ -1,12 +1,22 @@ import { describe, expect, test } from 'bun:test' -import { existsSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs' +import { + chmodSync, + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + statSync, + writeFileSync, +} from 'node:fs' import { tmpdir } from 'node:os' -import { join } from 'node:path' +import { dirname, join } from 'node:path' import { acquireRefreshFileLock } from '../core/refresh-file-lock' +import { flushForTest, setLogLevel } from '../logger' import { ACTIVE_ROUTING_MAX_AGE_MS, type AccountQuota, + clearSidebarStickyAssignment, computeQuotaPacing, DEFAULT_SIDEBAR_STATE, drainSidebarWrites, @@ -16,14 +26,19 @@ import { getPresentQuotaWindows, getSidebarState, getSidebarStateFile, + hashSidebarSessionId, isQuotaExhausted, isUsableRoutingEntry, normalizeSidebarState, pruneActiveRouting, + pruneStickyAssignments, removeSidebarActiveRouting, resolveActiveAccount, + resolveSessionSidebarRouting, + resolveSidebarStickyAssignment, type SidebarAccountState, type SidebarState, + STICKY_ASSIGNMENT_MAX_ENTRIES, setSidebarLegacyRouting, setSidebarMachineState, setSidebarState, @@ -352,6 +367,878 @@ describe('normalizeSidebarState', () => { expect(result.activeId).toBe('fallback-1') expect(result.route).toBe('fallback-first') }) + + test('normalizes sticky assignments without retaining malformed siblings', () => { + const result = normalizeSidebarState({ + ...DEFAULT_SIDEBAR_STATE, + stickyAssignments: { + [hashSidebarSessionId('valid-session')]: { + accountId: 'fallback-1', + assignedAt: 100, + lastSeenAt: 200, + inputBytes: 300, + quotaCheckedAt: 400, + }, + missingAccount: { assignedAt: 100, lastSeenAt: 200, inputBytes: 300 }, + invalidTimestamp: { + accountId: 'fallback-1', + assignedAt: Number.NaN, + lastSeenAt: 200, + inputBytes: 300, + }, + negativeBytes: { + accountId: 'fallback-1', + assignedAt: 100, + lastSeenAt: 200, + inputBytes: -1, + }, + }, + }) + + expect(result.stickyAssignments).toEqual({ + [hashSidebarSessionId('valid-session')]: { + accountId: 'fallback-1', + assignedAt: 100, + lastSeenAt: 200, + inputBytes: 300, + quotaCheckedAt: 400, + }, + }) + }) + + test('old files without sticky assignments remain valid', () => { + expect( + normalizeSidebarState(DEFAULT_SIDEBAR_STATE).stickyAssignments, + ).toBeUndefined() + }) +}) + +describe('sticky assignments', () => { + const now = 2 * 7 * 24 * 60 * 60 * 1000 + + test('hashes session ids without retaining the source identifier', () => { + expect(hashSidebarSessionId('session-a')).toMatch(/^[a-f0-9]{64}$/) + expect(hashSidebarSessionId('session-a')).not.toContain('session-a') + expect(hashSidebarSessionId('session-a')).toBe( + hashSidebarSessionId('session-a'), + ) + }) + + test('unknown roster keeps fresh fallback pins while expiry and explicit removal still prune', () => { + const result = pruneStickyAssignments( + { + freshFallback: { + accountId: 'fallback-1', + assignedAt: now, + lastSeenAt: now, + inputBytes: 1, + }, + expired: { + accountId: 'fallback-2', + assignedAt: 1, + lastSeenAt: now - 7 * 24 * 60 * 60 * 1000 - 1, + inputBytes: 2, + }, + explicitlyRemoved: { + accountId: 'fallback-3', + assignedAt: now, + lastSeenAt: now, + inputBytes: 3, + }, + }, + undefined, + now, + 'explicitlyRemoved', + ) + + expect(result).toEqual({ + freshFallback: { + accountId: 'fallback-1', + assignedAt: now, + lastSeenAt: now, + inputBytes: 1, + }, + }) + }) + + test('prunes expired, disabled, and explicitly removed assignments', () => { + const result = pruneStickyAssignments( + { + stale: { + accountId: 'main', + assignedAt: 1, + lastSeenAt: now - 7 * 24 * 60 * 60 * 1000 - 1, + inputBytes: 1, + }, + disabled: { + accountId: 'disabled-fallback', + assignedAt: now, + lastSeenAt: now, + inputBytes: 2, + }, + removed: { + accountId: 'main', + assignedAt: now, + lastSeenAt: now, + inputBytes: 3, + }, + keep: { + accountId: 'main', + assignedAt: now, + lastSeenAt: now, + inputBytes: 4, + }, + }, + new Set(['main']), + now, + 'removed', + ) + + expect(result).toEqual({ + keep: { + accountId: 'main', + assignedAt: now, + lastSeenAt: now, + inputBytes: 4, + }, + }) + }) + + test('logs pruned sticky assignments by reason without raw session ids', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-prune-log-')) + const logFile = join(tempDir, 'sidebar.log') + const originalLogFile = process.env.OPENCODE_OPENAI_AUTH_LOG_FILE + await flushForTest() + process.env.OPENCODE_OPENAI_AUTH_LOG_FILE = logFile + setLogLevel('debug') + + try { + pruneStickyAssignments( + { + 'raw-session-account': { + accountId: 'removed-fallback', + assignedAt: now, + lastSeenAt: now, + inputBytes: 1, + }, + 'raw-session-expired': { + accountId: 'main', + assignedAt: 1, + lastSeenAt: now - 7 * 24 * 60 * 60 * 1000 - 1, + inputBytes: 2, + }, + 'raw-session-explicit': { + accountId: 'main', + assignedAt: now, + lastSeenAt: now, + inputBytes: 3, + }, + }, + new Set(['main']), + now, + 'raw-session-explicit', + ) + await flushForTest() + + const text = readFileSync(logFile, 'utf8') + expect(text).toContain('[sidebar] pruned sticky assignments') + expect(text).toContain('"removed":3') + expect(text).toContain('"account-not-in-roster":1') + expect(text).toContain('"expired":1') + expect(text).toContain('"explicit-removal":1') + expect(text).not.toContain('raw-session-') + } finally { + await flushForTest() + setLogLevel(undefined) + if (originalLogFile === undefined) { + delete process.env.OPENCODE_OPENAI_AUTH_LOG_FILE + } else { + process.env.OPENCODE_OPENAI_AUTH_LOG_FILE = originalLogFile + } + } + }) + + test('returns a fresh valid sticky assignment without choosing', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-sticky-resolve-')) + const file = join(tempDir, 'sidebar-state.json') + const sessionId = 'fresh-session' + const assignment = { + accountId: 'account-a', + assignedAt: now - 1, + lastSeenAt: now, + inputBytes: 128, + quotaCheckedAt: 10, + } + await setSidebarState( + make({ + stickyAssignments: { [hashSidebarSessionId(sessionId)]: assignment }, + lastUpdated: now, + }), + file, + ) + + const result = await resolveSidebarStickyAssignment( + { + sessionId, + requestBytes: 128, + now, + validPinnedAccountIds: ['account-a'], + quotaCheckedAtByAccount: { 'account-a': 10 }, + choose: () => { + throw new Error('choose must not run for a fresh valid assignment') + }, + }, + file, + ) + + expect(result).toEqual(assignment) + }) + + test('replaces excluded, expired, and disabled sticky assignments', async () => { + const cases = [ + { + name: 'excluded', + lastSeenAt: now, + validPinnedAccountIds: ['account-a', 'account-b'], + excludeAccountIds: ['account-a'], + }, + { + name: 'expired', + lastSeenAt: now - SEVEN_DAY_MS - 1, + validPinnedAccountIds: ['account-a', 'account-b'], + }, + { + name: 'disabled', + lastSeenAt: now, + validPinnedAccountIds: ['account-b'], + }, + ] + + for (const scenario of cases) { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-sticky-replace-')) + const file = join(tempDir, 'sidebar-state.json') + const sessionId = `${scenario.name}-session` + await setSidebarState( + make({ + stickyAssignments: { + [hashSidebarSessionId(sessionId)]: { + accountId: 'account-a', + assignedAt: now - 100, + lastSeenAt: scenario.lastSeenAt, + inputBytes: 100, + quotaCheckedAt: 10, + }, + }, + lastUpdated: now, + }), + file, + ) + let chooseCalls = 0 + + const result = await resolveSidebarStickyAssignment( + { + sessionId, + requestBytes: 200, + now, + validPinnedAccountIds: scenario.validPinnedAccountIds, + ...(scenario.excludeAccountIds + ? { excludeAccountIds: scenario.excludeAccountIds } + : {}), + quotaCheckedAtByAccount: { 'account-a': 10, 'account-b': 20 }, + choose: () => { + chooseCalls += 1 + return { accountId: 'account-b', quotaCheckedAt: 20 } + }, + }, + file, + ) + + expect(chooseCalls).toBe(1) + expect(result).toEqual({ + accountId: 'account-b', + assignedAt: now, + lastSeenAt: now, + inputBytes: 200, + quotaCheckedAt: 20, + }) + } + }) + + test('evicts the least recently seen assignment when adding beyond the sticky cap', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-sticky-cap-')) + const file = join(tempDir, 'sidebar-state.json') + const sessionId = 'current-session' + const entries = Object.fromEntries( + Array.from({ length: STICKY_ASSIGNMENT_MAX_ENTRIES }, (_, index) => { + const entrySessionId = `existing-${index}` + return [ + hashSidebarSessionId(entrySessionId), + { + accountId: 'account-a', + assignedAt: now - index, + lastSeenAt: now - index, + inputBytes: 1, + }, + ] + }), + ) + await setSidebarState(make({ stickyAssignments: entries }), file) + + await resolveSidebarStickyAssignment( + { + sessionId, + requestBytes: 10, + now, + validPinnedAccountIds: ['account-a', 'account-b'], + quotaCheckedAtByAccount: { 'account-a': 10, 'account-b': 20 }, + choose: () => ({ accountId: 'account-b', quotaCheckedAt: 20 }), + }, + file, + ) + + const assignments = (await getSidebarState(file)).stickyAssignments + expect(Object.keys(assignments ?? {})).toHaveLength( + STICKY_ASSIGNMENT_MAX_ENTRIES, + ) + expect(assignments?.[hashSidebarSessionId('existing-255')]).toBeUndefined() + expect(assignments?.[hashSidebarSessionId(sessionId)]).toMatchObject({ + accountId: 'account-b', + lastSeenAt: now, + }) + }) + + test('does not lower sticky assignment input-byte high water', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-sticky-high-water-')) + const file = join(tempDir, 'sidebar-state.json') + const sessionId = 'high-water-session' + await setSidebarState( + make({ + stickyAssignments: { + [hashSidebarSessionId(sessionId)]: { + accountId: 'account-a', + assignedAt: now - 1, + lastSeenAt: now, + inputBytes: 200, + quotaCheckedAt: 10, + }, + }, + }), + file, + ) + + await resolveSidebarStickyAssignment( + { + sessionId, + requestBytes: 100, + now, + validPinnedAccountIds: ['account-a'], + quotaCheckedAtByAccount: { 'account-a': 10 }, + choose: () => { + throw new Error('choose must not run for a valid assignment') + }, + }, + file, + ) + + expect( + (await getSidebarState(file)).stickyAssignments?.[ + hashSidebarSessionId(sessionId) + ]?.inputBytes, + ).toBe(200) + }) + + test('raises sticky assignment input-byte high water', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-sticky-high-water-')) + const file = join(tempDir, 'sidebar-state.json') + const sessionId = 'raise-high-water-session' + await setSidebarState( + make({ + stickyAssignments: { + [hashSidebarSessionId(sessionId)]: { + accountId: 'account-a', + assignedAt: now - 1, + lastSeenAt: now, + inputBytes: 100, + quotaCheckedAt: 10, + }, + }, + }), + file, + ) + + const result = await resolveSidebarStickyAssignment( + { + sessionId, + requestBytes: 200, + now, + validPinnedAccountIds: ['account-a'], + quotaCheckedAtByAccount: { 'account-a': 10 }, + choose: () => { + throw new Error('choose must not run for a valid assignment') + }, + }, + file, + ) + + expect(result?.inputBytes).toBe(200) + expect( + (await getSidebarState(file)).stickyAssignments?.[ + hashSidebarSessionId(sessionId) + ]?.inputBytes, + ).toBe(200) + }) + + test('touches sticky assignment last-seen time only after one hour', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-sticky-last-seen-')) + const file = join(tempDir, 'sidebar-state.json') + const sessionId = 'last-seen-session' + const initialLastSeenAt = now - 30 * 60 * 1000 + await setSidebarState( + make({ + stickyAssignments: { + [hashSidebarSessionId(sessionId)]: { + accountId: 'account-a', + assignedAt: now - 1, + lastSeenAt: initialLastSeenAt, + inputBytes: 100, + quotaCheckedAt: 10, + }, + }, + }), + file, + ) + + await resolveSidebarStickyAssignment( + { + sessionId, + requestBytes: 100, + now, + validPinnedAccountIds: ['account-a'], + quotaCheckedAtByAccount: { 'account-a': 10 }, + choose: () => { + throw new Error('choose must not run for a valid assignment') + }, + }, + file, + ) + expect( + (await getSidebarState(file)).stickyAssignments?.[ + hashSidebarSessionId(sessionId) + ]?.lastSeenAt, + ).toBe(initialLastSeenAt) + + const afterOneHour = now + 31 * 60 * 1000 + await resolveSidebarStickyAssignment( + { + sessionId, + requestBytes: 100, + now: afterOneHour, + validPinnedAccountIds: ['account-a'], + quotaCheckedAtByAccount: { 'account-a': 10 }, + choose: () => { + throw new Error('choose must not run for a valid assignment') + }, + }, + file, + ) + expect( + (await getSidebarState(file)).stickyAssignments?.[ + hashSidebarSessionId(sessionId) + ]?.lastSeenAt, + ).toBe(afterOneHour) + }) + + test('clears only one hashed sticky assignment', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-sticky-clear-')) + const file = join(tempDir, 'sidebar-state.json') + const sessionId = 'session-to-clear' + const otherSessionId = 'session-to-keep' + await setSidebarState( + make({ + activeRouting: { + routing: { activeId: 'main', route: 'main-first', updatedAt: now }, + }, + stickyAssignments: { + [hashSidebarSessionId(sessionId)]: { + accountId: 'account-a', + assignedAt: now, + lastSeenAt: now, + inputBytes: 100, + }, + [hashSidebarSessionId(otherSessionId)]: { + accountId: 'account-b', + assignedAt: now, + lastSeenAt: now, + inputBytes: 200, + }, + }, + lastUpdated: now, + }), + file, + ) + const before = await getSidebarState(file) + + expect(await clearSidebarStickyAssignment(sessionId, file)).toBe(true) + const after = await getSidebarState(file) + const { + stickyAssignments: _beforeAssignments, + lastUpdated: _beforeUpdated, + ...beforeRest + } = before + const { stickyAssignments, lastUpdated, ...afterRest } = after + expect(afterRest).toEqual(beforeRest) + expect(lastUpdated).toBeGreaterThan(before.lastUpdated) + expect(stickyAssignments).toEqual({ + [hashSidebarSessionId(otherSessionId)]: { + accountId: 'account-b', + assignedAt: now, + lastSeenAt: now, + inputBytes: 200, + }, + }) + + const rawAfterFirstClear = readFileSync(file, 'utf8') + const serializedAssignments = + JSON.parse(rawAfterFirstClear).stickyAssignments + expect(Object.keys(serializedAssignments)).toEqual([ + hashSidebarSessionId(otherSessionId), + ]) + expect(Object.keys(serializedAssignments)[0]).toMatch(/^[a-f0-9]{64}$/) + expect(rawAfterFirstClear).not.toContain(sessionId) + expect(rawAfterFirstClear).not.toContain(otherSessionId) + + expect(await clearSidebarStickyAssignment(sessionId, file)).toBe(false) + expect(readFileSync(file, 'utf8')).toBe(rawAfterFirstClear) + }) + + test('concurrent resolution for one new session returns one assignment', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-sticky-same-session-')) + const file = join(tempDir, 'sidebar-state.json') + const sessionId = 'same-new-session' + let releaseFirstWrite: (() => void) | undefined + const firstWriteReleased = new Promise((resolve) => { + releaseFirstWrite = resolve + }) + let firstWriteEntered: (() => void) | undefined + const firstWriteReached = new Promise((resolve) => { + firstWriteEntered = resolve + }) + let chooseCalls = 0 + const input = { + sessionId, + requestBytes: 100, + now, + validPinnedAccountIds: ['account-a'], + quotaCheckedAtByAccount: { 'account-a': 10 }, + choose: () => { + chooseCalls += 1 + return { accountId: 'account-a', quotaCheckedAt: 10 } + }, + } + + const first = resolveSidebarStickyAssignment(input, file, { + beforeRecheck: async () => { + firstWriteEntered?.() + await firstWriteReleased + }, + }) + await firstWriteReached + const second = resolveSidebarStickyAssignment(input, file) + releaseFirstWrite?.() + + const [firstResult, secondResult] = await Promise.all([first, second]) + expect(chooseCalls).toBe(1) + expect(secondResult).toEqual(firstResult) + expect( + Object.keys((await getSidebarState(file)).stickyAssignments ?? {}), + ).toEqual([hashSidebarSessionId(sessionId)]) + }) + + test('disperses a thundering herd through shared pending bytes', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-sticky-herd-')) + const file = join(tempDir, 'sidebar-state.json') + let releaseFirstWrite: (() => void) | undefined + const firstWriteReleased = new Promise((resolve) => { + releaseFirstWrite = resolve + }) + let firstWriteEntered: (() => void) | undefined + const firstWriteReached = new Promise((resolve) => { + firstWriteEntered = resolve + }) + let secondPendingBytes: ReadonlyMap | undefined + + const first = resolveSidebarStickyAssignment( + { + sessionId: 'herd-first', + requestBytes: 400, + now, + validPinnedAccountIds: ['account-a', 'account-b'], + quotaCheckedAtByAccount: { 'account-a': 10, 'account-b': 10 }, + choose: () => ({ accountId: 'account-a', quotaCheckedAt: 10 }), + }, + file, + { + beforeRecheck: async () => { + firstWriteEntered?.() + await firstWriteReleased + }, + }, + ) + await firstWriteReached + const second = resolveSidebarStickyAssignment( + { + sessionId: 'herd-second', + requestBytes: 250, + now, + validPinnedAccountIds: ['account-a', 'account-b'], + quotaCheckedAtByAccount: { 'account-a': 10, 'account-b': 10 }, + choose: (pendingBytes) => { + secondPendingBytes = pendingBytes + return pendingBytes.get('account-a') === 400 + ? { accountId: 'account-b', quotaCheckedAt: 10 } + : { accountId: 'account-a', quotaCheckedAt: 10 } + }, + }, + file, + ) + releaseFirstWrite?.() + + const [firstResult, secondResult] = await Promise.all([first, second]) + expect(firstResult?.accountId).toBe('account-a') + expect(secondPendingBytes?.get('account-a')).toBe(400) + expect(secondResult?.accountId).toBe('account-b') + }) + + test('drops pending bytes when an account has a fresh quota snapshot', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-sticky-pending-')) + const file = join(tempDir, 'sidebar-state.json') + await setSidebarState( + make({ + stickyAssignments: { + [hashSidebarSessionId('prior-session')]: { + accountId: 'account-a', + assignedAt: now - 1, + lastSeenAt: now, + inputBytes: 400, + quotaCheckedAt: 10, + }, + }, + }), + file, + ) + let pendingBytes: ReadonlyMap | undefined + + const result = await resolveSidebarStickyAssignment( + { + sessionId: 'fresh-snapshot-session', + requestBytes: 100, + now, + validPinnedAccountIds: ['account-a', 'account-b'], + quotaCheckedAtByAccount: { 'account-a': 11, 'account-b': 10 }, + choose: (pending) => { + pendingBytes = pending + return pending.get('account-a') === undefined + ? { accountId: 'account-a', quotaCheckedAt: 11 } + : { accountId: 'account-b', quotaCheckedAt: 10 } + }, + }, + file, + ) + + expect(pendingBytes?.get('account-a')).toBeUndefined() + expect(result?.accountId).toBe('account-a') + }) +}) + +describe('session sidebar routing with sticky assignments', () => { + const now = 2 * 7 * 24 * 60 * 60 * 1000 + + function stickyState(overrides: Partial = {}): SidebarState { + return make({ + route: 'sticky-balanced', + fallbacks: [ + fb({ + id: 'fallback-1', + enabled: true, + killed: false, + quota: quota(20), + }), + fb({ + id: 'fallback-2', + enabled: true, + killed: false, + quota: quota(30), + }), + ], + ...overrides, + }) + } + + test('uses a usable active routing entry before a sticky pin', () => { + const sessionId = 'active-routing-wins' + const result = resolveSessionSidebarRouting( + stickyState({ + activeRouting: { + [sessionId]: { + activeId: 'fallback-1', + route: 'sticky-balanced', + updatedAt: now, + }, + }, + stickyAssignments: { + [hashSidebarSessionId(sessionId)]: { + accountId: 'fallback-2', + assignedAt: now, + lastSeenAt: now, + inputBytes: 1, + }, + }, + }), + sessionId, + now, + ) + + expect(result).toEqual({ activeId: 'fallback-1', route: 'sticky-balanced' }) + }) + + test('uses a hashed usable sticky pin when no active routing entry survives', () => { + const sessionId = 'hashed-sticky-session' + const result = resolveSessionSidebarRouting( + stickyState({ + stickyAssignments: { + [sessionId]: { + accountId: 'fallback-1', + assignedAt: now, + lastSeenAt: now, + inputBytes: 1, + }, + [hashSidebarSessionId(sessionId)]: { + accountId: 'fallback-2', + assignedAt: now, + lastSeenAt: now, + inputBytes: 2, + }, + }, + }), + sessionId, + now, + ) + + expect(result).toEqual({ activeId: 'fallback-2', route: 'sticky-balanced' }) + }) + + test('falls back to the existing mode routing when no usable sticky pin exists', () => { + const sessionId = 'stale-sticky-session' + const result = resolveSessionSidebarRouting( + stickyState({ + stickyAssignments: { + [hashSidebarSessionId(sessionId)]: { + accountId: 'fallback-2', + assignedAt: 1, + lastSeenAt: now - 7 * 24 * 60 * 60 * 1000 - 1, + inputBytes: 1, + }, + }, + }), + sessionId, + now, + ) + + expect(result).toEqual({ activeId: 'main', route: 'sticky-balanced' }) + }) + + test('leaves non-sticky routing modes unchanged even when a sticky pin exists', () => { + const sessionId = 'non-sticky-session' + const assignment = { + [hashSidebarSessionId(sessionId)]: { + accountId: 'fallback-2', + assignedAt: now, + lastSeenAt: now, + inputBytes: 1, + }, + } + + expect( + resolveSessionSidebarRouting( + stickyState({ route: 'main-first', stickyAssignments: assignment }), + sessionId, + now, + ), + ).toEqual({ activeId: 'main', route: 'main-first' }) + expect( + resolveSessionSidebarRouting( + stickyState({ route: 'fallback-first', stickyAssignments: assignment }), + sessionId, + now, + ), + ).toEqual({ activeId: 'fallback-1', route: 'fallback-first' }) + }) + + test('does not display a sticky pin whose account is exhausted, disabled, or killed', () => { + const cases = [ + { + name: 'exhausted', + fallback: fb({ + id: 'fallback-2', + enabled: true, + killed: false, + quota: { + primary: { + usedPercent: 100, + remainingPercent: 0, + resetsAt: new Date(now + 60_000).toISOString(), + }, + }, + }), + }, + { + name: 'disabled', + fallback: fb({ + id: 'fallback-2', + enabled: false, + killed: false, + quota: quota(20), + }), + }, + { + name: 'killed', + fallback: fb({ + id: 'fallback-2', + enabled: true, + killed: true, + quota: quota(20), + }), + }, + ] + for (const scenario of cases) { + const sessionId = `${scenario.name}-sticky-session` + const result = resolveSessionSidebarRouting( + stickyState({ + fallbacks: [ + fb({ + id: 'fallback-1', + enabled: true, + killed: false, + quota: quota(20), + }), + scenario.fallback, + ], + stickyAssignments: { + [hashSidebarSessionId(sessionId)]: { + accountId: 'fallback-2', + assignedAt: now, + lastSeenAt: now, + inputBytes: 1, + }, + }, + }), + sessionId, + now, + ) + + expect(result).toEqual({ activeId: 'main', route: 'sticky-balanced' }) + } + }) }) describe('getSidebarState — malformed file never throws', () => { @@ -935,6 +1822,194 @@ test('upsert creates a missing sidebar state directory before locking', async () }) }) +test('upsert preserves fresh fallback pins when the roster is unknown', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-unknown-roster-')) + const file = join(tempDir, 'sidebar-state.json') + const now = Date.now() + const fallbackPinHash = hashSidebarSessionId('fresh-fallback-pin') + await setSidebarState( + make({ + stickyAssignments: { + [fallbackPinHash]: { + accountId: 'fallback-1', + assignedAt: now, + lastSeenAt: now, + inputBytes: 1, + }, + }, + }), + file, + ) + + await upsertSidebarActiveRouting( + { + sessionId: 'request-session', + activeId: 'main', + route: 'main-first', + updatedAt: now, + }, + undefined, + file, + ) + await drainSidebarWrites() + + expect( + normalizeSidebarState(JSON.parse(readFileSync(file, 'utf8'))) + .stickyAssignments, + ).toEqual({ + [fallbackPinHash]: { + accountId: 'fallback-1', + assignedAt: now, + lastSeenAt: now, + inputBytes: 1, + }, + }) +}) + +test('upsert treats an empty roster as authoritative and prunes fallback pins', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-empty-roster-')) + const file = join(tempDir, 'sidebar-state.json') + const now = Date.now() + const fallbackPinHash = hashSidebarSessionId('empty-roster-fallback-pin') + await setSidebarState( + make({ + stickyAssignments: { + [fallbackPinHash]: { + accountId: 'fallback-1', + assignedAt: now, + lastSeenAt: now, + inputBytes: 1, + }, + }, + }), + file, + ) + + await upsertSidebarActiveRouting( + { + sessionId: 'request-session', + activeId: 'main', + route: 'main-first', + updatedAt: now, + }, + [], + file, + ) + await drainSidebarWrites() + + expect( + normalizeSidebarState(JSON.parse(readFileSync(file, 'utf8'))) + .stickyAssignments, + ).toBeUndefined() +}) + +test('upsert prunes only fallback pins absent from an authoritative roster', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-populated-roster-')) + const file = join(tempDir, 'sidebar-state.json') + const now = Date.now() + const retainedHash = hashSidebarSessionId('retained-fallback-pin') + const prunedHash = hashSidebarSessionId('pruned-fallback-pin') + await setSidebarState( + make({ + stickyAssignments: { + [retainedHash]: { + accountId: 'fallback-present', + assignedAt: now, + lastSeenAt: now, + inputBytes: 1, + }, + [prunedHash]: { + accountId: 'fallback-removed', + assignedAt: now, + lastSeenAt: now, + inputBytes: 2, + }, + }, + }), + file, + ) + + await upsertSidebarActiveRouting( + { + sessionId: 'request-session', + activeId: 'main', + route: 'main-first', + updatedAt: now, + }, + [{ id: 'fallback-present', enabled: true }], + file, + ) + await drainSidebarWrites() + + expect( + normalizeSidebarState(JSON.parse(readFileSync(file, 'utf8'))) + .stickyAssignments, + ).toEqual({ + [retainedHash]: { + accountId: 'fallback-present', + assignedAt: now, + lastSeenAt: now, + inputBytes: 1, + }, + }) +}) + +test('removal with an unknown roster preserves other pins and removes its explicit hash', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-unknown-removal-')) + const file = join(tempDir, 'sidebar-state.json') + const now = Date.now() + await setSidebarState( + make({ + stickyAssignments: { + [hashSidebarSessionId('removed-session')]: { + accountId: 'fallback-1', + assignedAt: now, + lastSeenAt: now, + inputBytes: 1, + }, + [hashSidebarSessionId('other-session')]: { + accountId: 'fallback-2', + assignedAt: now, + lastSeenAt: now, + inputBytes: 2, + }, + }, + }), + file, + ) + + await removeSidebarActiveRouting('removed-session', undefined, file) + await drainSidebarWrites() + + expect( + normalizeSidebarState(JSON.parse(readFileSync(file, 'utf8'))) + .stickyAssignments, + ).toEqual({ + [hashSidebarSessionId('other-session')]: { + accountId: 'fallback-2', + assignedAt: now, + lastSeenAt: now, + inputBytes: 2, + }, + }) +}) + +test('setSidebarState leaves an existing foreign directory permission unchanged', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-private-')) + const dir = join(tempDir, 'existing') + const file = join(dir, 'sidebar-state.json') + mkdirSync(dir, { mode: 0o755 }) + chmodSync(dir, 0o755) + writeFileSync(file, JSON.stringify(DEFAULT_SIDEBAR_STATE), { mode: 0o644 }) + chmodSync(file, 0o644) + + await setSidebarState(make({ lastUpdated: 1 }), file) + await drainSidebarWrites() + + expect(statSync(dirname(file)).mode & 0o777).toBe(0o755) + expect(statSync(file).mode & 0o777).toBe(0o600) +}) + test('upserting session B preserves session A and refreshes legacy fields', async () => { const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-routing-')) const file = join(tempDir, 'sidebar-state.json') @@ -1358,6 +2433,154 @@ test('machine writes preserve routing while retaining reset-credit fields', asyn expect(written.activeRouting?.['sess-a']?.activeId).toBe('fallback-1') }) +test('every routing writer preserves sticky assignments', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-sticky-rmw-')) + const file = join(tempDir, 'sidebar-state.json') + const now = Date.now() + const stickyAssignments = { + [hashSidebarSessionId('session-a')]: { + accountId: 'fallback-1', + assignedAt: now, + lastSeenAt: now, + inputBytes: 128, + }, + } + + await setSidebarState( + make({ + fallbacks: [fb({ id: 'fallback-1', enabled: true })], + stickyAssignments, + }), + file, + ) + await setSidebarMachineState( + { + main: main(null), + fallbacks: [fb({ id: 'fallback-1', enabled: true })], + route: 'main-first', + lastUpdated: now, + }, + file, + ) + expect((await getSidebarState(file)).stickyAssignments).toEqual( + stickyAssignments, + ) + + await upsertSidebarActiveRouting( + { + sessionId: 'routing-session', + activeId: 'fallback-1', + route: 'fallback-first', + updatedAt: now, + }, + [{ id: 'fallback-1', enabled: true }], + file, + ) + expect((await getSidebarState(file)).stickyAssignments).toEqual( + stickyAssignments, + ) + + await setSidebarLegacyRouting( + { activeId: 'main', route: 'main-first', updatedAt: now }, + file, + ) + expect((await getSidebarState(file)).stickyAssignments).toEqual( + stickyAssignments, + ) + + await removeSidebarActiveRouting( + 'routing-session', + [{ id: 'fallback-1', enabled: true }], + file, + ) + await drainSidebarWrites() + expect((await getSidebarState(file)).stickyAssignments).toEqual( + stickyAssignments, + ) +}) + +test('routing writers prune disabled and killed sticky assignments', async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-sticky-prune-rmw-')) + const now = Date.now() + const accounts = [ + { id: 'disabled', enabled: false }, + { id: 'killed', enabled: true, killed: true }, + { id: 'healthy', enabled: true }, + ] + const fallbacks = accounts.map((account) => fb(account)) + const expectedStickyAssignments = { + [hashSidebarSessionId('healthy-session')]: { + accountId: 'healthy', + assignedAt: now, + lastSeenAt: now, + inputBytes: 128, + }, + } + + async function seed(file: string) { + await setSidebarState( + make({ + fallbacks, + stickyAssignments: { + [hashSidebarSessionId('disabled-session')]: { + accountId: 'disabled', + assignedAt: now, + lastSeenAt: now, + inputBytes: 128, + }, + [hashSidebarSessionId('killed-session')]: { + accountId: 'killed', + assignedAt: now, + lastSeenAt: now, + inputBytes: 128, + }, + ...expectedStickyAssignments, + }, + }), + file, + ) + } + + const machineFile = join(tempDir, 'machine.json') + await seed(machineFile) + await setSidebarMachineState( + { + main: main(null), + fallbacks, + route: 'main-first', + lastUpdated: now, + }, + machineFile, + ) + expect((await getSidebarState(machineFile)).stickyAssignments).toEqual( + expectedStickyAssignments, + ) + + const upsertFile = join(tempDir, 'upsert.json') + await seed(upsertFile) + await upsertSidebarActiveRouting( + { + sessionId: 'routing-session', + activeId: 'healthy', + route: 'fallback-first', + updatedAt: now, + }, + accounts, + upsertFile, + ) + expect((await getSidebarState(upsertFile)).stickyAssignments).toEqual( + expectedStickyAssignments, + ) + + const removeFile = join(tempDir, 'remove.json') + await seed(removeFile) + await removeSidebarActiveRouting('routing-session', accounts, removeFile) + await drainSidebarWrites() + expect((await getSidebarState(removeFile)).stickyAssignments).toEqual( + expectedStickyAssignments, + ) +}) + test('machine writes cannot clobber fresher main and fallback quota from disk', async () => { const tempDir = mkdtempSync(join(tmpdir(), 'oai-sb-quota-fresh-')) const file = join(tempDir, 'sidebar-state.json') diff --git a/packages/opencode/src/tests/sticky-routing.test.ts b/packages/opencode/src/tests/sticky-routing.test.ts new file mode 100644 index 0000000..7e7f637 --- /dev/null +++ b/packages/opencode/src/tests/sticky-routing.test.ts @@ -0,0 +1,523 @@ +import { describe, expect, test } from 'bun:test' +import { + decideStickyBreak, + MIN_RESET_HOURS, + QUOTA_STALENESS_MS, + type StickySelectionCandidate, + selectStickyCandidate, + snapshotCheckedAt, + sustainableWindowWeight, +} from '../core/sticky-routing.ts' +import type { AccountQuota } from '../sidebar-state.ts' + +const now = Date.UTC(2026, 7, 10, 12, 0, 0) + +function quota( + remainingPercent: number, + checkedAt = now, + resetsAt?: string, +): AccountQuota { + return { + primary: { + usedPercent: 100 - remainingPercent, + remainingPercent, + checkedAt, + ...(resetsAt === undefined ? {} : { resetsAt }), + }, + } +} + +function candidate( + accountId: string, + accountQuota: AccountQuota | null | undefined, + configuredOrder: number, + overrides: Partial = {}, +): StickySelectionCandidate { + return { + accountId, + quota: accountQuota, + reservePercent: { primary: 0, secondary: 0 }, + configuredOrder, + ...overrides, + } +} + +function select( + candidates: StickySelectionCandidate[], + pendingBytes: ReadonlyMap = new Map(), + requestBytes = 1, +) { + const result = selectStickyCandidate({ + candidates, + pendingBytes, + requestBytes, + now, + }) + if (!result) throw new Error('test helper: no candidate selected') + return result +} + +describe('sustainableWindowWeight', () => { + test('keeps spendable capacity when the reset is unknown', () => { + expect(sustainableWindowWeight({ remainingPercent: 40 }, 10, now)).toBe(30) + }) + + test('uses the minimum reset duration for near resets', () => { + const remaining = 40 + const windowResettingIn30Seconds = { + remainingPercent: remaining, + resetsAt: new Date(now + 30_000).toISOString(), + } + + expect( + sustainableWindowWeight(windowResettingIn30Seconds, 0, now), + ).toBeCloseTo(remaining / MIN_RESET_HOURS) + }) + + test('keeps spendable capacity when the reset timestamp is past', () => { + expect( + sustainableWindowWeight( + { remainingPercent: 40, resetsAt: new Date(now - 1).toISOString() }, + 10, + now, + ), + ).toBe(30) + }) + + test('keeps spendable capacity when the reset timestamp is invalid', () => { + expect( + sustainableWindowWeight( + { remainingPercent: 40, resetsAt: 'not-a-date' }, + 10, + now, + ), + ).toBe(30) + }) + + test('returns zero at the reserve threshold', () => { + expect(sustainableWindowWeight({ remainingPercent: 10 }, 10, now)).toBe(0) + }) +}) + +describe('snapshotCheckedAt', () => { + test('prefers primary, then secondary, then snapshot, then cache entry timestamp', () => { + expect( + snapshotCheckedAt( + { + checkedAt: 20, + primary: { usedPercent: 10, remainingPercent: 90, checkedAt: 30 }, + }, + 10, + ), + ).toBe(30) + expect(snapshotCheckedAt({ checkedAt: 20 }, 10)).toBe(20) + expect(snapshotCheckedAt({}, 10)).toBe(10) + }) + + test('uses the secondary window when primary is missing', () => { + expect( + snapshotCheckedAt({ + checkedAt: 10, + secondary: { usedPercent: 5, remainingPercent: 95, checkedAt: 40 }, + }), + ).toBe(40) + }) +}) + +describe('decideStickyBreak', () => { + test.each([ + { + name: 'migrates permanent authorization failures before quota ignorance', + input: { quota: null, status: 401, now }, + want: { action: 'migrate', reason: 'permanent' }, + }, + { + name: 'migrates forbidden responses permanently', + input: { quota: quota(50), status: 403, now }, + want: { action: 'migrate', reason: 'permanent' }, + }, + { + name: 'retains an account with no quota snapshot', + input: { quota: undefined, status: 400, now }, + want: { action: 'retain', reason: 'unknown' }, + }, + { + name: 'retains an account with a stale snapshot', + input: { + quota: quota(0, now - QUOTA_STALENESS_MS - 1), + status: 400, + now, + }, + want: { action: 'retain', reason: 'stale' }, + }, + { + name: 'retains an account with a malformed snapshot timestamp', + input: { quota: { checkedAt: Number.NaN }, status: 400, now }, + want: { action: 'retain', reason: 'stale' }, + }, + { + name: 'migrates an exhausted fresh window with diagnostic reset metadata', + input: { + quota: quota(0, now, '2026-08-10T13:00:00.000Z'), + status: 400, + now, + }, + want: { + action: 'migrate', + reason: 'exhausted', + windowKey: 'primary', + resetsAt: '2026-08-10T13:00:00.000Z', + }, + }, + { + name: 'treats a rate limit with healthy quota as transient', + input: { quota: quota(50), status: 429, now }, + want: { action: 'retain', reason: 'transient' }, + }, + { + name: 'treats a rate limit with no present fresh quota windows as transient', + input: { quota: { checkedAt: now }, status: 429, now }, + want: { action: 'retain', reason: 'transient' }, + }, + { + name: 'treats server failures as transient', + input: { quota: quota(50), status: 500, now }, + want: { action: 'retain', reason: 'transient' }, + }, + { + name: 'treats indeterminate transport failures as transient', + input: { quota: quota(50), now }, + want: { action: 'retain', reason: 'transient' }, + }, + { + name: 'retains a healthy account for non-routing client failures', + input: { quota: quota(50), status: 400, now }, + want: { action: 'retain', reason: 'healthy' }, + }, + { + name: 'does not migrate malformed exhausted-looking percentages', + input: { quota: quota(Number.NaN), status: 400, now }, + want: { action: 'retain', reason: 'healthy' }, + }, + { + name: 'does not migrate non-finite exhausted-looking percentages', + input: { quota: quota(Number.NEGATIVE_INFINITY), status: 400, now }, + want: { action: 'retain', reason: 'healthy' }, + }, + ])('$name', ({ input, want }) => { + expect(decideStickyBreak(input)).toEqual(want) + }) + + test('skips healthy windows when a longer window is exhausted', () => { + const accountQuota = quota(50) + accountQuota.primary = { + usedPercent: 50, + remainingPercent: 50, + checkedAt: now, + windowMinutes: 300, + } + accountQuota.secondary = { + usedPercent: 100, + remainingPercent: 0, + checkedAt: now, + windowMinutes: 10_080, + resetsAt: '2026-08-17T12:00:00.000Z', + } + + expect( + decideStickyBreak({ quota: accountQuota, status: 400, now }), + ).toEqual({ + action: 'migrate', + reason: 'exhausted', + windowKey: 'secondary', + resetsAt: '2026-08-17T12:00:00.000Z', + }) + }) + + test('reports the longest exhausted window when every window is exhausted', () => { + const accountQuota = quota(0, now, '2026-08-10T13:00:00.000Z') + accountQuota.primary = { + usedPercent: 100, + remainingPercent: 0, + checkedAt: now, + windowMinutes: 300, + resetsAt: '2026-08-10T13:00:00.000Z', + } + accountQuota.secondary = { + usedPercent: 100, + remainingPercent: 0, + checkedAt: now, + windowMinutes: 10_080, + resetsAt: '2026-08-17T12:00:00.000Z', + } + + expect( + decideStickyBreak({ quota: accountQuota, status: 400, now }), + ).toEqual({ + action: 'migrate', + reason: 'exhausted', + windowKey: 'secondary', + resetsAt: '2026-08-17T12:00:00.000Z', + }) + }) + + test('omits non-string reset metadata from exhausted decisions', () => { + const accountQuota = quota(0) + accountQuota.primary = { + usedPercent: 100, + remainingPercent: 0, + checkedAt: now, + resetsAt: 1 as never, + } + + expect( + decideStickyBreak({ quota: accountQuota, status: 400, now }), + ).toEqual({ + action: 'migrate', + reason: 'exhausted', + windowKey: 'primary', + }) + }) + + test('never returns a hold action', () => { + const decisions = [ + decideStickyBreak({ quota: null, now }), + decideStickyBreak({ quota: quota(0), status: 400, now }), + decideStickyBreak({ quota: quota(50), status: 429, now }), + decideStickyBreak({ quota: quota(50), status: 400, now }), + ] + + for (const decision of decisions) { + expect(decision.action).not.toBe('hold') + } + }) + + test('migrates a fresh below-floor account when killswitchPasses is false', () => { + expect( + decideStickyBreak({ + quota: quota(45), + status: 400, + now, + killswitchPasses: false, + }), + ).toEqual({ action: 'migrate', reason: 'killswitch' }) + }) + + test('keeps a stale snapshot when killswitchPasses is false (stale wins)', () => { + expect( + decideStickyBreak({ + quota: quota(45, now - QUOTA_STALENESS_MS - 1), + status: 400, + now, + killswitchPasses: false, + }), + ).toEqual({ action: 'retain', reason: 'stale' }) + }) + + test('keeps a no-quota account when killswitchPasses is false (unknown wins)', () => { + expect( + decideStickyBreak({ + quota: undefined, + status: 400, + now, + killswitchPasses: false, + }), + ).toEqual({ action: 'retain', reason: 'unknown' }) + }) + + test('migrates before exhaustion when the killswitch and exhaustion both apply', () => { + // Below floor AND at 0% — killswitch is the more specific policy reason. + expect( + decideStickyBreak({ + quota: quota(0), + status: 400, + now, + killswitchPasses: false, + }), + ).toEqual({ action: 'migrate', reason: 'killswitch' }) + }) + + test('killswitchPasses true is a no-op on the healthy path', () => { + expect( + decideStickyBreak({ + quota: quota(50), + status: 400, + now, + killswitchPasses: true, + }), + ).toEqual({ action: 'retain', reason: 'healthy' }) + }) + + test('killswitchPasses undefined is a no-op (killswitch disabled / not opted in)', () => { + expect( + decideStickyBreak({ + quota: quota(45), + status: 400, + now, + }), + ).toEqual({ action: 'retain', reason: 'healthy' }) + }) +}) + +describe('selectStickyCandidate', () => { + test('excludes candidates with missing quota', () => { + expect( + select([candidate('unknown', null, 0), candidate('known', quota(1), 1)]) + .accountId, + ).toBe('known') + }) + + test('excludes candidates with stale quota snapshots', () => { + expect( + select([ + candidate('stale', quota(100, now - QUOTA_STALENESS_MS - 1), 0), + candidate('fresh', quota(1), 1), + ]).accountId, + ).toBe('fresh') + }) + + test('uses the tightest present quota window as the account weight', () => { + const tight = quota(80) + tight.secondary = { usedPercent: 90, remainingPercent: 10, checkedAt: now } + const roomy = quota(20) + roomy.secondary = { usedPercent: 80, remainingPercent: 20, checkedAt: now } + + expect( + select([candidate('tight', tight, 0), candidate('roomy', roomy, 1)]) + .accountId, + ).toBe('roomy') + }) + + test('selects the lower projected pressure', () => { + expect( + select( + [ + candidate('less-pressure', quota(50), 0), + candidate('more-pressure', quota(50), 1), + ], + new Map([ + ['less-pressure', 0], + ['more-pressure', 100], + ]), + 100, + ).accountId, + ).toBe('less-pressure') + }) + + test('changes the next pick when pending bytes change', () => { + const candidates = [ + candidate('a', quota(50), 0), + candidate('b', quota(50), 1), + ] + + expect(select(candidates, new Map([['a', 100]])).accountId).toBe('b') + expect(select(candidates, new Map([['b', 100]])).accountId).toBe('a') + }) + + test('resolves equal scores by configured order then account id', () => { + expect( + select([candidate('z', quota(50), 1), candidate('a', quota(50), 0)]) + .accountId, + ).toBe('a') + expect( + select([candidate('z', quota(50), 0), candidate('a', quota(50), 0)]) + .accountId, + ).toBe('a') + }) + + test('never selects zero capacity over positive capacity', () => { + expect( + select([ + candidate('empty', quota(0), 0), + candidate('usable', quota(1), 1), + ]).accountId, + ).toBe('usable') + }) + + test('falls back to configured order when every snapshot is stale', () => { + const selection = select([ + candidate('first', quota(50, now - QUOTA_STALENESS_MS - 1), 0), + candidate('second', quota(50, now - QUOTA_STALENESS_MS - 1), 1), + ]) + + expect(selection).toEqual({ + accountId: 'first', + quotaCheckedAt: now - QUOTA_STALENESS_MS - 1, + source: 'mode-fallback', + }) + }) + + test('notifies the caller when no weighted candidate survives', () => { + let emptySetCalls = 0 + + selectStickyCandidate({ + candidates: [ + candidate('stale', quota(50, now - QUOTA_STALENESS_MS - 1), 0), + ], + pendingBytes: new Map(), + requestBytes: 1, + now, + onEmptyWeightedSet: () => { + emptySetCalls += 1 + }, + }) + + expect(emptySetCalls).toBe(1) + }) + + test('prefers a positive optional reset-credit count in empty-set fallback', () => { + expect( + select([ + candidate('no-credit', null, 0, { resetCreditsApplicable: 0 }), + candidate('credit', null, 1, { resetCreditsApplicable: 1 }), + ]).accountId, + ).toBe('credit') + }) + + test('rejects an empty input candidate list', () => { + expect(() => select([])).toThrow( + 'Cannot select a sticky candidate: input.candidates is empty', + ) + }) + + test('excludes a killswitch-killed candidate from weighted placement', () => { + expect( + select([ + candidate('killed', quota(20), 0, { killswitchPasses: false }), + candidate('healthy', quota(50), 1), + ]).accountId, + ).toBe('healthy') + }) + + test('excludes a killswitch-killed candidate from mode-fallback fail-open', () => { + // All quotas stale → mode-fallback. Without the filter, 'killed' would win + // on configuredOrder (0). With the filter, 'killed' is excluded and the + // remaining candidate is selected. + expect( + select([ + candidate('killed', quota(50, now - QUOTA_STALENESS_MS - 1), 0, { + killswitchPasses: false, + }), + candidate('healthy', quota(50, now - QUOTA_STALENESS_MS - 1), 1), + ]).accountId, + ).toBe('healthy') + }) + + test('killswitchPasses true is a no-op on placement', () => { + const candidates = [ + candidate('explicit', quota(50), 0, { killswitchPasses: true }), + candidate('implicit', quota(50), 1), + ] + expect(select(candidates).accountId).toBe('explicit') + }) + + test('killswitchPasses undefined is a no-op on placement (killswitch disabled)', () => { + // The dominant path with killswitch disabled must be byte-identical. + const candidates = [ + candidate('a', quota(50), 0), + candidate('b', quota(50), 1), + ] + expect(select(candidates).accountId).toBe('a') + expect(select(candidates).accountId).toBe('a') + }) +}) diff --git a/packages/opencode/src/tests/tui-quota-render.test.ts b/packages/opencode/src/tests/tui-quota-render.test.ts index 2745c9b..212b08b 100644 --- a/packages/opencode/src/tests/tui-quota-render.test.ts +++ b/packages/opencode/src/tests/tui-quota-render.test.ts @@ -1,7 +1,9 @@ import { describe, expect, test } from 'bun:test' -import type { SidebarState } from '../sidebar-state.ts' +import { hashSidebarSessionId, type SidebarState } from '../sidebar-state.ts' import { + buildApplyRequest, buildQuotaRowsForDisplay, + buildRoutingRowsForDisplay, getQuotaMetadataRows, isQuotaLoaded, } from '../tui.tsx' @@ -103,4 +105,70 @@ describe('dynamic quota TUI rows', () => { ]) expect(tui.getAccountMetadataRows?.()).toEqual([]) }) + + test('modal routing apply sends sessionId on its RPC request', () => { + expect(buildApplyRequest('openai-routing', 'reset', 'session-a')).toEqual({ + command: 'openai-routing', + arguments: 'reset', + sessionId: 'session-a', + }) + }) + + test('sticky-balanced routing renders a compact pin row only when the session has a usable pin', () => { + const sessionId = 'sticky-render-session' + const state: SidebarState = { + main: { quota: null, killed: false }, + fallbacks: [ + { + id: 'fallback-1', + label: 'Work', + quota: null, + killed: false, + enabled: true, + }, + ], + activeId: undefined, + route: 'sticky-balanced', + stickyAssignments: { + 'not-the-session-hash': { + accountId: 'fallback-1', + assignedAt: now, + lastSeenAt: now, + inputBytes: 1, + }, + }, + lastUpdated: now, + } + + expect(buildRoutingRowsForDisplay(state, sessionId, now)).toEqual([ + { label: 'Route', value: 'sticky-balanced', tone: 'accent' }, + ]) + + state.stickyAssignments = { + [hashSidebarSessionId(sessionId)]: { + accountId: 'fallback-1', + assignedAt: now, + lastSeenAt: now, + inputBytes: 1, + }, + } + expect(buildRoutingRowsForDisplay(state, sessionId, now)).toEqual([ + { label: 'Route', value: 'sticky-balanced', tone: 'accent' }, + { label: 'Pin', value: 'Work', tone: 'accent' }, + ]) + }) + + test('non-sticky routing renders the existing route row without a pin row', () => { + const state: SidebarState = { + main: { quota: null, killed: false }, + fallbacks: [], + activeId: 'main', + route: 'main-first', + lastUpdated: now, + } + + expect(buildRoutingRowsForDisplay(state, 'session-a', now)).toEqual([ + { label: 'Route', value: 'main-first', tone: 'accent' }, + ]) + }) }) diff --git a/packages/opencode/src/tui.tsx b/packages/opencode/src/tui.tsx index 6b7be0a..9bdde94 100644 --- a/packages/opencode/src/tui.tsx +++ b/packages/opencode/src/tui.tsx @@ -17,6 +17,7 @@ import { Show, } from 'solid-js' import { createLogger } from './logger.js' +import type { ApplyRequest, CommandModalName } from './rpc/protocol.js' import { createRpcClient } from './rpc/rpc-client.js' import { getRpcDir } from './rpc/rpc-dir.js' import { @@ -30,6 +31,7 @@ import { type QuotaWindow, resolveActiveAccount, resolveSessionSidebarRouting, + resolveSessionStickyAccount, type SidebarState, } from './sidebar-state.js' import { openCommandDialog } from './tui/command-dialogs.js' @@ -51,6 +53,14 @@ const log = createLogger('rpc-tui') const ID = 'cortexkit.openai-auth' +export function buildApplyRequest( + command: CommandModalName, + arguments_: string, + sessionId?: string, +): ApplyRequest { + return { command, arguments: arguments_, sessionId } +} + // Read package metadata from either the raw src/ entry or its generated // src/tui-compiled/ counterpart. Avoid a JSON import because package.json sits // outside the declaration build's rootDir. @@ -498,6 +508,32 @@ export function resolveQuotaDialogActiveId( : state.activeId } +export interface RoutingDisplayRow { + label: string + value: string + tone: Tone +} + +export function buildRoutingRowsForDisplay( + state: SidebarState, + sessionId: string | undefined, + now = Date.now(), +): RoutingDisplayRow[] { + const routing = resolveSessionSidebarRouting(state, sessionId, now) + const rows: RoutingDisplayRow[] = [ + { label: 'Route', value: routing.route, tone: 'accent' }, + ] + const pinnedAccountId = resolveSessionStickyAccount(state, sessionId, now) + if (pinnedAccountId) { + const pinnedAccount = resolveActiveAccount({ + ...state, + activeId: pinnedAccountId, + }) + rows.push({ label: 'Pin', value: pinnedAccount.name, tone: 'accent' }) + } + return rows +} + interface SidebarController { prefs: () => OpenaiAuthTuiPrefs collapsed: () => boolean @@ -782,12 +818,16 @@ function QuotaSidebar(props: { {/* Routing */} - + + {(row) => ( + + )} + {/* Plan and credits — whichever are present */} @@ -897,7 +937,8 @@ const tui: TuiPlugin = async (api) => { openCommandDialog( api, message.payload, - (command, args) => rpcClient.apply({ command, arguments: args }), + (command, args) => + rpcClient.apply(buildApplyRequest(command, args, sessionId)), sessionId, ) } diff --git a/packages/opencode/src/tui/command-dialogs.tsx b/packages/opencode/src/tui/command-dialogs.tsx index b922b6e..64fe210 100644 --- a/packages/opencode/src/tui/command-dialogs.tsx +++ b/packages/opencode/src/tui/command-dialogs.tsx @@ -18,6 +18,7 @@ type ApplyFn = ( export function buildCachekeepDialogOptions(payload: OpenDialogPayload) { const enabled = payload.knobs.enabled === true const subagents = payload.knobs.subagents === true + const sustain = payload.knobs.sustain === true const running = payload.knobs.running === true const tracked = Number(payload.knobs.tracked ?? 0) const generatedAt = Number(payload.knobs.generatedAt ?? Date.now()) @@ -31,7 +32,7 @@ export function buildCachekeepDialogOptions(payload: OpenDialogPayload) { | undefined const windowLabel = windowKnob ? `${String(windowKnob.startHour).padStart(2, '0')}-${String(windowKnob.endHour).padStart(2, '0')}` - : 'always' + : 'always (no window)' const lastWarm = lastWarmAt ? `${Math.ceil((generatedAt - lastWarmAt) / 1000)}s ago` : 'none yet' @@ -44,6 +45,7 @@ export function buildCachekeepDialogOptions(payload: OpenDialogPayload) { `last warm ${lastWarm}`, `window ${windowLabel}`, `${idleWindow}m idle cap`, + `sustain ${sustain ? 'on' : 'off'}`, `subagent idle ${subIdleWindow}m`, ].filter((part) => part.length > 0) @@ -67,6 +69,14 @@ export function buildCachekeepDialogOptions(payload: OpenDialogPayload) { ? `Warm subagent sessions (${subIdleWindow}m idle cap). Disable to skip subagents.` : `Skip subagent sessions. Enable to warm them too (${subIdleWindow}m idle cap).`, }, + { + title: sustain + ? 'Sustain main sessions: on' + : 'Sustain main sessions: off', + value: sustain ? 'sustain off' : 'sustain on', + description: + 'main-only, idle-cap only; the configured clock window still applies.', + }, { title: windowKnob ? `Warm window: ${windowLabel}` : 'Set warm window…', value: 'set_window', @@ -166,6 +176,18 @@ export function openCommandDialog( value: 'fallback-first', description: 'Prefer fallback accounts, preserve main', }, + { + title: 'Sticky balanced', + value: 'sticky-balanced', + description: + 'Keep each session on one account while balancing new sessions', + }, + { + title: "Reset this session's pin", + value: 'reset', + description: + 'Unpin this session; the next request may still choose the same account', + }, ]} onSelect={(option) => { void apply('openai-routing', String(option.value)).then((r) => { @@ -436,8 +458,8 @@ function openAccountDialog( showAddFlow() return } - // Main has no per-account actions (routing is mode-driven; main is - // not removable or reorderable) — the row is informational only. + // Main has no per-account actions: it is not removable or reorderable, + // regardless of whether the active session uses mode or sticky routing. if (option.value === 'main') return showL2Fallback(option.value) }}