fix(sandbox): authenticate drain and fail closed on unset secrets

[larryro]

Invariant

The sandbox control plane authenticates every state-changing / admin call and never runs silently unauthenticated: an unset secret is a hard failure, not a bypass. The spawner holds the host docker socket and its listener sits on tale-sandbox-net — the network every session container shares — and the LLM gateway serves inference and /api/* on one port on that same network. Nothing on that network may drive either without the credential.

Findings

1. /v1/drain + /v1/drain-status unauthenticated — fixed (high, flagged by 2 reviewers)

Confirmed: server.ts dispatched both routes straight to the handlers with no authorize; drain is a one-way latch (all creates → 503; after maxLingerMs every session is reaped) and status listed every live session id.

• services/sandbox/src/request-auth.ts — the one HMAC verifier (moved out of server.ts, no unsigned-mode branch).
• services/sandbox/src/control-routes.ts — drain / drain-status behind the same readAndAuth gate as the session routes, plus the linger self-reap state (takeLingerReap fires once). Router falls through for anything else.
• services/sandbox/src/control-cli.ts — the signed in-container client the deploy runs: docker exec <spawner> bun /app/src/control-cli.ts drain|drain-status. It signs with the container's own SANDBOX_TOKEN so the secret never crosses the CLI's argv/logs (same shape as the backend control door's $TALE_CONTROL_TOKEN). Uses node:http, not fetch: Bun's fetch honours HTTP(S)_PROXY from the env at process start (probed: NO_PROXY is the only in-process escape), and a fenced deployment's .env carries exactly those vars; node:http ignores them (probed).
• tools/cli/src/lib/actions/drain-sandbox.ts — docker exec <spawner> sh -c 'if [ -f /app/src/control-cli.ts ]; then exec bun … drain; else exec curl … ; fi'. The curl branch fires only against a pre-signed-client image (the one deploy that rolls the old spawner, whose routes were still open) — a one-release shim, labelled as such.

Tests: control-routes.test.ts (unsigned / wrong-token / cross-path-signature drain → 401 and isDraining stays false; unsigned status → 401 with no id leak; signed drain → 200 then signed status → {draining, sessions, sessionIds}; idempotent; falls through for other paths; linger reap once-only). control-cli.test.ts (headers verify with the real verify(); the client drains a live Bun.serve running the real ControlRoutes + createRequestAuth; wrong token → 401). drain-sandbox.test.ts (argv shape exec <c> sh -c <script>; secret canary never in argv; sh -n on both scripts).

2. HMAC silently disabled when SANDBOX_TOKEN is unset — fixed (high)

Confirmed: config.ts → sandboxToken: null, authorize() returned null (bypass), main() only warned. No compose file set the token; only tale deploy (ensure-env) and root bun run dev mint it into .env, so docker:dev, raw compose and CI's test stack ran the spawner open (CI's smoke test passed only because of the bypass — compose.test.yml handed .env.test to every service except sandbox).

• loadConfig() throws SANDBOX_TOKEN is required … on unset / empty / whitespace (trimmed); SpawnerConfig.sandboxToken: string; the runnerd per-session token is always derived (null branches removed in session-routes / docker + k8s backends).
• Secrets supplied everywhere the stack is real: compose.dev.yml x-dev-secrets anchor (insecure dev default; the sandbox service now receives the anchor), compose.test.yml (sandbox reads .env.test), services/platform/scripts/dev-secrets.ts fallback with the same literal as compose.dev.yml (lockstep pinned by dev-secrets.test.ts), .env.example (REQUIRED section). tale deploy path unchanged (ensure-env already mints it; generated compose uses env_file: .env).
• Deploy preflight checkSandboxToken: unset → fail (was warn: HMAC disabled).

Red proven on base: the updated server.test.ts fail-closed tests ran against the unmodified config.ts and failed with sandboxToken: null (3 fails); the pre-existing test literally asserted the bypass (returns null token on a fresh env (opt-in verification)).

3. LLM gateway admin API anonymous when the password is unset — fixed (high)

Confirmed: adminPassword() returned ''; managementHeaders() sent Basic and applyGatewayConfig() pushed auth_config only when set; no compose file set it.

• requireGatewayAdminPassword() fails closed (blank = unset; pre-rename LLM_GATEWAY_ADMIN_PASSWORD still honoured). Every management call sends Basic; applyGatewayConfig always enables auth_config; provisionSessionGatewayKey calls it first so a session fails once, clearly, before any credential resolve or gateway call.
• Supplied in compose.dev.yml (anchor default), .env.test, root scripts/dev.ts (minted + persisted — the gateway stores the hash in its volume, so it must stay stable), dev-secrets.ts fallback (lockstep-tested), .env.example (REQUIRED). tale deploy unchanged.

Tests: llm_gateway_admin.test.ts (unset → throws before any fetch, for reprovisionProvider and applyGatewayConfig; blank → throws; Basic on every call; auth_config always in the PUT); gateway_provisioning.test.ts (unset → rejects before resolve/provision/apply/mint). Honest note: these were written alongside the implementation, not run red against base — the base behaviour is evidenced by the pre-existing assertion I had to change (applyGatewayConfig PUT body with no auth_config).

Verified by reading only (no docker build / compose up / deploy run here)

• The in-container docker exec … sh -c path end-to-end (the script is sh -n-checked and unit-tested for shape; the client is tested against a live server, but not inside the real image).
• docker compose config was run read-only for the dev overlay (compose.yml + compose.dev.yml + compose.docs.yml) and the test overlay (+ compose.test.yml --env-file .env.test): the merged sandbox service carries SANDBOX_TOKEN, backends carry both secrets. No container was started.
• tale deploy / ensure-env behaviour (read: requiredAutoVars already contains both secrets; generated compose passes env_file: .env).
• Upgrade path: the first deploy after this change drains the old spawner via the curl branch (its routes are still open); the new image then boots fail-closed.

Gates (observed)

• @tale/sandbox: bun test 234 pass / 0 fail; tsc --noEmit 0; oxlint --type-aware clean; oxfmt --check clean.
• @tale/cli: bun test 329 pass / 0 fail / 18 skip (after bun run generate for the gitignored embed); tsc --noEmit 0; oxlint clean.
• @tale/platform: vitest (server project) on the three touched files 53 pass; oxlint --type-aware on changed files clean; tsc --noEmit 0 (whole workspace).
• sh -n on both control scripts; bun build --no-bundle scripts/dev.ts transpiles; pre-commit oxfmt + local SAST: 0 findings on every commit.

Cross-class discoveries (not fixed here)

1. CLI drain protocol mismatch: drain-sandbox.ts readDrainStatus expects inFlight, but the spawner returns {draining, sessions, sessionIds} — every deploy waits out the full 5-minute drain budget; the "linger while sessions remain" flow the server documents has no CLI consumer of sessionIds. Reliability, separate fix.
2. runnerd empty-token bypass (services/sandbox-runtime/daemon/src/main.ts tokenOk): unreachable via the spawner now (token always derived) but latent; left alone because the runtime-image conformance test boots runnerd tokenless.
3. docs/en/self-hosted/configuration/environment-reference.md documents no sandbox variables at all (needs an en/de/fr pass).
4. Backend clients (session_client.ts, screencast-relay.ts) still send unsigned when SANDBOX_TOKEN is unset — they now get a clean 401; a backend/env.ts precondition would fail faster.
5. Gateway first-boot window: bifrost's /api/* is anonymous until the first applyGatewayConfig push (then persisted in its volume); seeding auth_config at gateway boot would close it.
6. services/sandbox/src/auth.ts header still names Convex as the client (stale wording).