fix(platform): revoke the gateway key on every session teardown - #3148
fix(platform): revoke the gateway key on every session teardown#3148Israeltheminer wants to merge 1 commit into
Conversation
A sandbox session mints an LLM gateway virtual key and hands its value into the container. revokeVirtualKey had one caller where 0.4 had four, so the TTL expiry sweep, admin destroy, phantom heal and the task-agent lane all tore sessions down and left the key live. The mint body states the invariant twice: the gateway has no native TTL, and reset_duration '1M' is safe BECAUSE teardown revokes the key long before any reset matters. It did not. So each abandoned key is a permanent bearer credential with a self-refilling monthly allowance against the org's own provider keys, one per expired, deadline-failed or destroyed session. All teardown edges now go through one seam, domains/sandbox/gateway-keys.ts. The token flip IS the election: UPDATE ... WHERE revoked_at_ms IS NULL RETURNING hands each live key to exactly one caller, so a watchdog sweeping twice or a destroy racing the expiry revokes once. An exec-scoped teardown narrows to that turn's key, so a sibling turn on the same standing session keeps its own credential. Best-effort per key, as 0.4 was: an unreachable gateway must never wedge a teardown. But the failure logs at console.error naming the key id, because that key stays spendable and only an operator can delete it by hand.
2ada8bb to
5a54362
Compare
|
Rebased onto current Two things the rebase turned up. Both conflicts were adjacent insertions in The check now sets Verified against a real Postgres and MinIO — 374/380:
|
|
Superseded by #3194 — rebased onto current main (10 commits behind), plus unit coverage. Same gap as #3171: a 487-line new module with no test in CI. Its only proof was the |
A sandbox session mints an LLM gateway virtual key, hands its value into the container, and never revoked it.
revokeVirtualKeyhad one caller where 0.4 had four.Refs #3142.
Why
The mint body states the invariant it depends on, twice: the gateway has no native TTL on a virtual key, and
reset_duration: '1M'is safe because "the key is revoked at session end, long before any reset matters."Nothing revoked at session end. Three edges tore sessions down and left the key live:
sandbox_session_tokens, never touchedllm_gateway_key_id— while reading that columnSo each abandoned key is a permanent bearer credential with a self-refilling monthly allowance against the organization's own provider keys, and one accumulates per expired, deadline-failed or destroyed session. It is scoped —
allow_all_keys: false, an explicit model list, bound to the org's own provider key ids — so it reaches no other tenant and no unlisted model. What it does is spend, every month, forever.What changed
One seam,
domains/sandbox/gateway-keys.ts, that all three edges call. Three copies of this behaviour would drift.The token flip is the election.
UPDATE … WHERE revoked_at_ms IS NULL RETURNING llm_gateway_key_idhands each live key to exactly one caller, so a watchdog sweeping the same session twice, or a destroy racing the expiry, revokes once and the second pass is a no-op. The gateway is idempotent too — an unknown key is a 404, treated as success — so even a genuine double delete cannot fail a teardown.Exec-scoped teardown narrows to one key. A deadline-failed task-agent run revokes only the key its own exec minted, so a sibling turn still running on the same standing session keeps its credential.
Revocation runs before the token flip in the destroy path — running it after would find nothing left to claim.
The session row's
llm_gateway_key_idis cleared so a later sweep can tell a revoked key from a live one. The token table keeps its id and carriesrevoked_at_msas the mark instead, because the run provenance ledger matches turns by it.Failure posture
Best-effort per key, as 0.4 was: an unreachable gateway must never wedge a teardown. But the failure logs at
console.errornaming the key id, because that key stays spendable and only an operator can delete it by hand. It is not a warning.Tests
integration-check.ts, run against a real Postgres and MinIO — 278/280, with the gateway check passing on every edge:again=falseis the idempotency proof.sibling=0is the exec-scoping proof.posturedrives a 503 from the gateway and asserts the session still reachesdestroyedwhile the leak is logged — the log line appears in the run output.The two suite failures are environment, not diff: a warm-MinIO bucket collision and a
yt-dlp-dependent probe absent from this host. Both fail identically on the unmodified base.Scope
Keys already minted on a running deployment are not recoverable by this change. Nothing records the abandoned key ids once the rows are gone, so an operator has to list the org's virtual keys on the gateway and delete the ones with no live session. Worth doing on any deployment that has run sandboxes since #3107.
Three adjacent sandbox findings are deliberately untouched and tracked in #3142: the expiry sweep can expire a session under a live turn, the admission-ticket reap runs behind more probe time than its job expiry allows, and drift reconcile never re-asserts a dropped pin.
Gate:
typecheck,oxlint --type-aware,oxfmt --check,lint:sastgreen.