diff --git a/docs/architecture.md b/docs/architecture.md index 74c5cd0c..7d6cb285 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -205,7 +205,7 @@ When a launch resolves `headroom` on, the launcher (synchronous end to end, righ The front door, not the headroom supervisor, hosts the second server: the CONNECT surface OAuth launches point `HTTPS_PROXY` at. It terminates TLS for the intercept hosts, each with its own leaf signed by a local CA persisted under `/frontdoor/ca/` (the key mode 0600): `api.anthropic.com`, whose `/v1/` paths run through the same ordered pipeline the provider listener serves, and `platform.claude.com`, the claude.ai control plane, which is never HTTP-parsed and is pumped byte for byte; every path on the API host the pipeline does not serve, and every host outside the intercept set, is piped untouched. The transparent surface, a second listener for traffic that ignores proxies, terminates redirected connections the same way, keyed on the TLS SNI name the client presents. None of that happens for a client that has not authenticated: the CONNECT request must carry a live launch's capability as its `Proxy-Authorization` credential (the launcher puts it in the `HTTPS_PROXY` URL, from which clients derive that header), checked against the session registry with the same constant-time comparison the pipeline's admission uses, or it is answered 407 before the target is parsed or dialled. A connection ledger inside the surface enforces `ConnectLimits`: a head deadline, a pending-head cap that evicts the oldest waiting connection, a cap on authenticated connections, and a periodic revalidation that closes every connection whose launch's capability has since been pruned. Authenticated tunnels deliberately have no idle timeout, since Remote Control streams may sit silent indefinitely. That shape is what keeps Claude Code's Remote Control and connectors working under compression: they refuse any `ANTHROPIC_BASE_URL` the child can be given, but honour the proxy layer, so the base URL stays pointing at the real API. All of the surface's effects (TCP listening, TLS termination, HTTP parsing, forwarding) sit behind a `ConnectEffects` port in the same style as `SupervisorPorts`, and its routing decisions are pure functions over strings; the TLS round-trip tests run the real server against certificates this module issues with only the upstream target redirected. -The same CA signs the front door's provider listener, which serves HTTPS with a leaf for `127.0.0.1` and `localhost`. Provider sessions send their credential and launch capability on every request, so the child must be able to tell the door from anything else holding the port; a TLS leaf whose CA key only this user can read is that proof, and every routed child trusts the CA through `NODE_EXTRA_CA_CERTS` (combined with any bundle the parent named, since Node reads only one file). The launcher checks the same thing before trusting a door it finds: `ensureFrontDoor` probes the listener's health endpoint over TLS against the CA, from a worker thread that the synchronous launcher waits on with `Atomics.wait`, and a listener that keeps failing is distrusted and replaced, or the launch refused, before any capability is written. This tool's own supervisor that misses its probes is retired first, so the replacement keeps the sticky ports, but only when no live session is registered against it: a miss is what a busy door looks like, so a door with live sessions is waited on until the launch's own deadline and reported, never signalled, and `frontdoor restart` is the deliberate way to replace it: an explicit replace retires the door even when it is not answering, asking it to exit and killing it if it has not exited after a grace period, because a door whose event loop is blocked never runs its signal handler. A launch planned inside the serving door itself (the usage refresh, the window opener, a job) registers its session and returns the door's own addresses without probing, because the thread that would wait on the probe is the one event loop that has to answer it. The door enters its process role (`src/processRole.ts`) before it binds anything, and the primitives that block the calling thread (the synchronous sleep and the synchronous probe) refuse to run while a role is held, so a path that would stall the door throws `BlockingInEventLoopError` at its first wait instead of freezing every session. The door also carries a watchdog on a thread of its own (`frontdoor/watchdog.ts`): the supervisor beats a shared counter every tick and the thread, whose event loop a blocked main loop cannot stall, ends the process when the counter has not moved for twice the longest a launcher waits on a door before calling it unresponsive, after recording the stall in the state file's `lastError`. It is started once the listeners are bound and stopped before any retirement begins, and it restarts its clock when it was itself descheduled for longer than its cadence, so a suspended or starved machine is not taken for a hung door. The next launch starts a fresh door on the same sticky ports with the session registry intact. `doctor` and `frontdoor status` ask the listeners themselves whether they accept (`frontdoor/listenerReadings.ts`): a TLS handshake against the CA on the provider listener and a plain TCP accept on the CONNECT surface, each timed, because a live supervisor pid cannot tell a door that serves from one whose event loop is blocked, and the two listeners carry different sessions (provider sessions on one, every plain OAuth session on the other). The direct listener headroom forwards back to stays plain HTTP, because headroom is what connects to it; what keeps that leg harmless is that no credential travels on it. The hop parks the provider's credential headers in an in-memory custody record and sends headroom placeholders and a hop id, and the direct listener's admission step redeems a live id (for the same provider, with the generation's hop secret) for the real headers before routing. Admission is a pipeline stage in its own right, returning the headers a request is routed with, so restoring credentials is admission's job and no route knows about it. +The same CA signs the front door's provider listener, which serves HTTPS with a leaf for `127.0.0.1` and `localhost`. Provider sessions send their credential and launch capability on every request, so the child must be able to tell the door from anything else holding the port; a TLS leaf whose CA key only this user can read is that proof, and every routed child trusts the CA through `NODE_EXTRA_CA_CERTS` (combined with any bundle the parent named, since Node reads only one file). The launcher checks the same thing before trusting a door it finds: `ensureFrontDoor` probes the listener's health endpoint over TLS against the CA, from a worker thread that the synchronous launcher waits on with `Atomics.wait`, and a listener that keeps failing is distrusted and replaced, or the launch refused, before any capability is written. This tool's own supervisor that misses its probes is retired first, so the replacement keeps the sticky ports, but only when no live session is registered against it: a miss is what a busy door looks like, so a door with live sessions is waited on until the launch's own deadline and reported, never signalled, and `frontdoor restart` is the deliberate way to replace it: an explicit replace retires the door even when it is not answering, asking it to exit and killing it if it has not exited after a grace period, because a door whose event loop is blocked never runs its signal handler. A launch planned inside the serving door itself (the usage refresh, the window opener, a job) registers its session and returns the door's own addresses without probing, because the thread that would wait on the probe is the one event loop that has to answer it. The door enters its process role (`src/processRole.ts`) before it binds anything, and the primitives that block the calling thread (the synchronous sleep and the synchronous probe) refuse to run while a role is held, so a path that would stall the door throws `BlockingInEventLoopError` at its first wait instead of freezing every session. The door also carries a watchdog on a thread of its own (`frontdoor/watchdog.ts`): the supervisor beats a shared counter every tick and the thread, whose event loop a blocked main loop cannot stall, ends the process when the counter has not moved for twice the longest a launcher waits on a door before calling it unresponsive, after recording the stall in the state file's `lastError`. It is started once the listeners are bound and stopped before any retirement begins, and it restarts its clock when it was itself descheduled for longer than its cadence, so a suspended or starved machine is not taken for a hung door. A stall always leaves a replacement behind, started while the old pid is still dying so it can bind the same sticky ports the moment they free (the session registry is on disk either way); the chain stops after a bounded run of consecutive replacements, because a stall that has recurred through that many doors in a row is the machine's, and the next launch's ensure is what starts the door after that. `doctor` and `frontdoor status` ask the listeners themselves whether they accept (`frontdoor/listenerReadings.ts`): a TLS handshake against the CA on the provider listener and a plain TCP accept on the CONNECT surface, each timed, because a live supervisor pid cannot tell a door that serves from one whose event loop is blocked, and the two listeners carry different sessions (provider sessions on one, every plain OAuth session on the other). The direct listener headroom forwards back to stays plain HTTP, because headroom is what connects to it; what keeps that leg harmless is that no credential travels on it. The hop parks the provider's credential headers in an in-memory custody record and sends headroom placeholders and a hop id, and the direct listener's admission step redeems a live id (for the same provider, with the generation's hop secret) for the real headers before routing. Admission is a pipeline stage in its own right, returning the headers a request is routed with, so restoring credentials is admission's job and no route knows about it. The front door records the wire protocol it speaks (`FRONT_DOOR_PROTOCOL` in `src/frontdoor/state.ts`) in its state once it is serving. A launcher that finds a live door whose protocol is lower than its own (a door from before the field counts as 0) stops it, after its listener has authenticated as this tool's own, and spawns a replacement that rebinds the same sticky ports. That is safe because the capability registry is on disk and sessions already running keep reaching the addresses they froze at launch. A door of the same or a newer protocol is always joined, so an older launcher never downgrades a newer door. The rule for a bump: a door accepts every lower protocol's launches (protocol 1 accepts the former `x-claude-use-*` header names). diff --git a/docs/testing.md b/docs/testing.md index 8fde5b5e..6fd046f5 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -29,7 +29,7 @@ The credential cache's reported state is tested as a pure function over a fake c `claudeShim.test.ts` follows `identityManager.test.ts`'s real-temp-directory convention (a fake "own executable" file standing in for the running `agent-shim` binary), rather than `doctor.ts`'s pure-function style, since `enableClaudeShim`/`disableClaudeShim` are themselves real filesystem operations, not something to keep separate from a thin wiring layer. Coverage includes the version-drift case that motivates persisting `claude-shim.json` at all (the source file overwritten in place between two `shim enable` runs, proving the marker — not the inode — is what lets the second run refresh cleanly instead of refusing), a foreign file at the target being refused without `--force` and accepted with it, and the cross-device (`EXDEV`) copy-fallback path via a small injectable `LinkFs` seam (mirroring `config/store.ts`'s own `StoreFs`/`nodeStoreFs` pattern), since a real cross-filesystem rig isn't practical in CI. One test also reproduces Homebrew's actual layout (a symlink into a separate "Cellar" directory) to confirm the shim lands next to the symlink users invoke, not buried in the directory its realpath resolves to. `resolveOwnBinaryCheck`, which feeds `discoverClaudeBinary`'s PATH fallback, is covered in the same file over the same temp layout: a different package's claude beside agent-shim (one npm global prefix) reads as foreign, a `shim enable` hardlink and a symlink to the running binary read as this tool with or without a marker, the recorded target still reads as this tool after version drift, and a `--dir` marker elsewhere does not tar a foreign claude next to agent-shim. The fallback's own loop is covered purely in `versionDiscovery.test.ts`, including the co-location case (a claude in agent-shim's own directory must be found) and the self-reference case (a candidate that is this tool must be skipped, and loud failure when it is the only one), and `doctor.test.ts` runs the real wiring once end to end over a temp prefix to prove `collectDoctorReport` reports the co-located claude. -The codex translation and the front-door daemon that serves it keep the same discipline: no real network (beyond loopback listeners the test itself starts), file, clock or process in a test. `codex/translate.test.ts` covers the pure core against the script it replaces (text, images both ways, tool calls and results linked by call id, long tool names shortened and restored, system turns folded into user turns, the tier mapping's order and overrides, effort levels including omitting reasoning for `none`, the prompt cache key from `metadata.user_id`). `codex/auth.test.ts` drives the `~/.codex/auth.json` store over an injected filesystem and token endpoint: one refresh for any number of concurrent 401s, a re-read before every refresh that reuses a token the Codex CLI already refreshed, rotated tokens merged into the file as it stands, the rotated refresh token on disk before the new access token is handed out, and a refusal that leaves a valid file behind when the persist is killed before the rename. `codex/route.test.ts` covers the translation route itself against a scripted upstream (a stream failure answered as an Anthropic-shaped error, a 429 mapped onto the Anthropic rate-limit surface with retry-after, per-provider translation settings), and `codex/agent.test.ts` pins the keep-alive ceiling at ten seconds against a server whose hint asks for longer. `frontdoor/route.test.ts` covers the identity step (the session split out of the headers, exactly the named internal headers stripped, case-insensitively) and the provider path space; `frontdoor/pipeline.test.ts` covers the ordered pipeline over fakes (backpressure waiting for drain and settling when the client goes away, observers run in order at each response head including unrouted errors, an unauthorized request refused as an Anthropic-shaped 401 before any route is asked, a throwing observer dropped without breaking routing, a route failure answered as an Anthropic-shaped 500 before the head and a destroy after it); `frontdoor/providerRoute.test.ts` covers the per-request provider resolution (a codex provider mounted headroom-eligible with its own address as the hop's upstream, an http provider refused by name until the pass-through route exists, unknown and invalid providers, paths naming no provider), the credential an http provider's route attaches (whatever the child presented replaced by the provider file's own resolved token in the bearer target's header form, an apiKey target answered with x-api-key and the presented bearer dropped, an unresolvable block answered as an Anthropic-shaped 502 naming the provider with nothing forwarded, and a re-resolution after a provider file edit applying to the next request with no restart). `frontdoor/server.test.ts` runs the real listener end to end against a scripted codex upstream: the translation served through the whole stack, the identity and session headers stripped before any route sees the request, a request with no or a wrong launch capability refused 401 without any route being asked, streaming delivered chunk by chunk rather than buffered, the client's disconnect aborting the route, an unrouted target answered as an Anthropic-shaped error, the health probe, and a closed door rebinding the same sticky port so a frozen base URL works again, while a sticky port something else holds is left for a free one without the bind failure reaching the listener's fatal error handler (which used to exit the door, so a replacement could never come up beside a squatter). Its TLS cases use real certificates issued by `x509.ts`: a client that trusts only agent-shim's CA completes a handshake with the provider listener serving the loopback leaf (and the CA-verified health probe passes only against that CA), while the same client cannot talk to a complete door whose leaf a different CA signed, which never sees the request or its credential. `frontdoor/x509.test.ts` checks the certificate encoder against Node's own parser (chain verification, issuer matching, host and IP SAN matching, the validity encoding either side of 2050, positive serials, IPv6 address forms) and mutating the issuer name was confirmed to fail the tests that name it. `frontdoor/headroomHop.test.ts` builds the door from the production `createDoorPipelines`, with the provider listener on TLS, and runs the hop against a scripted daemon serving on a real unix socket inside a real mode 0700 directory under a throwaway root, reached through the production path from the state file through the socket check to `socketPath`, which forwards by `x-headroom-base-url` the way the real one does (its own `x-headroom-*` controls stripped, every other header verbatim): a streamed response relayed event by event, a process squatting on the old TCP port (named by an earlier release's state file, or by a current state file rewritten with a port) never offered a connection whether the daemon is down or serving, a socket directory left world accessible, a symlinked socket directory and a symlinked socket each refused with a 502 naming the reason and nothing sent to the daemon, headroom receiving the Anthropic-shaped request and the translator serving what the daemon forwards back through the direct listener (exactly one daemon request and one upstream call, so the loop cannot double-apply), the project identity re-set on the hop with no session header reaching it, a 502 while the daemon is between restarts, a client disconnect unwinding the hop's own request, and a flag-less session served directly. The credential custody cases: for an http provider, headroom sees only `Bearer agent-shim-sequestered` or `agent-shim-sequestered` and a distinct hop id per request, never the credential, while the provider's upstream receives the door-attached credential from the provider file and neither the client's own presentations nor any of the door's hop headers; with the daemon holding a hop open, the direct listener refuses an unknown hop id, the wrong generation secret, a live id presented for another provider and a missing id, the upstream seeing none of them, and refuses the same id again once the hop has ended; an OAuth session's bearer reaches the daemon untouched with no hop id. `frontdoor/passthrough.test.ts` runs the pass-through against a fake upstream (the provider prefix stripped with the query kept, the host replaced, internal headers absent, an unreachable upstream answered as an Anthropic-shaped 502), `frontdoor/connect.test.ts` keeps the moved TLS round-trip suite (termination, streaming, blind tunnels, port release), every client in it now presenting a live capability the way a real one derives it from `HTTPS_PROXY`, and adds the surface's authentication and limits over real sockets with the tunnel dialler recording every target: a CONNECT with no credential, a wrong capability, a non-Basic scheme, a repeated header or a credential with no password is answered 407 with the Basic challenge, for the intercept host and a blind-tunnel host alike, and nothing is dialled, intercepted or routed for it; a client that has not finished its head by the deadline gets 408 and is closed while a tunnel established earlier still carries bytes past that deadline; an oversized head gets 431; with the pending cap full of half-sent heads, a newcomer evicts the oldest (408) and a live launch's connection still gets through; an authenticated CONNECT beyond the tunnel cap gets 503 without a dial, and is admitted again once a tunnel closes; and a launch whose registry record is pruned has its open tunnel closed at the next revalidation and its capability refused 407 from then on, using the production check over the real registry functions. Each of those guarantees was checked to discriminate by mutating `connect.ts` (skipping the capability check, never arming the deadline, dropping the eviction, the tunnel cap, the revalidation or the size check on a head whose terminator has arrived): every mutant fails the case naming it. `frontdoor/capability.test.ts` covers the credential itself: the proxy URL percent-encodes a token holding every userinfo delimiter and a client's Basic credential carries it back exactly, the password half is read scheme case-insensitively with colons kept, other schemes and empty or missing passwords present nothing, and only an exact live token is accepted, and `frontdoor/latency.test.ts` bounds the added hop: the median through-door request may cost at most two milliseconds over the direct one, a bound two orders of magnitude above a loopback hop's real cost and still small enough to fail if the hop ever grows a request's worth of work. The ordering test discriminates by construction: mutating the pipeline to serve the route before the hop fails three hop tests at once. The authentication and custody tests were checked the same way: letting the probe skip certificate verification, making `ensureFrontDoor` skip the probe, sending the credential to headroom unsequestered, admitting any hop id, and serving the provider listener without TLS each fail the tests that name that guarantee. The launcher suites cover both launch shapes ending at the right place: a provider session keeps the OAuth shape (no base URL, HTTPS_PROXY on the door's CONNECT surface carrying the launch's capability as its proxy credential, the provider named in the session headers, and no credential variable at all) with NODE_EXTRA_CA_CERTS on the door's trust bundle, and an OAuth session keeps its base URL untouched with the same proxy, capability and trust bundle. Both shapes gain NO_PROXY and its lowercase spelling exempting every loopback name, appended to a parent's own exclusions without replacing them and with a repeat entry folded away. The parent's own NODE_EXTRA_CA_CERTS reaches the door port, and a bundle warning is logged. `frontdoor/supervisor.test.ts` runs the listener-host lifecycle over injected ports with a clock advanced only by sleeps: all three sticky ports bound and inherited by a successor generation after a crash, moves off occupied ports logged per listener, failed listener starts recorded as `lastError` with the start lock cleared and the already-bound listeners closed, dead sessions pruned, the idle shutdown closing all three listeners and keeping the sticky addresses (the Remote Control stream hub closed exactly once after the last listener, which a startup failure path performs as well, and never while the door is still serving), the stream hub's close idempotent (a second close persists nothing, asserted in the stream suite's boundary-save case), and the idle clock reset by a new session, the binary turnover at the first empty registry once the executable the door serves as is replaced on disk (before the idle window elapses, through the same graceful shutdown and with the sticky addresses kept), a replaced binary still served while a session is live, the turnover firing at the first empty registry tick after a session live at the replacement ends rather than after fresh idle minutes, and no turnover while the executable is unstatable at startup, gone mid-run, or returning to the exact captured identity after an absent window (the removal window a remove-then-link install leaves), a door superseded by another live supervisor exiting on the first tick after the last session it served ends while a launch made against the owner is still live, leaving the owner's state record exactly as the owner wrote it, a launch registering in the tick after the takeover counted as the superseded door's own, the sessions keyed by either door's own pid ignored so a superseded door with no launches exits at the grace tick, a state file naming a dead supervisor leaving the door serving as the owner, and the supersession and drained exit published on the backbone, with the launch lifecycle published on the event backbone as the registry changes between ticks (a registration, an end by the launcher's own exit hook, a prune of a dead launch, and nothing for the baseline a generation inherits). `frontdoor/ensure.test.ts` covers the launcher bring-up (one supervisor under the lock, a dead holder's lock cleared, joining a serving door, a fatal `lastError` refused immediately, a timeout naming the daemon log) and its listener authentication: the listener is verified before anything is registered, a listener failing verification behind a supervisor this launch spawned is refused with the reason and the log named and no session written, and a recorded live pid whose listener fails is distrusted and replaced by a verified supervisor on another port. `usage/identityClaude.inDoor.test.ts` drives the helper the door's background work plans its launches with (`withIdentityClaude`) from inside the front door's role, over the real launch wiring with a stand-in `claude` on PATH and a throwaway state root, and requires the launch to fail with the named error at its first wait without running anything; with the role guard disabled the same test blocks for the launch's whole wait and fails, which is the stall it exists to catch. `frontdoor/watchdog.test.ts` runs the real watchdog thread with a short cadence and bound (a loop that never beats reported with the stall recorded in `lastError` and the rest of the state record kept, a loop that beats within the bound left alone, the stall recorded and reported while the main thread is blocked in a synchronous wait, and nothing after a stop), and pins its derived constants against the launcher's probe patience; the supervisor suite adds that the watchdog starts after the listeners bind, is beaten each tick, is stopped before the first listener closes, and never starts for a door whose bind failed. `processRole.test.ts` pins the guard that keeps the door's event loop unblocked: a process with no role sleeps and probes as the launcher does, a process holding the front door role has its synchronous sleep and synchronous probe refuse with a named error before any wait or worker starts, and the role is put back when left; the ensure suite adds that a launch which would have to wait fails at its first wait inside the role and that a launch planned inside its own door still takes its shortcut. `frontdoor/probe.test.ts` runs the real synchronous probe against listeners on worker threads (the probe blocks its own thread, so a listener there could never answer): it accepts the loopback leaf from agent-shim's CA and rejects a leaf from another CA for the same names, a leaf from the right CA that does not name 127.0.0.1, a plain-HTTP listener, a closed port, and a squatter that accepts the connection and never answers (failed by the probe's own deadline, not the launcher's outer wait). `frontdoor/trust.test.ts` covers the child's CA bundle: the CA alone when the parent set nothing, a combined bundle that keeps the parent's certificates and is written once per combination, an inherited bundle that already holds the CA reused unchanged, and an unreadable inherited file replaced by the CA alone with a warning naming it. `frontdoor/detach.test.ts` the daemon's independence from the launcher's terminal, and `frontdoor/commands.test.ts` plus `codex/commands.test.ts` the two status commands' output over a fake filesystem, and `frontdoor/commands.test.ts` also runs the real turnover stat against a real temporary directory: the entry's identity reported and followed through a symlink, a staged-then-renamed replacement seen as a different identity (the updater's own install shape), and a missing entry or a directory reported as undefined. `watchedInstallEntry`'s own table lives in the supervisor suite: the resolved entry watched as-is when it is named `agent-shim` (with or without Windows' `.exe`), and the `agent-shim` sibling watched when the door was spawned through the `claude` hardlink, the legacy `claude-use` symlink or any other name, and the own entry watched when no sibling exists (the npm bundle's `dist/cli.cjs` shape). The launcher's own front-door tests inject a fake `FrontDoorPort` to cover the child's base URL, the bring-up order (front door before headroom, since headroom's allowlist needs the door's address), the loud refusal when no port is wired, and that an http provider's launch goes through the door too. `frontdoor/state.test.ts` covers the front-door session registry: a launch whose record carries a capability token is listed (the supervisor's idle decision depends on it, and the supervisor tests register sessions through the real writer for the same reason), the token never appears in a listing, malformed and token-less records are skipped, dead launches are pruned along with their capability, and the registry directory and records are written owner-only; `realPorts.test.ts` proves the real filesystem port produces mode 0700 and 0600 under a permissive umask and narrows entries that already existed wider. `frontdoor/eventHub.test.ts` covers the door's event backbone: delivery to two subscribers of one source and an every-source subscriber with a detached one receiving nothing further, per-source filtering with each source's sequence stamped independently and monotonically (and one name's re-registration publishing through the same sequence), the registered-source listing, the bounded drop-oldest stream bridge whose first survivor's sequence names the gap the dropped events left, a closed stream resolving done and delivering nothing published after the close, and the wrap that puts the Remote Control fan-out on as the backbone's first publisher beside its own unchanged delivery, session filtering included. `frontdoor/quotaExpiringEvents.test.ts` covers the expiring-quota publisher against a fake clock: one event when an unspent window enters its final span and silence on the ticks inside it, nothing for a window outside the span, past its reset or fully used, a re-notice for the next window instance's new reset instant, and the backbone-fed freshness (a written usage event alone moves a window into a span the next check publishes, with no file read). `src/usage/store.test.ts` gains the store's publish dep: every written snapshot is handed to the caller's callback exactly as it stands on disk, a record without an identity writes no snapshot so publishes nothing, and the store runs with no dep wired at all. The supervisor suite gains the door's own sources: the generation and idle-shutdown health events in the order the supervisor lives them, the binary-turnover event before the exit a replaced binary causes, the listener-failure event naming which listener and why, and the expiring-quota check called on every tick. The pool ranking's yielding stickiness is covered in `usage/pick.test.ts`: a member whose seven-day window would run dry at its own observed pace is demoted below one that would not (and a five-hour pace is never converted into seven-day fractions, so a member with only five-hour readings stays unprojected), and the move a resumed conversation makes is named with its reason for the run-dry tail and the extra-usage billing tail, while a kept member and a fresh launch report no move; `launcherPool.test.ts` drives the same move through the real launch and asserts the decision line says which member the conversation moved off and why. `frontdoor/launchEvents.test.ts` covers the launch lifecycle publisher: the silent baseline, the register, end and prune classification carrying the registry's own start times and the observing clock, the departures-before-arrivals ordering with pid order inside each group, and a launch dead on arrival pruned and never registered. `frontdoor/eventsApi.test.ts` mounts the door's whole typed API on the real listener shape over TLS and drives a registry mutation through the real writer, asserting a typed subscriber receives the source-tagged event over the wire, that a caller without the generation's token and a filter naming an unknown source are each refused having delivered no event, and that a source-filtered subscriber receives only the source it named. `frontdoor/listenerReadings.test.ts` reads both listeners of a live door through scripted probes and a stepped clock (provider first, each timed alone, a failing listener's reason carried while the other is still read, nothing probed when no supervisor is alive, a port the state does not record skipped) and formats them for `frontdoor status`; `frontdoor/probe.test.ts` runs the real accept probe against a listener on a worker thread and a closed port; `doctorFrontDoor.test.ts` gains the listener findings (a pass stating the probe's time, a failure naming the sessions the listener cuts off and the restart that replaces it). +The codex translation and the front-door daemon that serves it keep the same discipline: no real network (beyond loopback listeners the test itself starts), file, clock or process in a test. `codex/translate.test.ts` covers the pure core against the script it replaces (text, images both ways, tool calls and results linked by call id, long tool names shortened and restored, system turns folded into user turns, the tier mapping's order and overrides, effort levels including omitting reasoning for `none`, the prompt cache key from `metadata.user_id`). `codex/auth.test.ts` drives the `~/.codex/auth.json` store over an injected filesystem and token endpoint: one refresh for any number of concurrent 401s, a re-read before every refresh that reuses a token the Codex CLI already refreshed, rotated tokens merged into the file as it stands, the rotated refresh token on disk before the new access token is handed out, and a refusal that leaves a valid file behind when the persist is killed before the rename. `codex/route.test.ts` covers the translation route itself against a scripted upstream (a stream failure answered as an Anthropic-shaped error, a 429 mapped onto the Anthropic rate-limit surface with retry-after, per-provider translation settings), and `codex/agent.test.ts` pins the keep-alive ceiling at ten seconds against a server whose hint asks for longer. `frontdoor/route.test.ts` covers the identity step (the session split out of the headers, exactly the named internal headers stripped, case-insensitively) and the provider path space; `frontdoor/pipeline.test.ts` covers the ordered pipeline over fakes (backpressure waiting for drain and settling when the client goes away, observers run in order at each response head including unrouted errors, an unauthorized request refused as an Anthropic-shaped 401 before any route is asked, a throwing observer dropped without breaking routing, a route failure answered as an Anthropic-shaped 500 before the head and a destroy after it); `frontdoor/providerRoute.test.ts` covers the per-request provider resolution (a codex provider mounted headroom-eligible with its own address as the hop's upstream, an http provider refused by name until the pass-through route exists, unknown and invalid providers, paths naming no provider), the credential an http provider's route attaches (whatever the child presented replaced by the provider file's own resolved token in the bearer target's header form, an apiKey target answered with x-api-key and the presented bearer dropped, an unresolvable block answered as an Anthropic-shaped 502 naming the provider with nothing forwarded, and a re-resolution after a provider file edit applying to the next request with no restart). `frontdoor/server.test.ts` runs the real listener end to end against a scripted codex upstream: the translation served through the whole stack, the identity and session headers stripped before any route sees the request, a request with no or a wrong launch capability refused 401 without any route being asked, streaming delivered chunk by chunk rather than buffered, the client's disconnect aborting the route, an unrouted target answered as an Anthropic-shaped error, the health probe, and a closed door rebinding the same sticky port so a frozen base URL works again, while a sticky port something else holds is left for a free one without the bind failure reaching the listener's fatal error handler (which used to exit the door, so a replacement could never come up beside a squatter). Its TLS cases use real certificates issued by `x509.ts`: a client that trusts only agent-shim's CA completes a handshake with the provider listener serving the loopback leaf (and the CA-verified health probe passes only against that CA), while the same client cannot talk to a complete door whose leaf a different CA signed, which never sees the request or its credential. `frontdoor/x509.test.ts` checks the certificate encoder against Node's own parser (chain verification, issuer matching, host and IP SAN matching, the validity encoding either side of 2050, positive serials, IPv6 address forms) and mutating the issuer name was confirmed to fail the tests that name it. `frontdoor/headroomHop.test.ts` builds the door from the production `createDoorPipelines`, with the provider listener on TLS, and runs the hop against a scripted daemon serving on a real unix socket inside a real mode 0700 directory under a throwaway root, reached through the production path from the state file through the socket check to `socketPath`, which forwards by `x-headroom-base-url` the way the real one does (its own `x-headroom-*` controls stripped, every other header verbatim): a streamed response relayed event by event, a process squatting on the old TCP port (named by an earlier release's state file, or by a current state file rewritten with a port) never offered a connection whether the daemon is down or serving, a socket directory left world accessible, a symlinked socket directory and a symlinked socket each refused with a 502 naming the reason and nothing sent to the daemon, headroom receiving the Anthropic-shaped request and the translator serving what the daemon forwards back through the direct listener (exactly one daemon request and one upstream call, so the loop cannot double-apply), the project identity re-set on the hop with no session header reaching it, a 502 while the daemon is between restarts, a client disconnect unwinding the hop's own request, and a flag-less session served directly. The credential custody cases: for an http provider, headroom sees only `Bearer agent-shim-sequestered` or `agent-shim-sequestered` and a distinct hop id per request, never the credential, while the provider's upstream receives the door-attached credential from the provider file and neither the client's own presentations nor any of the door's hop headers; with the daemon holding a hop open, the direct listener refuses an unknown hop id, the wrong generation secret, a live id presented for another provider and a missing id, the upstream seeing none of them, and refuses the same id again once the hop has ended; an OAuth session's bearer reaches the daemon untouched with no hop id. `frontdoor/passthrough.test.ts` runs the pass-through against a fake upstream (the provider prefix stripped with the query kept, the host replaced, internal headers absent, an unreachable upstream answered as an Anthropic-shaped 502), `frontdoor/connect.test.ts` keeps the moved TLS round-trip suite (termination, streaming, blind tunnels, port release), every client in it now presenting a live capability the way a real one derives it from `HTTPS_PROXY`, and adds the surface's authentication and limits over real sockets with the tunnel dialler recording every target: a CONNECT with no credential, a wrong capability, a non-Basic scheme, a repeated header or a credential with no password is answered 407 with the Basic challenge, for the intercept host and a blind-tunnel host alike, and nothing is dialled, intercepted or routed for it; a client that has not finished its head by the deadline gets 408 and is closed while a tunnel established earlier still carries bytes past that deadline; an oversized head gets 431; with the pending cap full of half-sent heads, a newcomer evicts the oldest (408) and a live launch's connection still gets through; an authenticated CONNECT beyond the tunnel cap gets 503 without a dial, and is admitted again once a tunnel closes; and a launch whose registry record is pruned has its open tunnel closed at the next revalidation and its capability refused 407 from then on, using the production check over the real registry functions. Each of those guarantees was checked to discriminate by mutating `connect.ts` (skipping the capability check, never arming the deadline, dropping the eviction, the tunnel cap, the revalidation or the size check on a head whose terminator has arrived): every mutant fails the case naming it. `frontdoor/capability.test.ts` covers the credential itself: the proxy URL percent-encodes a token holding every userinfo delimiter and a client's Basic credential carries it back exactly, the password half is read scheme case-insensitively with colons kept, other schemes and empty or missing passwords present nothing, and only an exact live token is accepted, and `frontdoor/latency.test.ts` bounds the added hop: the median through-door request may cost at most two milliseconds over the direct one, a bound two orders of magnitude above a loopback hop's real cost and still small enough to fail if the hop ever grows a request's worth of work. The ordering test discriminates by construction: mutating the pipeline to serve the route before the hop fails three hop tests at once. The authentication and custody tests were checked the same way: letting the probe skip certificate verification, making `ensureFrontDoor` skip the probe, sending the credential to headroom unsequestered, admitting any hop id, and serving the provider listener without TLS each fail the tests that name that guarantee. The launcher suites cover both launch shapes ending at the right place: a provider session keeps the OAuth shape (no base URL, HTTPS_PROXY on the door's CONNECT surface carrying the launch's capability as its proxy credential, the provider named in the session headers, and no credential variable at all) with NODE_EXTRA_CA_CERTS on the door's trust bundle, and an OAuth session keeps its base URL untouched with the same proxy, capability and trust bundle. Both shapes gain NO_PROXY and its lowercase spelling exempting every loopback name, appended to a parent's own exclusions without replacing them and with a repeat entry folded away. The parent's own NODE_EXTRA_CA_CERTS reaches the door port, and a bundle warning is logged. `frontdoor/supervisor.test.ts` runs the listener-host lifecycle over injected ports with a clock advanced only by sleeps: all three sticky ports bound and inherited by a successor generation after a crash, moves off occupied ports logged per listener, failed listener starts recorded as `lastError` with the start lock cleared and the already-bound listeners closed, dead sessions pruned, the idle shutdown closing all three listeners and keeping the sticky addresses (the Remote Control stream hub closed exactly once after the last listener, which a startup failure path performs as well, and never while the door is still serving), the stream hub's close idempotent (a second close persists nothing, asserted in the stream suite's boundary-save case), and the idle clock reset by a new session, the binary turnover at the first empty registry once the executable the door serves as is replaced on disk (before the idle window elapses, through the same graceful shutdown and with the sticky addresses kept), a replaced binary still served while a session is live, the turnover firing at the first empty registry tick after a session live at the replacement ends rather than after fresh idle minutes, and no turnover while the executable is unstatable at startup, gone mid-run, or returning to the exact captured identity after an absent window (the removal window a remove-then-link install leaves), a door superseded by another live supervisor exiting on the first tick after the last session it served ends while a launch made against the owner is still live, leaving the owner's state record exactly as the owner wrote it, a launch registering in the tick after the takeover counted as the superseded door's own, the sessions keyed by either door's own pid ignored so a superseded door with no launches exits at the grace tick, a state file naming a dead supervisor leaving the door serving as the owner, and the supersession and drained exit published on the backbone, with the launch lifecycle published on the event backbone as the registry changes between ticks (a registration, an end by the launcher's own exit hook, a prune of a dead launch, and nothing for the baseline a generation inherits). `frontdoor/ensure.test.ts` covers the launcher bring-up (one supervisor under the lock, a dead holder's lock cleared, joining a serving door, a fatal `lastError` refused immediately, a timeout naming the daemon log) and its listener authentication: the listener is verified before anything is registered, a listener failing verification behind a supervisor this launch spawned is refused with the reason and the log named and no session written, and a recorded live pid whose listener fails is distrusted and replaced by a verified supervisor on another port. `usage/identityClaude.inDoor.test.ts` drives the helper the door's background work plans its launches with (`withIdentityClaude`) from inside the front door's role, over the real launch wiring with a stand-in `claude` on PATH and a throwaway state root, and requires the launch to fail with the named error at its first wait without running anything; with the role guard disabled the same test blocks for the launch's whole wait and fails, which is the stall it exists to catch. `frontdoor/watchdog.test.ts` runs the real watchdog thread with a short cadence and bound (a loop that never beats reported with the stall recorded in `lastError` and the rest of the state record kept, a loop that beats within the bound left alone, the stall recorded and reported while the main thread is blocked in a synchronous wait, and nothing after a stop), pins its derived constants against the launcher's probe patience, and covers the stall's replacement (a replacement always started, naming the stalled pid and the next generation however young the stalled door, the generation stepped per replacement, and none started once the chain has run through every generation); the supervisor suite adds that the watchdog starts after the listeners bind, is beaten each tick, is stopped before the first listener closes, and never starts for a door whose bind failed. `processRole.test.ts` pins the guard that keeps the door's event loop unblocked: a process with no role sleeps and probes as the launcher does, a process holding the front door role has its synchronous sleep and synchronous probe refuse with a named error before any wait or worker starts, and the role is put back when left; the ensure suite adds that a launch which would have to wait fails at its first wait inside the role and that a launch planned inside its own door still takes its shortcut. `frontdoor/probe.test.ts` runs the real synchronous probe against listeners on worker threads (the probe blocks its own thread, so a listener there could never answer): it accepts the loopback leaf from agent-shim's CA and rejects a leaf from another CA for the same names, a leaf from the right CA that does not name 127.0.0.1, a plain-HTTP listener, a closed port, and a squatter that accepts the connection and never answers (failed by the probe's own deadline, not the launcher's outer wait). `frontdoor/trust.test.ts` covers the child's CA bundle: the CA alone when the parent set nothing, a combined bundle that keeps the parent's certificates and is written once per combination, an inherited bundle that already holds the CA reused unchanged, and an unreadable inherited file replaced by the CA alone with a warning naming it. `frontdoor/detach.test.ts` the daemon's independence from the launcher's terminal, and `frontdoor/commands.test.ts` plus `codex/commands.test.ts` the two status commands' output over a fake filesystem, and `frontdoor/commands.test.ts` also runs the real turnover stat against a real temporary directory: the entry's identity reported and followed through a symlink, a staged-then-renamed replacement seen as a different identity (the updater's own install shape), and a missing entry or a directory reported as undefined. `watchedInstallEntry`'s own table lives in the supervisor suite: the resolved entry watched as-is when it is named `agent-shim` (with or without Windows' `.exe`), and the `agent-shim` sibling watched when the door was spawned through the `claude` hardlink, the legacy `claude-use` symlink or any other name, and the own entry watched when no sibling exists (the npm bundle's `dist/cli.cjs` shape). The launcher's own front-door tests inject a fake `FrontDoorPort` to cover the child's base URL, the bring-up order (front door before headroom, since headroom's allowlist needs the door's address), the loud refusal when no port is wired, and that an http provider's launch goes through the door too. `frontdoor/state.test.ts` covers the front-door session registry: a launch whose record carries a capability token is listed (the supervisor's idle decision depends on it, and the supervisor tests register sessions through the real writer for the same reason), the token never appears in a listing, malformed and token-less records are skipped, dead launches are pruned along with their capability, and the registry directory and records are written owner-only; `realPorts.test.ts` proves the real filesystem port produces mode 0700 and 0600 under a permissive umask and narrows entries that already existed wider. `frontdoor/eventHub.test.ts` covers the door's event backbone: delivery to two subscribers of one source and an every-source subscriber with a detached one receiving nothing further, per-source filtering with each source's sequence stamped independently and monotonically (and one name's re-registration publishing through the same sequence), the registered-source listing, the bounded drop-oldest stream bridge whose first survivor's sequence names the gap the dropped events left, a closed stream resolving done and delivering nothing published after the close, and the wrap that puts the Remote Control fan-out on as the backbone's first publisher beside its own unchanged delivery, session filtering included. `frontdoor/quotaExpiringEvents.test.ts` covers the expiring-quota publisher against a fake clock: one event when an unspent window enters its final span and silence on the ticks inside it, nothing for a window outside the span, past its reset or fully used, a re-notice for the next window instance's new reset instant, and the backbone-fed freshness (a written usage event alone moves a window into a span the next check publishes, with no file read). `src/usage/store.test.ts` gains the store's publish dep: every written snapshot is handed to the caller's callback exactly as it stands on disk, a record without an identity writes no snapshot so publishes nothing, and the store runs with no dep wired at all. The supervisor suite gains the door's own sources: the generation and idle-shutdown health events in the order the supervisor lives them, the binary-turnover event before the exit a replaced binary causes, the listener-failure event naming which listener and why, and the expiring-quota check called on every tick. The pool ranking's yielding stickiness is covered in `usage/pick.test.ts`: a member whose seven-day window would run dry at its own observed pace is demoted below one that would not (and a five-hour pace is never converted into seven-day fractions, so a member with only five-hour readings stays unprojected), and the move a resumed conversation makes is named with its reason for the run-dry tail and the extra-usage billing tail, while a kept member and a fresh launch report no move; `launcherPool.test.ts` drives the same move through the real launch and asserts the decision line says which member the conversation moved off and why. `frontdoor/launchEvents.test.ts` covers the launch lifecycle publisher: the silent baseline, the register, end and prune classification carrying the registry's own start times and the observing clock, the departures-before-arrivals ordering with pid order inside each group, and a launch dead on arrival pruned and never registered. `frontdoor/eventsApi.test.ts` mounts the door's whole typed API on the real listener shape over TLS and drives a registry mutation through the real writer, asserting a typed subscriber receives the source-tagged event over the wire, that a caller without the generation's token and a filter naming an unknown source are each refused having delivered no event, and that a source-filtered subscriber receives only the source it named. `frontdoor/listenerReadings.test.ts` reads both listeners of a live door through scripted probes and a stepped clock (provider first, each timed alone, a failing listener's reason carried while the other is still read, nothing probed when no supervisor is alive, a port the state does not record skipped) and formats them for `frontdoor status`; `frontdoor/probe.test.ts` runs the real accept probe against a listener on a worker thread and a closed port; `doctorFrontDoor.test.ts` gains the listener findings (a pass stating the probe's time, a failure naming the sessions the listener cuts off and the restart that replaces it). The self-hosted Remote Control surface's worker web proxies are covered at two layers. `frontdoor/rcWebFetch.test.ts` drives the fetch's bounds over an injected dial and resolver: the private-address rule itself (every refused block, the v4-mapped form, and the fail-closed non-address), the environment lift's exact-value vocabulary, the redirect walk answering the final hop's facts with the requested url, the scheme gate that refuses without dialling, the private refusal by name, by literal and mid-redirect (and the lift that dials it), the unresolvable and malformed refusals, the redirect bound (a chain exactly at it settling, one past it refused), the byte cap, the deadline against a hanging dial, the non-2xx refusal naming its status, and the content-type gate carrying the readable families as text while refusing binary; its last case runs the production dial itself against a loopback stand-in, fetching through the door's own agents with the lift set and refusing the loopback without it. `frontdoor/rcSelfHost.test.ts` covers the serving of both endpoints: the worker web-fetch handing the request's one url to the injected fetch and answering the fetched facts, the OAuth bearer refused where the bridge's own JWT is required, the refusal shape answered as a 200 error object (the form the CLI surfaces as the tool's own failure), the framed answer that escapes the reader cap refused rather than sent to be dropped, and the web-search family (the clear refusal naming that no backend ships, the query and domain-filter validation, a wired backend's results, and a wired backend's refusal). The e2e in `frontdoor/rcSelfHost.e2e.test.ts` adds the real mount: a redirected TLS client speaking the 2.1.289 request shape (worker JWT bearer, anthropic-version, one url) has its fetch served through the transparent surface, the door's own dial following the stand-in's redirect to the final page, and the scheme gate refusing live on the same served path. Breaking the wiring was confirmed to fail the tests that name it: renaming the served branch fails the three serving cases at once, and disabling the private-address check inside the target gate fails the four refusal cases that name it. diff --git a/src/frontdoor/commands.ts b/src/frontdoor/commands.ts index e501b315..4a69a11a 100644 --- a/src/frontdoor/commands.ts +++ b/src/frontdoor/commands.ts @@ -70,7 +70,7 @@ import { answerRcControlRequest, authenticateRcSessionMcpServer, endRcSession, g import { RC_STREAM_BACKOFF_MS, createRcEventFanout, createRcStreamHub, type RcStreamHub } from "./rcStream"; import type { RoutedRequest } from "./route"; import { createFrontDoorServer, listenFrontDoor } from "./server"; -import { DOOR_AWAIT_PID_ENV, DOOR_RESPAWN_MIN_SERVING_MS, DOOR_STALL_BOUND_MS, DOOR_WATCHDOG_CHECK_MS, startThreadWatchdog } from "./watchdog"; +import { DOOR_AWAIT_PID_ENV, DOOR_RESPAWN_GENERATION_ENV, DOOR_RESPAWN_MAX_GENERATIONS, DOOR_STALL_BOUND_MS, DOOR_WATCHDOG_CHECK_MS, startThreadWatchdog } from "./watchdog"; import { readFrontDoorListeners, realFrontDoorListenerProbes } from "./listenerReadings"; import { collectFrontDoorStatus, formatFrontDoorListeners, formatFrontDoorStatus, headroomSocketTarget } from "./status"; import { liveSessionTokens, writeFrontDoorSession, readFrontDoorSessionModels } from "./state"; @@ -84,6 +84,12 @@ import { createUsageStore } from "../usage/store"; import type { UsageRecord } from "../usage/schema"; /** How a door's watchdog starts a replacement: this very executable running the supervisor subcommand, pinned to this door's state root exactly as `spawnDetachedSupervisor` pins it. Every environment value is a string, as the watchdog thread's `spawn` requires. */ +/** The respawn generation this door was started as (`DOOR_RESPAWN_GENERATION_ENV`), or 0 for a door no watchdog started. Read once at startup so the env can be scrubbed before anything is served and the value handed to the supervisor ports explicitly. */ +function readRespawnGeneration(): number { + const generation = Number(process.env[DOOR_RESPAWN_GENERATION_ENV]); + return Number.isInteger(generation) && generation > 0 ? generation : 0; +} + function respawnInvocation(paths: LayoutPaths): { readonly command: string; readonly args: readonly string[]; readonly env: Readonly> } { const env: Record = {}; for (const [name, value] of Object.entries({ ...process.env, AGENT_SHIM_HOME: paths.root })) { @@ -127,7 +133,8 @@ export function realStatOwnExecutable(watchPath: string): () => OwnExecutableSta /** * The real supervisor ports. The three listener starts are where the whole routing assembly is built: the codex translation's real ports, the provider route resolver over them, both pipelines (see `createDoorPipelines`), the three listeners that feed them, and the CA that signs both TLS-serving listeners' leaves. */ -function realFrontDoorSupervisorPorts(paths: LayoutPaths): FrontDoorSupervisorPorts { +/** `respawnGeneration` is the generation this door was started as (see `readRespawnGeneration`), threaded in because the env it came from is scrubbed before the supervisor's lazy `startWatchdog` closure would read it. */ +function realFrontDoorSupervisorPorts(paths: LayoutPaths, respawnGeneration: number): FrontDoorSupervisorPorts { const log = (line: string): void => { appendLog(paths, `frontdoor ${String(process.pid)}: ${line}`); }; @@ -592,7 +599,8 @@ function realFrontDoorSupervisorPorts(paths: LayoutPaths): FrontDoorSupervisorPo action: "kill-process", logFile: paths.frontdoorLogPath, respawn: respawnInvocation(paths), - respawnAfterServingMs: DOOR_RESPAWN_MIN_SERVING_MS, + respawnGeneration, + respawnMaxGeneration: DOOR_RESPAWN_MAX_GENERATIONS, }), // The launch lifecycle publisher over the same backbone the typed API serves: the tick's registry facts go in here, and the door's own register, prune and end moments come out as source-tagged events. observeLaunchRegistry: createLaunchEventPublisher(doorEvents, () => Date.now()).observe, @@ -659,7 +667,13 @@ export function registerFrontDoorCommand(program: Command, deps: CommandDeps): v .allowUnknownOption() .action(async () => { const idleShutdownMinutes = readGlobalConfig(paths)?.frontdoor?.idleShutdownMinutes ?? FRONTDOOR_DEFAULT_IDLE_SHUTDOWN_MINUTES; - const supervisorPorts = realFrontDoorSupervisorPorts(paths); + // A door started by its watchdog's stall handling names the stalled door's pid and waits for it to be gone before binding (see `awaitPidExit`), and carries the respawn generation it was started as, so its own watchdog knows how much of the chain is left. Both are read here, before the ports are built, so the supervisor's lazy watchdog wiring never has to see the env again. + const awaited = Number(process.env[DOOR_AWAIT_PID_ENV]); + const respawnGeneration = readRespawnGeneration(); + // Not inherited by anything this door starts. + Reflect.deleteProperty(process.env, DOOR_AWAIT_PID_ENV); + Reflect.deleteProperty(process.env, DOOR_RESPAWN_GENERATION_ENV); + const supervisorPorts = realFrontDoorSupervisorPorts(paths, respawnGeneration); const ownPid = process.pid; // Every way this process can end writes its reason to the log first, because the log is the only record that outlives it: a door that exits silently looks, from outside, like one that was never there. process.on("exit", (code) => { @@ -672,10 +686,7 @@ export function registerFrontDoorCommand(program: Command, deps: CommandDeps): v deps.exit(0); }); } - // A door started by its watchdog's stall handling names the stalled door's pid and waits for it to be gone before binding (see `awaitPidExit`). - const awaited = Number(process.env[DOOR_AWAIT_PID_ENV]); - // Not inherited by anything this door starts. - Reflect.deleteProperty(process.env, DOOR_AWAIT_PID_ENV); + // A door started by its watchdog's stall handling names the stalled door's pid and waits for it to be gone before binding (see `awaitPidExit`), and carries the respawn generation it was started as, so its own watchdog knows how much of the chain is left. if (Number.isInteger(awaited) && awaited > 0) { const waitedMs = await awaitPidExit(awaited, supervisorPorts); supervisorPorts.log(`frontdoor ${String(ownPid)}: stalled door ${String(awaited)} released the sticky ports after ${String(waitedMs)}ms`); diff --git a/src/frontdoor/watchdog.test.ts b/src/frontdoor/watchdog.test.ts index f8fd6b1a..1fb62833 100644 --- a/src/frontdoor/watchdog.test.ts +++ b/src/frontdoor/watchdog.test.ts @@ -6,7 +6,10 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { FRONTDOOR_PROBE_ATTEMPTS } from "./ensure"; import { PROBE_TIMEOUT_MS } from "./probe"; -import { DOOR_AWAIT_PID_ENV, DOOR_STALL_BOUND_MS, DOOR_WATCHDOG_CHECK_MS, startThreadWatchdog, type DoorWatchdog, type ThreadWatchdogParams } from "./watchdog"; +import { DOOR_AWAIT_PID_ENV, DOOR_RESPAWN_GENERATION_ENV, DOOR_STALL_BOUND_MS, DOOR_WATCHDOG_CHECK_MS, startThreadWatchdog, type DoorWatchdog, type ThreadWatchdogParams } from "./watchdog"; + +/** The respawn ceiling the tests run at: small so the chain-end case needs no real doors, and one above the middle generation so the stepping case can observe a spawn that is not the last. */ +const TEST_RESPAWN_MAX_GENERATION = 3; /** The check cadence the thread tests run at: far shorter than the production one so a stall is seen in a few checks. */ const TEST_CHECK_MS = 20; @@ -60,7 +63,8 @@ describe("startThreadWatchdog", () => { boundMs: TEST_BOUND_MS, checkEveryMs: TEST_CHECK_MS, pauseWaitMs: NO_PAUSE_WAIT_MS, action: "report", - respawnAfterServingMs: 0, + respawnGeneration: 0, + respawnMaxGeneration: TEST_RESPAWN_MAX_GENERATION, onReport: (message) => { if (timer !== undefined) clearInterval(timer); resolve(message); @@ -91,7 +95,7 @@ describe("startThreadWatchdog", () => { it("acts while the main thread is blocked, which is the case it exists for", async () => { const reported = new Promise((resolve) => { - watchdog = startThreadWatchdog({ stateFile, ownPid: process.pid, boundMs: TEST_BOUND_MS, checkEveryMs: TEST_CHECK_MS, pauseWaitMs: NO_PAUSE_WAIT_MS, action: "report", respawnAfterServingMs: 0, onReport: (message) => { + watchdog = startThreadWatchdog({ stateFile, ownPid: process.pid, boundMs: TEST_BOUND_MS, checkEveryMs: TEST_CHECK_MS, pauseWaitMs: NO_PAUSE_WAIT_MS, action: "report", respawnGeneration: 0, respawnMaxGeneration: TEST_RESPAWN_MAX_GENERATION, onReport: (message) => { resolve(message); } }); }); @@ -104,7 +108,7 @@ describe("startThreadWatchdog", () => { it("does nothing once stopped, so a slow orderly retirement is never taken for a stall", async () => { const reported: string[] = []; - watchdog = startThreadWatchdog({ stateFile, ownPid: process.pid, boundMs: TEST_BOUND_MS, checkEveryMs: TEST_CHECK_MS, pauseWaitMs: NO_PAUSE_WAIT_MS, action: "report", respawnAfterServingMs: 0, onReport: (message) => { + watchdog = startThreadWatchdog({ stateFile, ownPid: process.pid, boundMs: TEST_BOUND_MS, checkEveryMs: TEST_CHECK_MS, pauseWaitMs: NO_PAUSE_WAIT_MS, action: "report", respawnGeneration: 0, respawnMaxGeneration: TEST_RESPAWN_MAX_GENERATION, onReport: (message) => { reported.push(message); } }); watchdog.stop(); @@ -118,11 +122,11 @@ describe("startThreadWatchdog", () => { describe("on a stall", () => { const MARKER = "respawned"; - /** Waits for the thread's report, then for the log to hold `needle`, which the thread writes before it reports. */ - async function stallWith(extra: { readonly respawnAfterServingMs: number; readonly respawn?: ThreadWatchdogParams["respawn"] }): Promise { + /** Waits for the thread's report, then for the log, which the thread writes before it reports. */ + async function stallWith(extra: { readonly respawnGeneration?: number; readonly respawnMaxGeneration?: number; readonly respawn?: ThreadWatchdogParams["respawn"] }): Promise { const logFile = path.join(directory, "frontdoor.log"); await new Promise((resolve) => { - watchdog = startThreadWatchdog({ stateFile, ownPid: process.pid, boundMs: TEST_BOUND_MS, checkEveryMs: TEST_CHECK_MS, pauseWaitMs: NO_PAUSE_WAIT_MS, action: "report", logFile, ...extra, onReport: () => { + watchdog = startThreadWatchdog({ stateFile, ownPid: process.pid, boundMs: TEST_BOUND_MS, checkEveryMs: TEST_CHECK_MS, pauseWaitMs: NO_PAUSE_WAIT_MS, action: "report", logFile, respawnGeneration: 0, respawnMaxGeneration: TEST_RESPAWN_MAX_GENERATION, ...extra, onReport: () => { resolve(); } }); }); @@ -130,24 +134,24 @@ describe("startThreadWatchdog", () => { } it("writes the stall to the door's log, the one record that outlives the process", async () => { - const log = await stallWith({ respawnAfterServingMs: 0 }); + const log = await stallWith({}); expect(log).toMatch(new RegExp(`frontdoor ${String(process.pid)}: the front door's event loop did not run`)); }); it("records the machine's load, so a stall the machine caused is told apart from the door's own", async () => { - const log = await stallWith({ respawnAfterServingMs: 0 }); + const log = await stallWith({}); expect(log).toMatch(/machine load at the stall: \d+\.\d \(1m\), \d+\.\d \(5m\), \d+\.\d \(15m\) across \d+ cpus/); }); it("says the main thread could not be paused when the wait is over before it was, rather than naming a frame", async () => { - const log = await stallWith({ respawnAfterServingMs: 0 }); + const log = await stallWith({}); expect(log).toMatch(/the main thread could not be paused within 0ms/); }); it("names the function a blocked main thread was held in", async () => { const logFile = path.join(directory, "frontdoor.log"); const reported = new Promise((resolve) => { - watchdog = startThreadWatchdog({ stateFile, ownPid: process.pid, boundMs: TEST_BOUND_MS, checkEveryMs: TEST_CHECK_MS, pauseWaitMs: PAUSE_WAIT_GUARD_MS, action: "report", logFile, respawnAfterServingMs: 0, onReport: () => { + watchdog = startThreadWatchdog({ stateFile, ownPid: process.pid, boundMs: TEST_BOUND_MS, checkEveryMs: TEST_CHECK_MS, pauseWaitMs: PAUSE_WAIT_GUARD_MS, action: "report", logFile, respawnGeneration: 0, respawnMaxGeneration: TEST_RESPAWN_MAX_GENERATION, onReport: () => { resolve(); } }); }); @@ -164,21 +168,31 @@ describe("startThreadWatchdog", () => { expect(fs.readFileSync(logFile, "utf8")).toMatch(/the main thread was in .*holdTheLoopHere/); }); - it("starts a replacement naming the stalled pid when the door had served long enough", async () => { + it("starts a replacement naming the stalled pid and the next generation, however young the stalled door was", async () => { + const markerFile = path.join(directory, MARKER); + const respawn = { command: process.execPath, args: ["-e", `require("node:fs").writeFileSync(${JSON.stringify(markerFile)}, process.env.${DOOR_AWAIT_PID_ENV} + " gen " + process.env.${DOOR_RESPAWN_GENERATION_ENV})`], env: { PATH: process.env.PATH ?? "" } }; + const log = await stallWith({ respawn }); + expect(log).toMatch(/started replacement door \d+ of generation 1/); + await vi.waitFor(() => { + expect(fs.readFileSync(markerFile, "utf8")).toBe(`${String(process.pid)} gen 1`); + }); + }); + + it("steps the generation a replacement is started as", async () => { const markerFile = path.join(directory, MARKER); - const respawn = { command: process.execPath, args: ["-e", `require("node:fs").writeFileSync(${JSON.stringify(markerFile)}, process.env.${DOOR_AWAIT_PID_ENV})`], env: { PATH: process.env.PATH ?? "" } }; - const log = await stallWith({ respawnAfterServingMs: 0, respawn }); - expect(log).toMatch(/started replacement door \d+/); + const respawn = { command: process.execPath, args: ["-e", `require("node:fs").writeFileSync(${JSON.stringify(markerFile)}, process.env.${DOOR_RESPAWN_GENERATION_ENV})`], env: { PATH: process.env.PATH ?? "" } }; + const log = await stallWith({ respawnGeneration: 1, respawn }); + expect(log).toMatch(/started replacement door \d+ of generation 2/); await vi.waitFor(() => { - expect(fs.readFileSync(markerFile, "utf8")).toBe(String(process.pid)); + expect(fs.readFileSync(markerFile, "utf8")).toBe("2"); }); }); - it("leaves the next launch to start one when the door stalled before it had served long enough", async () => { + it("leaves the next launch's ensure to start one when the stall has already recurred through every replacement generation", async () => { const markerFile = path.join(directory, MARKER); const respawn = { command: process.execPath, args: ["-e", `require("node:fs").writeFileSync(${JSON.stringify(markerFile)}, "x")`], env: { PATH: process.env.PATH ?? "" } }; - const log = await stallWith({ respawnAfterServingMs: Number.MAX_SAFE_INTEGER, respawn }); - expect(log).toMatch(/less than the \d+ms it must have served to be replaced/); + const log = await stallWith({ respawnGeneration: TEST_RESPAWN_MAX_GENERATION, respawn }); + expect(log).toMatch(new RegExp(`not starting another replacement: this door was replacement generation ${String(TEST_RESPAWN_MAX_GENERATION)} of ${String(TEST_RESPAWN_MAX_GENERATION)}`)); expect(fs.existsSync(markerFile)).toBe(false); }); }); diff --git a/src/frontdoor/watchdog.ts b/src/frontdoor/watchdog.ts index 19e81073..95dd67d8 100644 --- a/src/frontdoor/watchdog.ts +++ b/src/frontdoor/watchdog.ts @@ -1,6 +1,5 @@ import { Worker } from "node:worker_threads"; -import { FRONTDOOR_DEFAULT_IDLE_SHUTDOWN_MINUTES } from "../config/schema"; import { FRONTDOOR_PROBE_ATTEMPTS } from "./ensure"; import { PROBE_TIMEOUT_MS } from "./probe"; @@ -9,15 +8,18 @@ import { PROBE_TIMEOUT_MS } from "./probe"; */ export const DOOR_STALL_BOUND_MS = 2 * FRONTDOOR_PROBE_ATTEMPTS * PROBE_TIMEOUT_MS; -const MS_PER_MINUTE = 60_000; +/** The environment variable naming the pid a respawned door waits to see gone before it binds, so the sticky ports are free when it asks for them. */ +export const DOOR_AWAIT_PID_ENV = "AGENT_SHIM_FRONTDOOR_AWAIT_PID"; /** - * How long a door must have served before its watchdog replaces it on a stall: the idle shutdown window a door already stays up through. A door that stalled sooner than that after binding has not shown it can serve, so a stall that recurs at once is treated as deterministic and left to the next launch's ensure rather than respawned in a loop. + * The environment variable naming how many doors the watchdog's respawns have gone through before this one: a door started by `frontdoor restart` or a launch carries nothing here and counts as generation 0, and each watchdog respawn adds one. A door reads it once at startup and passes the number to its own watchdog, which refuses to respawn past `DOOR_RESPAWN_MAX_GENERATIONS`. */ -export const DOOR_RESPAWN_MIN_SERVING_MS = FRONTDOOR_DEFAULT_IDLE_SHUTDOWN_MINUTES * MS_PER_MINUTE; +export const DOOR_RESPAWN_GENERATION_ENV = "AGENT_SHIM_FRONTDOOR_RESPAWN_GENERATION"; -/** The environment variable naming the pid a respawned door waits to see gone before it binds, so the sticky ports are free when it asks for them. */ -export const DOOR_AWAIT_PID_ENV = "AGENT_SHIM_FRONTDOOR_AWAIT_PID"; +/** + * The highest replacement generation the watchdog will start. Every stall gets a replacement as long as a further generation is left, because a young door killed by a stall the machine caused (observed in production: an overloaded host stalling the loop before the door had served a minute) left every live session with connection refused until some later launch happened to start a door. What must not happen is an endless chain of doors each killed at once, so after this many consecutive replacements the chain stops and the next launch's ensure takes over: each link already costs at least one stall bound, so a stall that has recurred this many times with no door serving in between is the machine's, and respawning faster than a launch cannot fix it. + */ +export const DOOR_RESPAWN_MAX_GENERATIONS = 3; /** How many times the thread looks at the beat counter within one probe deadline, so a stall is noticed within a small fraction of the bound. */ const CHECKS_PER_PROBE_DEADLINE = 2; @@ -37,15 +39,14 @@ export interface DoorWatchdog { type StallAction = "kill-process" | "report"; /** - * The watchdog thread: its own event loop, which a blocked main loop cannot stall. It compares a shared beat counter against its previous reading on a fixed cadence, and when the counter has not moved for `boundMs` it records the stall in the state file (the existing `lastError` field, so no reader of the file sees a new key) and in the door's log (the next generation's first state write replaces `lastError`, and a SIGKILL runs no handler, so the log line is the only record that outlives the replacement), logs the machine's load and where the main thread was (see `describeMainThread`, so a stall that was the door's own code is told apart from one the machine caused), starts a replacement door when the door had served long enough to have proved it can, and ends the process. A thread that was itself descheduled for longer than its cadence (the machine was suspended or starved) restarts its clock instead of judging, since the main loop stalled by the same pause is not a hung one. + * The watchdog thread: its own event loop, which a blocked main loop cannot stall. It compares a shared beat counter against its previous reading on a fixed cadence, and when the counter has not moved for `boundMs` it records the stall in the state file (the existing `lastError` field, so no reader of the file sees a new key) and in the door's log (the next generation's first state write replaces `lastError`, and a SIGKILL runs no handler, so the log line is the only record that outlives the replacement), logs the machine's load and where the main thread was (see `describeMainThread`, so a stall that was the door's own code is told apart from one the machine caused), starts a replacement door (as long as this door's own generation has not reached the respawn ceiling, so a stall that recurs through consecutive doors ends the chain rather than looping) and ends the process. A thread that was itself descheduled for longer than its cadence (the machine was suspended or starved) restarts its clock instead of judging, since the main loop stalled by the same pause is not a hung one. */ const WATCHDOG_SOURCE = ` const { workerData, parentPort } = require("node:worker_threads"); const fs = require("node:fs"); const os = require("node:os"); const inspector = require("node:inspector"); -const { shared, boundMs, checkEveryMs, pauseWaitMs, stateFile, ownPid, action, logFile, respawn, respawnAfterServingMs, awaitPidEnv } = workerData; -const startedAt = Date.now(); +const { shared, boundMs, checkEveryMs, pauseWaitMs, stateFile, ownPid, action, logFile, respawn, respawnGeneration, respawnMaxGeneration, awaitPidEnv, respawnGenerationEnv } = workerData; const counter = new Int32Array(shared); let lastValue = Atomics.load(counter, 0); let lastChangeAt = Date.now(); @@ -111,17 +112,16 @@ const describeMainThread = (waitMs, resume) => new Promise((resolve) => { }); const spawnReplacement = () => { if (respawn === undefined || logFile === undefined) return; - const servedMs = Date.now() - startedAt; - if (servedMs < respawnAfterServingMs) { - log("the door served for " + String(servedMs) + "ms before stalling, less than the " + String(respawnAfterServingMs) + "ms it must have served to be replaced; leaving the next launch to start one"); + if (respawnGeneration >= respawnMaxGeneration) { + log("not starting another replacement: this door was replacement generation " + String(respawnGeneration) + " of " + String(respawnMaxGeneration) + ", so the stall has recurred through consecutive doors and the machine, not the door, is the one refusing to serve; leaving the next launch's ensure to start one"); return; } try { const out = fs.openSync(logFile, "a"); - const child = require("node:child_process").spawn(respawn.command, respawn.args, { detached: true, stdio: ["ignore", out, out], env: Object.assign({}, respawn.env, { [awaitPidEnv]: String(ownPid) }) }); + const child = require("node:child_process").spawn(respawn.command, respawn.args, { detached: true, stdio: ["ignore", out, out], env: Object.assign({}, respawn.env, { [awaitPidEnv]: String(ownPid), [respawnGenerationEnv]: String(respawnGeneration + 1) }) }); child.unref(); fs.closeSync(out); - log("started replacement door " + String(child.pid) + ", which binds once this process is gone"); + log("started replacement door " + String(child.pid) + " of generation " + String(respawnGeneration + 1) + ", which binds once this process is gone"); } catch (error) { log("could not start a replacement door: " + (error instanceof Error ? error.message : String(error))); } @@ -176,9 +176,11 @@ export interface ThreadWatchdogParams { readonly action: StallAction; /** The door's log file, which the thread appends the stall to before ending the process. Omitted by a host with no log of its own. */ readonly logFile?: string; - /** How long the door must have served before a stall is answered with a replacement (`DOOR_RESPAWN_MIN_SERVING_MS` in production). */ - readonly respawnAfterServingMs: number; - /** See `respawnAfterServingMs`: how to start a replacement door on a stall: the invocation and environment of a door, to which the thread adds `DOOR_AWAIT_PID_ENV`. Omitted by a host that does not respawn. */ + /** How many watchdog respawns produced this door (`0` for a door no watchdog started), from `DOOR_RESPAWN_GENERATION_ENV` in production. */ + readonly respawnGeneration: number; + /** The highest replacement generation this door's watchdog may start (`DOOR_RESPAWN_MAX_GENERATIONS` in production). */ + readonly respawnMaxGeneration: number; + /** How to start a replacement door on a stall: the invocation and environment of a door, to which the thread adds `DOOR_AWAIT_PID_ENV` and `DOOR_RESPAWN_GENERATION_ENV`. Omitted by a host that does not respawn. */ readonly respawn?: { readonly command: string; readonly args: readonly string[]; readonly env: Readonly> }; /** Called on the main thread with the recorded message when `action` is `report` and the stall was seen. */ readonly onReport?: (message: string) => void; @@ -190,7 +192,7 @@ export function startThreadWatchdog(params: Readonly): Doo const counter = new Int32Array(shared); const worker = new Worker(WATCHDOG_SOURCE, { eval: true, - workerData: { shared, boundMs: params.boundMs, checkEveryMs: params.checkEveryMs, pauseWaitMs: params.pauseWaitMs, stateFile: params.stateFile, ownPid: params.ownPid, action: params.action, logFile: params.logFile, respawn: params.respawn, respawnAfterServingMs: params.respawnAfterServingMs, awaitPidEnv: DOOR_AWAIT_PID_ENV }, + workerData: { shared, boundMs: params.boundMs, checkEveryMs: params.checkEveryMs, pauseWaitMs: params.pauseWaitMs, stateFile: params.stateFile, ownPid: params.ownPid, action: params.action, logFile: params.logFile, respawn: params.respawn, respawnGeneration: params.respawnGeneration, respawnMaxGeneration: params.respawnMaxGeneration, awaitPidEnv: DOOR_AWAIT_PID_ENV, respawnGenerationEnv: DOOR_RESPAWN_GENERATION_ENV }, }); worker.unref(); worker.on("message", (message: unknown) => {