Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/technical-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -563,14 +563,14 @@ A long-running standalone proxy (v0.1.208 field report: dsh-harness lane, last l

Design:

- **Watchdog** (`src/event-loop-watchdog.ts`): a separate worker thread OWNS the clock — the worker's own event loop keeps ticking even when the main thread is permanently blocked, which is the whole point. Main posts a heartbeat every `WATCHDOG_HEARTBEAT_MS` (5 s); the worker checks every 5 s: gap ≥ `WATCHDOG_SOFT_WARN_MS` (30 s) → ONE forensic warn line (uptime/RSS); gap ≥ `WATCHDOG_HARD_TIMEOUT_MS` (300 s) → FATAL line to bili.log + stderr, then `process.exit(1)` (exit mode). Thresholds are hard-coded by design (the config surface is owner-gated): legitimate big compressions finish in seconds-to-minutes and yield the loop between chunks, so a CONTINUOUS five-minute silence is only ever a deadlock. Armed at the ENTRY of `startServer` (before the twin probe, engine/storage init and hydrate all run under it — a wedge can arrive during startup, and #4762's own repro class is sync-I/O hotspots) — standalone processes ONLY; plugin-injection mode never arms (it would kill the HOST process). Forensic lines route to bili.log whenever a file leg is configured; the worker's STDERR leg ships only when stderr is a TTY or there is no file leg at all — a blocking write into a full, undrained pipe never fires an error event (#5012 class), so a worker writing its FATAL line into exactly that dead pipe would wedge ITSELF instead of tripping the exit; the gate is structural, not defensive. The worker is unref'd so the safety net can never hold a clean natural process exit hostage. The worker runs as inline `eval` code because `dist/index.js` is a single bundle with no sibling files to reference; its only import is a guarded `require("worker_threads")` — the `parentPort` global does NOT exist in eval workers (ReferenceError, verified on Node 22). Worker-spawn failure degrades to unwatched with one warn and never breaks startup (availability over guardrail).
- **Watchdog** (`src/event-loop-watchdog.ts`): a separate worker thread OWNS the clock — the worker's own event loop keeps ticking even when the main thread is permanently blocked, which is the whole point. Main posts a heartbeat every `WATCHDOG_HEARTBEAT_MS` (5 s); the worker checks every 5 s: gap ≥ `WATCHDOG_SOFT_WARN_MS` (30 s) → ONE forensic warn line (uptime/RSS); gap ≥ `WATCHDOG_HARD_TIMEOUT_MS` (300 s) → FATAL line to bili.log + stderr, then a whole-process kill — `process.kill(process.pid, "SIGKILL")` issued FROM THE WORKER (exit mode). Not `process.exit()`: inside a worker thread that ends only the worker, which is exactly how the wedged proxy used to survive its own watchdog and run on unwatched (#6157); SIGKILL needs nothing from the blocked main thread (a JS SIGTERM listener would never run), and on Windows `process.kill` maps to TerminateProcess. Thresholds are hard-coded by design (the config surface is owner-gated): legitimate big compressions finish in seconds-to-minutes and yield the loop between chunks, so a CONTINUOUS five-minute silence is only ever a deadlock. Armed at the ENTRY of `startServer` (before the twin probe, engine/storage init and hydrate all run under it — a wedge can arrive during startup, and #4762's own repro class is sync-I/O hotspots) — standalone processes ONLY; plugin-injection mode never arms (it would kill the HOST process). Forensic lines route to bili.log whenever a file leg is configured; the worker's STDERR leg ships only when stderr is a TTY or there is no file leg at all — a blocking write into a full, undrained pipe never fires an error event (#5012 class), so a worker writing its FATAL line into exactly that dead pipe would wedge ITSELF instead of tripping the exit; the gate is structural, not defensive. The worker is unref'd so the safety net can never hold a clean natural process exit hostage. The worker runs as inline `eval` code because `dist/index.js` is a single bundle with no sibling files to reference; its only import is a guarded `require("worker_threads")` — the `parentPort` global does NOT exist in eval workers (ReferenceError, verified on Node 22). Worker-spawn failure degrades to unwatched with one warn and never breaks startup (availability over guardrail).
- **Holder diagnostics** (`src/port-owner.ts` + both EADDRINUSE death paths in `src/server.ts`): at the moment of death, RE-probe the existing listener's `/__bili/health` once (2 s budget; wildcard binds probe through a loopback origin — `holderProbeOrigin`). Answered → the hint states the peer IS serving and the spawner should attach to it; no answer → the hint states the holder appears wedged or dead and names the concrete release: `taskkill /PID N` (Windows) / `kill N` (POSIX) when a pid is known (responder pid beats the stale registry entry; falls back to the same-lane registry pid), else `netstat/lsof`. Logged as `[eaddrinuse-diag]` and written to the spawn port log when `BILI_LAUNCH_TOKEN` is set (this child's stderr is muted into bili.log, #2864). Covers BOTH paths: the pre-engine early twin-collision exit and the post-bind `server.on("error")` terminal branch.
- **Why re-probe instead of trusting the boot-time verdict:** the early twin check (`findVerifiedSameLaneTwin`) runs seconds before bind, and the late path's `classifySameLanePeer` is registry-only (pid alive, NO health check). A wedge developing in between — exactly the #4762 shape — flips the truth under both. And one shallow probe cannot distinguish slow-alive from wedged-but-draining (the zombie answered some probes), which is why the wording says "appears wedged" and leaves the kill decision to the operator/spawner rather than acting on it.
- **Out of scope (documented):** a parent-side force-kill after repeated EADDRINUSE belongs to the SPAWNER (the DeepSeek Harness), not this repo — the diagnostic line now tells it exactly what to do.

Behavior change (disclosed): OLD = a wedged standalone proxy lived forever holding its port and every respawn died with an opaque EADDRINUSE; NEW = continuous main-loop silence ≥ 5 min ⇒ one warn at 30 s, one FATAL log line, then `process.exit(1)` so a spawner's normal respawn recovers the port, plus a serving/wedged adjudication + kill guidance appended to every terminal EADDRINUSE message. No new config/env/CLI surface — thresholds are module constants. Plugin-mode behavior is unchanged (watchdog not armed there).
Behavior change (disclosed): OLD = a wedged standalone proxy lived forever holding its port and every respawn died with an opaque EADDRINUSE; NEW = continuous main-loop silence ≥ 5 min ⇒ one warn at 30 s, one FATAL log line, then a whole-process SIGKILL from the watchdog worker (#6157: `process.exit(1)` in a worker ends only the worker) so a spawner's normal respawn recovers the port — POSIX children end by signal (`code: null`), Windows via TerminateProcess — plus a serving/wedged adjudication + kill guidance appended to every terminal EADDRINUSE message. No new config/env/CLI surface — thresholds are module constants. Plugin-mode behavior is unchanged (watchdog not armed there).

Tests: `tests/event-loop-watchdog.test.ts` (verdict boundary table incl. warn-once semantics; default pin; REAL-worker wedge integration — a synchronous spin blocks the main loop past the hard threshold, the trip fires, both forensic lines land in the log file with the `[v=…]` tag; healthy stream never trips; spawn-failure seam degrades cleanly) + `tests/port-owner.test.ts` additions (wildcard-origin rewrite; serving/wedged/anonymous hint wording; live-responder pid beats stale registry pid; dead holder reads as not-serving; pre-registration "starting" reads as not-serving).
Tests: `tests/event-loop-watchdog.test.ts` (verdict boundary table incl. warn-once semantics; default pin; REAL-worker wedge integration — a synchronous spin blocks the main loop past the hard threshold, the trip fires, both forensic lines land in the log file with the `[v=…]` tag; healthy stream never trips; spawn-failure seam degrades cleanly; real "exit" mode in a child process — a wedged main thread dies with the whole process and the FATAL line lands first, #6157) + `tests/port-owner.test.ts` additions (wildcard-origin rewrite; serving/wedged/anonymous hint wording; live-responder pid beats stale registry pid; dead holder reads as not-serving; pre-registration "starting" reads as not-serving).

## Dependency audit — zod advisory GHSA-mrhx-c7xc-m3w5 and the jimp nested-zod false positive (#4820)

Expand Down
15 changes: 14 additions & 1 deletion src/codex-compact.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { randomUUID } from "node:crypto";
import { createHash, randomUUID } from "node:crypto";
import type { CompressionBlock } from "acp-kernel";
import { statusInputBaseline, type Session } from "./session.js";
import { codexCompactMode as knobCodexCompactMode } from "./knobs.js";
Expand Down Expand Up @@ -161,6 +161,19 @@ export function isBiliCompactionItem(item: unknown): boolean {
return false;
}

/** Native opaque checkpoints are explicit epoch boundaries; never decode them.
* The last checkpoint identifies the current epoch across echoes and restarts. */
export function nativeCompactionKey(input: unknown): string | undefined {
if (!Array.isArray(input)) return undefined;
for (let i = input.length - 1; i >= 0; i--) {
const item = input[i] as { type?: unknown; encrypted_content?: unknown } | null;
if (!item || item.type !== "compaction" || isBiliCompactionItem(item)) continue;
if (typeof item.encrypted_content !== "string" || item.encrypted_content.length === 0) continue;
return createHash("sha256").update(item.encrypted_content).digest("hex");
}
return undefined;
}

export function stripBiliCompactionItems<T>(input: T[]): T[] {
return input.filter((item) => !isBiliCompactionItem(item));
}
Expand Down
13 changes: 13 additions & 0 deletions src/codex-models.ts
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,19 @@ export function readCodexModelWindow(model: string, codexHome: string): number |
return readCodexLiveModelWindow(model, codexHome)?.window;
}

/** True when a codex-aligned window ALREADY nets out codex's own output
* headroom: the live table sources multiply by effective_context_window_percent,
* which codex defines as the share "usable for inputs, after reserving headroom
* for system prompts, tool overhead, and model output". Reserving bili's output
* headroom on top of such a window counts the output budget twice and puts the
* preflight wall below the point where codex itself auto-compacts (90% of the
* raw window), so a long codex session hits an unrecoverable 413 before codex
* ever compacts it. Other sources (client metadata, config.toml window, bundled
* snapshot, 272K fallback) are raw windows and keep the reservation. */
export function codexWindowNetsOutputHeadroom(source: string | undefined): boolean {
return source === "codex-catalog" || source === "codex-models-cache";
}

function alignedResult(limit: number, w: number, source: CodexWindowSource): { limit: number; clamped: boolean; source: CodexWindowSource } {
return limit > w ? { limit: w, clamped: true, source } : { limit, clamped: false, source };
}
Expand Down
29 changes: 20 additions & 9 deletions src/event-loop-watchdog.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,16 @@
* Mechanism: a worker thread has its OWN event loop, so it keeps running while
* the main thread spins. The main thread posts a heartbeat every 5 s; the
* worker measures the gap and, after SOFT_WARN_MS of continuous silence, logs
* a warning; after HARD_TIMEOUT_MS it writes a forensic line and exits the
* WHOLE process (code 1) so the spawner respawns onto the freed port. Legit
* work never blocks the loop for five straight minutes — compression runs in
* sync slices that complete in seconds to minutes and yield between requests —
* so the hard threshold is deadlock detection, not load detection.
* a warning; after HARD_TIMEOUT_MS it writes a forensic line and kills the
* WHOLE process so the spawner respawns onto the freed port. The kill is a
* SIGKILL to its own pid, NOT process.exit(): inside a worker thread
* process.exit() ends only the worker, so the wedged proxy used to survive its
* own watchdog and run on unwatched. SIGKILL needs nothing from the blocked
* main thread (a JS SIGTERM listener would never run); on Windows
* process.kill maps to TerminateProcess. Legit work never blocks the loop for
* five straight minutes — compression runs in sync slices that complete in
* seconds to minutes and yield between requests — so the hard threshold is
* deadlock detection, not load detection.
*
* Scope discipline: plugin-injection mode (dist/agent/*) never calls
* startServer, so a wedged HOST process is never at risk of being killed by
Expand Down Expand Up @@ -68,9 +73,10 @@ interface ArmEventLoopWatchdogOptions {
softWarnMs?: number;
hardTimeoutMs?: number;
checkIntervalMs?: number;
/** "exit" (production): the worker calls process.exit itself — required,
* because a wedged main thread can never receive a message asking it to
* die. "report": the worker posts {type:"trip"} and stays alive — test
/** "exit" (production): the worker SIGKILLs its own process itself,
* required because a wedged main thread can never receive a message
* asking it to die (and process.exit() in a worker ends only the worker).
* "report": the worker posts {type:"trip"} and stays alive — test
* seam so the suite process is not killed. */
fatalMode?: "exit" | "report";
/** Test seam: swap the Worker constructor. */
Expand Down Expand Up @@ -138,6 +144,11 @@ const WORKER_CODE = [
" if (emitStderr) { try { process.stderr.write(row); } catch (e) {} }",
"}",
"",
"function killWholeProcess() {",
" try { process.kill(process.pid, \"SIGKILL\"); } catch (e) {}",
" process.exit(1);",
"}",
"",
"function tick() {",
" const gap = Date.now() - lastBeat;",
" const v = verdict(gap);",
Expand All @@ -148,7 +159,7 @@ const WORKER_CODE = [
" tripped = true;",
" line(\"error\", \"[event-loop-watchdog] FATAL: main event loop silent for \" + fmt(gap) + \" (hard threshold \" + fmt(hardTimeoutMs) + \") — exiting so the spawner can respawn onto the freed port (#4762)\");",
" try { parentPort.postMessage({ type: \"trip\", gapMs: gap }); } catch (e) {}",
" if (fatalMode === \"exit\") process.exit(1);",
" if (fatalMode === \"exit\") killWholeProcess();",
" }",
"}",
"",
Expand Down
Loading
Loading