diff --git a/CONFIGURATION.md b/CONFIGURATION.md index be710a94c8..79decc9ead 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -175,11 +175,9 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | Repeat interval for the EPERM alert; 0 logs it once. | | `storage` | object | {} (all defaults) | — | Unified SQLite storage sidecar settings (#2671). | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | Tiered storage-migration switch (#2671): shadow = unified SQLite double-write dry-run with digest reconciliation; full = drop-in read path (unified-first loads with organic import, legacy fallback). Absent everywhere now defaults to FULL (#3818 flip - the shadow dry-run reconciled cleanly since #2815, so the index is the load path; JSON envelopes keep being written, and legacy remains the one-line escape hatch). File tier is strict legacy/shadow/full; the env tier also accepts 1/true (=shadow) and unified (=full) and wins over the file. Resolved once per lane boot - hot edits need a restart. Encryption (BILI_ENCRYPTION_KEY) hard-blocks the sidecar. | -| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | Storage backend selection, step 1 of the pluggable-storage plan (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract, with sqlite as the default and today the only real implementation (the existing UnifiedStore behind a Promise-wrapped adapter - zero behavior change, no schema change). mysql is a REGISTERED STUB: the name validates so configs do not bounce between unknown/unimplemented as the real backend ships (step 2, mysql2 client via the optional-dependency + self-download discipline), but selecting it logs one notice line at lane boot and the store stays on sqlite; paths that open a backend directly (export) fail with the documented error. Unknown names also fall back to sqlite with a notice. Resolved once per lane boot. | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER RUNTIME, because a binding cannot be loaded outside the range its package declares: Node >= 22 with N-API >= 10 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), while Node 20 and the Node 22.0-22.12 band - which passes v13's major floor but only ships N-API 9 - use v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding). That keeps a low Node on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | | `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | Storage backend selection (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract. sqlite is the default (the existing UnifiedStore behind a Promise-wrapped adapter). mysql is FULLY WIRED (steps 2+3): openBackend("mysql", dsn) opens the real MysqlBackend against MySQL 8.0 — the full contract surface (ingest with CAS refcounts, reads, session_ext, reap journal, archive/restore/gc, checkpoints, conversations), with mysql2 bundled as a regular dependency and loaded lazily (boots that never select mysql never load it). The proxy hot path is wired through the async shadow facade: saves enqueue onto a serial backend queue (watermark deltas, orphan/reap journal, digest sampling — sqlite keeps its zero-await sync fast path unchanged), SYNC reads ride the JSON envelope (the sidecar attaches asynchronously via ready(); viewFor/importRecord sync facades miss by design), replicas/ext/GC/export speak the async twins. writerOff stays sqlite-only (the envelope is the mysql sync read path). Unknown names fall back to sqlite with a notice. Resolved once per lane boot. | | `storage.mysql.dsn` | string | unset | BILI_MYSQL_DSN | MySQL DSN for storage.backend="mysql" (#3388 step 2, #3931): mysql://user:pass@host:3306/db. Consumed by the StorageBackend contract paths (hot path included — the async shadow queue). env BILI_MYSQL_DSN wins over the file key. Empty/unset = the mysql backend cannot open — the documented error names both knobs and the working default. | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | +| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER RUNTIME, because a binding cannot be loaded outside the range its package declares: Node >= 22 with N-API >= 10 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), while Node 20 and the Node 22.0-22.12 band - which passes v13's major floor but only ships N-API 9 - use v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding). That keeps a low Node on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). Env wins over the file tier. | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 5ae78cbb7f..3c6dbbf737 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -175,11 +175,9 @@ | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | EPERM 告警的重复间隔;0 只记录一次。 | | `storage` | object | {} (all defaults) | — | 统一 SQLite 存储边车设置(#2671)。 | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | 分层存储迁移开关(#2671):shadow = 统一 SQLite 双写干跑 + digest 对账;full = 平替读路径(unified-first 加载 + 有机导入 + legacy 兜底)。两处均未设置时默认 full(#3818 翻转 —— shadow 干跑自 #2815 起对账一直干净,索引因此成为默认加载路径;JSON 信封照写,legacy 仍是一行逃生口)。文件层只接受 legacy/shadow/full;环境变量层还接受 1/true(=shadow)与 unified(=full)并优先于文件。每个 lane 启动时解析一次 —— 热改动需重启生效。加密(BILI_ENCRYPTION_KEY)会硬阻断边车。 | -| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | 存储后端选择,可插拔存储计划第一步(#3388 / #3931):统一存储消费操作级的异步 StorageBackend 契约,sqlite 为默认且是当前唯一真实现(现有 UnifiedStore 套一层 Promise 包装适配器 —— 零行为变化、零 schema 变化)。mysql 是已注册占位:该名字能通过校验(真实后端随第二步落地 —— mysql2 客户端,走可选依赖+自下载纪律 —— 期间配置不会在 unknown/unimplemented 之间来回弹),但选中它只会在 lane 启动时打一行提示并继续用 sqlite;直接开后端的路径(export)按文档化的错误失败。未知名字同样回退 sqlite 并提示。每个 lane 启动时解析一次。 | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按运行时特性分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 且 N-API >= 10 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),而 Node 20 与 Node 22.0–22.12(满足 v13 的大版本下限但只带 N-API 9)用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入)—— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | | `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | 存储后端选择(#3388 / #3931):统一存储消费操作级的异步 StorageBackend 契约。sqlite 为默认(现有 UnifiedStore 套一层 Promise 包装适配器)。mysql 已完整接线(第二+三步):openBackend("mysql", dsn) 打开真正的 MysqlBackend,对接 MySQL 8.0 —— 完整契约面(带 CAS 引用计数的 ingest、读取、session_ext、收割台账、归档/恢复/GC、checkpoint、会话绑定),mysql2 作为常规依赖打包并按需懒加载(从不选 mysql 的启动永不加载它)。代理热路径经异步影子 facade 接线:保存进串行后端队列(水位 delta、孤儿/收割台账、digest 采样 —— sqlite 保持零 await 同步快车道不变),同步读走 JSON 信封(sidecar 经 ready() 异步挂载;viewFor/importRecord 同步面按设计 miss),replicas/ext/GC/export 走异步孪生。writerOff 仅 sqlite(信封就是 mysql 的同步读路径)。未知名字回退 sqlite 并提示。每个 lane 启动时解析一次。 | | `storage.mysql.dsn` | string | unset | BILI_MYSQL_DSN | storage.backend="mysql" 的 MySQL DSN(#3388 第二步,#3931):mysql://user:pass@host:3306/db。由 StorageBackend 契约路径消费(含热路径 —— 异步影子队列)。环境变量 BILI_MYSQL_DSN 优先于文件键。空/未设置 = mysql 后端无法打开 —— 文档化错误会同时列出两个旋钮和可用的默认值。 | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | +| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按运行时特性分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 且 N-API >= 10 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),而 Node 20 与 Node 22.0–22.12(满足 v13 的大版本下限但只带 N-API 9)用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入)—— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。环境变量优先于配置文件层。 | diff --git a/src/agent/native-intercept.ts b/src/agent/native-intercept.ts index c968ed67e3..90b5fdb1f5 100644 --- a/src/agent/native-intercept.ts +++ b/src/agent/native-intercept.ts @@ -499,6 +499,14 @@ export async function readyOrigin(state: NativeInterceptState): Promise(); + /** Record the origin of an already-routed model URL on the shared state and * fire the owner hook on a transition (first observation or a NEW origin). * MUST be called before any gate/await in the dispatch path: the recording @@ -515,6 +523,15 @@ export function noteRoutedOrigin(state: NativeInterceptState, url: string): void } if (state.routedOrigin === origin) return; state.routedOrigin = origin; + if (state.onRoutedOriginObserved === undefined) { + const held = state.origin; + const normalizedHeld = held === undefined ? undefined : held.replace(/\/+$/, "").toLowerCase(); + if (held !== undefined && normalizedHeld !== origin.toLowerCase() && !routedMismatchWarned.has(origin)) { + if (routedMismatchWarned.size >= 128) routedMismatchWarned.clear(); + routedMismatchWarned.add(origin); + logHostEvent("error", "[native-client]", `bili-native: model traffic observed at ${origin} but this process's proxy is ${held} — requests will split across two bili instances and same-conversation state will desync (#394); align your client's /bili/ base URL with ${held} or stop one of the instances (#3923)`); + } + } state.onRoutedOriginObserved?.(origin); } diff --git a/src/dead-ranges.ts b/src/dead-ranges.ts new file mode 100644 index 0000000000..e41b791d4b --- /dev/null +++ b/src/dead-ranges.ts @@ -0,0 +1,96 @@ +import { markDirty, type Session } from "./session.js"; + +// #2638: session-level memory of spans whose preflight preview returned the kernel's +// STRUCTURAL "no new compressible messages" verdict. The kernel keeps advertising such spans +// (its recommend side does not mirror applySingleRange's live-carrier carve-out — see the +// issue for the minimal geometry), so without this memory every request re-nominates and +// re-rejects them: wasted preview work, warn spam, and — because the span's mass can never +// fold through plain refs — an auto-fold no-progress backoff cycle that resurrects classic +// nudges pointing the model at the same dead span. Entries are keyed by the exact +// startRef:endRef the kernel resolved; chunk-boundary shifts across turns self-heal through +// one extra preview (record again), and the cap bounds total memory. + +const MAX_DEAD_RANGES = 64; +const STORAGE_KEY = "preflightDeadRanges"; + +interface DeadRangeEntry { + key: string; + startRef: string; + endRef: string; + /** Block ids named in the verdict ("covered by active block(s) b3, b5") — diagnostics + * only; invalidation deliberately does NOT trust the name list (see activeSnapshot). */ + blocks: string[]; + /** Sorted active block ids at record time. The entry stays dead only while EVERY one of + * them is still active (subsumption). */ + activeSnapshot: string[]; +} + +function readStore(session: Session): DeadRangeEntry[] { + const raw = session.metadata?.[STORAGE_KEY]; + if (!Array.isArray(raw)) return []; + return raw.filter((e): e is DeadRangeEntry => + !!e && typeof e === "object" + && typeof (e as DeadRangeEntry).key === "string" + && typeof (e as DeadRangeEntry).startRef === "string" + && typeof (e as DeadRangeEntry).endRef === "string" + && Array.isArray((e as DeadRangeEntry).blocks) + && Array.isArray((e as DeadRangeEntry).activeSnapshot)); +} + +function writeStore(session: Session, store: DeadRangeEntry[]): void { + (session.metadata ?? (session.metadata = {}))[STORAGE_KEY] = store; + markDirty(session); +} + +function activeBlockIds(session: Session): string[] { + return session.state.blocks.filter((b) => b.active).map((b) => b.blockId).sort(); +} + +/** Both structural-empty wordings the kernel emits for a span with zero foldable content: + * - the #199 livelock guard: "...contains no new compressible messages — every message in it + * is already covered by active block(s) b3..." + * - validateCompressionRange: "Range contains no compressible messages — all are already + * covered by active blocks or protected." + * Returns the named covering blocks ([] for the unnamed variant); null when the verdict is + * NOT structural-empty — transient, zone, min-gate and length rejections must stay retryable + * (they can clear mid-invocation or next turn). Feed the RAW kernel error strings, not a + * truncated/joined rendering: the named-block list must survive intact. */ +export function parseStructuralEmptyVerdict(verdict: string): string[] | null { + if (!/no (?:new )?compressible messages/.test(verdict)) return null; + const named = /active block\(s\) ([A-Za-z0-9]+(?:,\s*[A-Za-z0-9]+)*)/.exec(verdict)?.[1] ?? ""; + return named ? named.split(/,\s*/).filter(Boolean) : []; +} + +export function recordDeadRange(session: Session, startRef: string, endRef: string, namedBlocks: string[]): void { + const key = `${startRef}:${endRef}`; + const store = readStore(session).filter((e) => e.key !== key); + store.push({ key, startRef, endRef, blocks: namedBlocks, activeSnapshot: activeBlockIds(session) }); + while (store.length > MAX_DEAD_RANGES) store.shift(); + writeStore(session, store); +} + +/** Keys still dead under the current state. Prunes invalidated entries in place. + * + * Invalidation is SNAPSHOT SUBSUMPTION, not the verdict's name list: the livelock guard names + * only the NESTED blocks whose coverage made the span empty, but the span's deadness also + * rests on whatever else carved its members out — notably LIVE CARRIER blocks (a checkpoint + * message's summaryOfBlockId target must stay active or the carrier becomes plain-foldable + * again), and those targets are never named in the verdict. Any active block dying anywhere + * in the session (consumed by a distillation, deactivated, expanded) therefore wakes every + * entry conservatively; append-only message growth never does, and a false wake costs exactly + * one CPU-only preview before the entry re-records itself. */ +export function deadSkipKeys(session: Session): Set { + const current = new Set(activeBlockIds(session)); + const seen = readStore(session); + const kept = seen.filter((e) => e.activeSnapshot.every((id) => current.has(id))); + if (kept.length !== seen.length) writeStore(session, kept); + return new Set(kept.map((e) => e.key)); +} + +/** Drop remembered-dead ranges from an advertised list (nudge surface, preflight candidates). + * Exact key match only — a wider advertised range that merely CONTAINS a dead chunk stays: + * its live members outside the chunk are genuinely foldable. */ +export function filterDeadRanges(session: Session, ranges: readonly T[]): T[] { + const dead = deadSkipKeys(session); + return ranges.filter((r) => !dead.has(`${r.startRef}:${r.endRef}`)); +} diff --git a/src/persist.ts b/src/persist.ts index 608e5a8650..bf5919bdbe 100644 --- a/src/persist.ts +++ b/src/persist.ts @@ -349,6 +349,17 @@ export class SessionStore { meta?: { protocol?: string; upstreamOrigin?: string }; }>(); + /** #3923: cross-instance contention observability hook — fired on EVERY + * state commit (fresh build AND adopted disk record, both storage modes, + * see noteOwnCommit). The standalone server wires it to the served-marker + * writer via setGlobalSessionCommitListener; other hosts leave it unset + * (undefined = no-op). A throwing listener must not break the save chain. */ + private sessionCommitListener?: (sessionId: string) => void; + + setSessionCommitListener(fn?: (sessionId: string) => void): void { + this.sessionCommitListener = fn; + } + /** #3049 test seam: how many times the stale guard took the FULL * envelope read (the expensive path). The fast path collapses this to * one read per external writer, not one read per save. */ @@ -2072,6 +2083,11 @@ export class SessionStore { ? { protocol: committed.meta.protocol, upstreamOrigin: committed.meta.upstreamOrigin } : prev?.meta, }); + try { + this.sessionCommitListener?.(id); + } catch { + // observability hook — a failure must never break the save chain + } } /** #3406: stamp the envelope's post-write stat into the ownership token — @@ -2853,6 +2869,16 @@ function defaultLogger(level: string, m: string): void { /** Singleton store for the running proxy. */ let _store: SessionStore | null = null; +/** #3923: process-global commit listener — registered by the standalone + * server BEFORE the lazy singleton exists; applied at construction (and to + * any store injected later via _setStoreForTest). */ +let globalSessionCommitListener: ((sessionId: string) => void) | undefined; + +export function setGlobalSessionCommitListener(fn: ((sessionId: string) => void) | undefined): void { + globalSessionCommitListener = fn; + _store?.setSessionCommitListener(fn); +} + /** #3135 ③/#3818: stamp-resolution helper — observe the singleton store if it * exists, never construct one (see effectiveFileStorageMode). */ function peekStoreForStamp(): SessionStore | null { @@ -2862,6 +2888,7 @@ function peekStoreForStamp(): SessionStore | null { export function getStore(): SessionStore { if (!_store) { _store = new SessionStore({ enabled: persistEnabled(), log: defaultLogger }); + if (globalSessionCommitListener !== undefined) _store.setSessionCommitListener(globalSessionCommitListener); } return _store; } @@ -2869,4 +2896,5 @@ export function getStore(): SessionStore { /** Test hook: inject a store with a temp dir. */ export function _setStoreForTest(store: SessionStore): void { _store = store; + if (globalSessionCommitListener !== undefined) store.setSessionCommitListener(globalSessionCommitListener); } diff --git a/src/preflight.ts b/src/preflight.ts index 49941a9706..9b1b61b486 100644 --- a/src/preflight.ts +++ b/src/preflight.ts @@ -20,6 +20,7 @@ import { proxyDispatcher } from "./upstream-proxy.js"; import { isTlsTrustCode, tlsTrustRemediation, TLS_TRUST_CODES } from "./upstream-fail.js"; import { lastCompressSuffix, type Session } from "./session.js"; import { epochConfirmPending } from "./fold-reconcile.js"; +import { deadSkipKeys, filterDeadRanges, parseStructuralEmptyVerdict, recordDeadRange } from "./dead-ranges.js"; import { peekRegistryOutputLimit } from "./registry.js"; import { safePrefix } from "./text-safe.js"; import { applyEstimateCalibration, currentCalibrationFactor } from "./util.js"; @@ -313,6 +314,12 @@ export interface PreflightResult { * ran; calls > 0 && attempts === 0 = pool dropped every batch before * dispatch (congestion, not provider failure). */ externalDispatch?: { calls: number; attempts: number }; + /** #2638: some round saw at least one range that passed every list-level gate + * (viability, minCompressRange, dead-range memory). The caller uses it to tell a + * failed fold chain (candidates existed, none delivered — backoff is right) from + * lean steady state (nothing foldable remained at all — arming the auto-fold + * backoff would only cycle classic nudges at dead spans). */ + hadFoldableCandidates: boolean; } function refMaps(messages: CoreMessage[], state: Session["state"]): { refToIdx: Map; idxToRef: Map } { @@ -1468,7 +1475,7 @@ export async function preflightCompress(deps: PreflightDeps, messages: CoreMessa if (epochConfirmPending(deps.session)) { deps.log("debug", `[preflight] fold epoch confirmation pending for ${deps.session.id} — compression paused (verbatim passthrough) until the resent history realigns (#3641)`); const paused = estimateCoreMessages(messages) + (deps.wireOverhead ?? 0); - return { compressedRanges: 0, savedTokens: 0, payloadEstimate: paused, rangesRemaining: 0, fitsWindow: paused <= deps.config.modelContextLimit }; + return { compressedRanges: 0, savedTokens: 0, payloadEstimate: paused, rangesRemaining: 0, fitsWindow: paused <= deps.config.modelContextLimit, hadFoldableCandidates: false }; } const limit = deps.config.modelContextLimit; // #1843 dual-channel accounting: fold decisions run on the TEXT channel — @@ -1499,7 +1506,7 @@ export async function preflightCompress(deps: PreflightDeps, messages: CoreMessa // the uncalibrated #1841 path stays byte-identical. const calibrateFoldMass = (rawMass: number): number => applyEstimateCalibration(rawMass, kFactor, kOrigin, deps.upstreamOrigin); let textTarget = Math.max(0, Math.min(limit, deps.compressionTarget ?? limit) - imageReserve); - const result: PreflightResult = { compressedRanges: 0, savedTokens: 0, payloadEstimate: applyEstimateCalibration(estimateCoreMessages(messages) + wireOverhead, kFactor, kOrigin, deps.upstreamOrigin) + imageReserve, rangesRemaining: 0, fitsWindow: true }; + const result: PreflightResult = { compressedRanges: 0, savedTokens: 0, payloadEstimate: applyEstimateCalibration(estimateCoreMessages(messages) + wireOverhead, kFactor, kOrigin, deps.upstreamOrigin) + imageReserve, rangesRemaining: 0, fitsWindow: true, hadFoldableCandidates: false }; if (limit <= 0) return result; if (deps.externalSummary === undefined) { try { @@ -1544,6 +1551,11 @@ export async function preflightCompress(deps: PreflightDeps, messages: CoreMessa // skipSet keys are stable across folds because refs are content-fingerprinted, // so a range found unusable is never retried within this invocation. const skipSet = new Set(); + // #2638: seed with the session-level memory of spans whose previews returned a + // structural "no new compressible messages" verdict — without this, every request + // re-nominates and re-rejects them (the permanent rejection loop). The kernel keeps + // advertising them, so the list-level filter below and this seed share one source. + for (const key of deadSkipKeys(deps.session)) skipSet.add(key); // #1372: every silent skip leaves a trace — preflight and the plugin compress // path judge the same range at different pipeline positions, so their verdicts // can legitimately diverge; recording where+why makes the divergence diffable. @@ -1552,6 +1564,8 @@ export async function preflightCompress(deps: PreflightDeps, messages: CoreMessa if (skipReasons.length < 8 && !skipReasons.includes(reason)) skipReasons.push(safePrefix(reason, 200)); }; let subMinNoted = false; + let deadNoted = false; + let deadDroppedTotal = 0; let summaryCalls = 0; let budgetHit = false; let transientRetryBudget = TRANSIENT_EMPTY_RETRY_BUDGET; @@ -1921,16 +1935,31 @@ export async function preflightCompress(deps: PreflightDeps, messages: CoreMessa // rounds and misreports "N viable ranges tried"; with them gone the // empty-list path below can reach the #330 soft-zone relaxation. const viable = viableRanges(turn.nudge?.compressibleRanges ?? []); - const ranges = viable.filter((r) => minChars <= 0 || (r.chars ?? r.tokens * 4) >= minChars); + const aboveMin = viable.filter((r) => minChars <= 0 || (r.chars ?? r.tokens * 4) >= minChars); + // #2638: spans whose previews returned a structural "no new compressible + // messages" verdict can never fold through plain refs — the kernel keeps + // advertising them, so without this filter every round re-walks and + // re-previews them. Exact key match only: a wider range that merely + // contains a dead chunk keeps its live members. + let ranges = filterDeadRanges(deps.session, aboveMin); + const deadDropped = aboveMin.length - ranges.length; + if (deadDropped > 0) { + deadDroppedTotal += deadDropped; + if (!deadNoted) { + deadNoted = true; + deps.log("debug", `[preflight] ${deadDropped} candidate range(s) suppressed: previously-dead structural verdicts (#2638)`); + } + } // #1372: the list-level minCompressRange filter used to drop sub-minimum // ranges silently — "no compressible ranges remain" gave no hint that // ranges existed but were all under the gate. - if (ranges.length === 0 && viable.length > 0 && !subMinNoted) { + if (ranges.length === 0 && aboveMin.length === 0 && viable.length > 0 && !subMinNoted) { subMinNoted = true; deps.log("warn", `[preflight] ${viable.length} viable range(s) are below minCompressRange (${minChars} chars); none foldable`); noteSkip(`all ${viable.length} viable range(s) below minCompressRange (${minChars} chars)`); } rangesRemaining = ranges.length; + if (ranges.length > 0) result.hadFoldableCandidates = true; // Field reports of the preflight marathon cut at the #568 hold line // with zero round visibility (both 0.1.195 fork incidents): 19 LLM // rounds × tens of seconds upstream each, nothing in between. One @@ -1985,7 +2014,8 @@ export async function preflightCompress(deps: PreflightDeps, messages: CoreMessa renderTags: "text-only", contentStore: contentStoreOf(deps.session), }); - const relaxedRanges = viableRanges(probe.nudge?.compressibleRanges ?? []).filter((r) => minChars <= 0 || (r.chars ?? r.tokens * 4) >= minChars); + // #2638: structural verdicts hold under relax too (coverage-based, zone-independent) — dead mass cannot close the gap either. + const relaxedRanges = filterDeadRanges(deps.session, viableRanges(probe.nudge?.compressibleRanges ?? []).filter((r) => minChars <= 0 || (r.chars ?? r.tokens * 4) >= minChars)); const relaxedPotential = resolvableMass(relaxedRanges); if (calibrateFoldMass(relaxedPotential) * FUTILITY_SLACK >= deficit) { activeConfig = relaxedConfig(deps.config); @@ -2048,7 +2078,9 @@ export async function preflightCompress(deps: PreflightDeps, messages: CoreMessa // change the verdict. failure = { kind: "exhausted", deterministic: true, detail: relaxed ? relaxedExhaustedDetail : subMinNoted ? `no foldable compressible ranges remain: all ${viable.length} viable range(s) are below minCompressRange (${minChars} chars)` - : "no compressible ranges remain in the conversation" }; + : deadNoted + ? `no foldable compressible ranges remain: ${deadDroppedTotal} range(s) carry previously-dead structural verdicts (#2638)` + : "no compressible ranges remain in the conversation" }; break; } // #2682: two-tier dispatch order. Tier 1 = ranges entirely outside the @@ -2462,6 +2494,17 @@ export async function preflightCompress(deps: PreflightDeps, messages: CoreMessa || "the kernel created no block and reported no error"; deps.log("warn", `[preflight] preview rejected range ${skipKey}: ${verdict}`); noteSkip(`${skipKey}: preview rejected — ${verdict}`); + // #2638: ONLY the structural "no new compressible messages" verdicts are + // regime-independent — coverage-based, they hold under the #330 relax flip + // too, so the span is skipped for the rest of this invocation and + // remembered across invocations. Zone/min-gate/length verdicts can clear + // mid-invocation (relax, newer messages) and stay retryable. Raw errors, + // not the truncated verdict: the named-block list must survive intact. + const named = parseStructuralEmptyVerdict(preview.result.errors.join("\n")); + if (named !== null) { + skipSet.add(skipKey); + recordDeadRange(deps.session, startRef, endRef, named); + } continue; } // Direct raw messages render host-side: #781 image notes live in BiliMessage diff --git a/src/served-markers.ts b/src/served-markers.ts new file mode 100644 index 0000000000..5be7c45b9e --- /dev/null +++ b/src/served-markers.ts @@ -0,0 +1,235 @@ +// #3923: per-session writer markers — session-level split-brain visibility. +// Two live bili processes sharing one sessions directory are a KNOWN topology: +// the #394 registration warning and the #2401 slow-tick peer warning both fire +// at MACHINE level, once each — but the actual damage is per-SESSION (both +// instances committing state for the SAME conversation; the #405 counter guard +// stops monotonic rollback only, divergent state with equal-or-greater counters +// still clobbers last-writer-wins). That damage was previously discoverable +// only by manually diffing [v=…] tags in the shared log. Each standalone +// instance therefore records which sessions it recently COMMITTED to a tiny +// marker file (/writers/.json, throttled atomic writes) and +// names both sides in an error line when it commits a session another live +// instance committed within the window. Pure observability: no routing, +// locking, or serving behavior changes. + +import fs from "node:fs"; +import path from "node:path"; +import { isPidAlive } from "./instance.js"; + +export interface WriterIdentity { + instanceId: string; + pid: number; + origin: string; + version: string; +} + +interface WriterMarkerFile { + v: 1; + instanceId: string; + pid: number; + origin: string; + version: string; + /** sessionId → last commit time (ms epoch). */ + servedAt: Record; +} + +/** A peer that committed the same session within this window counts as + * concurrently active on it. */ +const CONTEST_WINDOW_MS = 10 * 60_000; +/** Minimum gap between marker-file writes (per-turn commits coalesce). */ +const FLUSH_MIN_INTERVAL_MS = 5_000; +/** Drop per-session entries older than this (bounds the map). */ +const ENTRY_PRUNE_MS = 60 * 60_000; +/** Hard cap on tracked sessions (oldest evicted first). */ +const MAX_TRACKED_SESSIONS = 128; +/** Re-evaluate (and re-log) a contested verdict at most this often per session. */ +const VERDICT_TTL_MS = 60_000; +/** A peer counts as live when its pid is alive OR its marker was written + * within this window (covers platforms where the pid check cannot see the + * peer process). */ +const PEER_FILE_FRESH_MS = 30_000; +/** reapStaleWriterMarkers removes files whose pid is dead AND whose mtime is + * older than this. */ +const STALE_REAP_MS = 60 * 60_000; + +export function writerMarkerDir(stateDirPath: string): string { + return path.join(stateDirPath, "writers"); +} + +let tmpSeq = 0; + +function durableAtomicWrite(file: string, text: string): void { + const tmp = path.join(path.dirname(file), `.tmp-${path.basename(file)}-${process.pid}-${tmpSeq++}`); + try { + const fd = fs.openSync(tmp, "w"); + try { + fs.writeSync(fd, text); + fs.fsyncSync(fd); + } finally { + fs.closeSync(fd); + } + fs.renameSync(tmp, file); + } catch (err) { + try { fs.rmSync(tmp, { force: true }); } catch { /* best-effort: a stuck temp is swept at next boot */ } + throw err; + } +} + +export interface ServedMarkerWriter { + /** Record that THIS instance just committed state for a session. Cheap on + * the hot path: an in-memory set plus a throttled file flush; the + * contention verdict is evaluated at most once per VERDICT_TTL_MS per + * session. Never throws into the save chain. */ + noteServed(sessionId: string): void; + /** Final flush + timer teardown (graceful shutdown). */ + dispose(): void; +} + +export function createServedMarkerWriter(identity: WriterIdentity, dir: string, log: (level: string, msg: string) => void): ServedMarkerWriter & { __flushForTest(): void; __servedForTest(): Record } { + const servedAt = new Map(); + const verdicts = new Map(); + let flushTimer: ReturnType | undefined; + let lastFlushAt = 0; + let disposed = false; + + const prune = (now: number): void => { + for (const [id, at] of [...servedAt]) { + if (now - at > ENTRY_PRUNE_MS) servedAt.delete(id); + } + while (servedAt.size > MAX_TRACKED_SESSIONS) { + let oldest: string | undefined; + let oldestAt = Number.MAX_SAFE_INTEGER; + for (const [id, at] of servedAt) if (at < oldestAt) { oldest = id; oldestAt = at; } + if (oldest === undefined) break; + servedAt.delete(oldest); + } + }; + + const writeMarker = (): void => { + const now = Date.now(); + prune(now); + const body: WriterMarkerFile = { + v: 1, + instanceId: identity.instanceId, + pid: identity.pid, + origin: identity.origin, + version: identity.version, + servedAt: Object.fromEntries(servedAt), + }; + try { + fs.mkdirSync(dir, { recursive: true }); + durableAtomicWrite(path.join(dir, `${identity.instanceId}.json`), JSON.stringify(body)); + } catch { + // best-effort observability — a failed marker write must never + // surface through the save chain + } + }; + + const flush = (): void => { + flushTimer = undefined; + if (disposed) return; + lastFlushAt = Date.now(); + writeMarker(); + }; + + const scheduleFlush = (): void => { + if (flushTimer !== undefined || disposed) return; + const wait = Math.max(0, FLUSH_MIN_INTERVAL_MS - (Date.now() - lastFlushAt)); + flushTimer = setTimeout(flush, wait); + flushTimer.unref?.(); + }; + + const checkContested = (sessionId: string, now: number): void => { + const last = verdicts.get(sessionId); + if (last !== undefined && now - last < VERDICT_TTL_MS) return; + verdicts.set(sessionId, now); + for (const peer of findContestedWriters(dir, identity.instanceId, sessionId, now)) { + log("error", `[split-brain] session ${sessionId} committed by BOTH this instance (pid ${identity.pid}, ${identity.origin}, v${identity.version}) and another live instance (pid ${peer.pid}, ${peer.origin}, v${peer.version}) within ${CONTEST_WINDOW_MS / 60000}m — fold decisions are invisible across instances and the last writer wins (#394/#405); stop one of them or align your client config so one conversation rides one proxy (#3923)`); + } + }; + + return { + noteServed(sessionId: string): void { + if (disposed) return; + const now = Date.now(); + servedAt.set(sessionId, now); + scheduleFlush(); + checkContested(sessionId, now); + }, + dispose(): void { + disposed = true; + if (flushTimer !== undefined) clearTimeout(flushTimer); + // final write (bypasses the disposed guard): peers should still be + // able to attribute this instance's last commits after shutdown + writeMarker(); + }, + __flushForTest(): void { + flush(); + }, + __servedForTest(): Record { + return Object.fromEntries(servedAt); + }, + }; +} + +/** Peer marker files that record a commit of `sessionId` inside the contest + * window by a LIVE foreign instance (pid alive or file fresh). Malformed or + * tearing files are skipped silently — a marker is observability, not data. */ +export function findContestedWriters(dir: string, selfInstanceId: string, sessionId: string, now: number): Array { + let entries: fs.Dirent[]; + try { + entries = fs.readdirSync(dir, { withFileTypes: true }); + } catch { + return []; + } + const out: Array = []; + for (const entry of entries) { + if (!entry.isFile() || !entry.name.endsWith(".json")) continue; + const file = path.join(dir, entry.name); + try { + const raw = JSON.parse(fs.readFileSync(file, "utf8")) as Partial; + if (raw.v !== 1 || typeof raw.instanceId !== "string" || raw.instanceId === selfInstanceId) continue; + const at = raw.servedAt !== null && typeof raw.servedAt === "object" ? (raw.servedAt as Record)[sessionId] : undefined; + if (typeof at !== "number" || now - at > CONTEST_WINDOW_MS) continue; + const mtimeMs = fs.statSync(file).mtimeMs; + const alive = typeof raw.pid === "number" ? isPidAlive(raw.pid) : false; + if (!alive && now - mtimeMs > PEER_FILE_FRESH_MS) continue; + out.push({ + instanceId: raw.instanceId, + pid: typeof raw.pid === "number" ? raw.pid : -1, + origin: typeof raw.origin === "string" ? raw.origin : "unknown", + version: typeof raw.version === "string" ? raw.version : "unknown", + servedAt: at, + }); + } catch { + // malformed/tearing marker — skip + } + } + return out; +} + +/** Startup hygiene: drop marker files whose instance is provably gone (pid + * dead) and stale beyond the reap horizon. Fresh files from a just-crashed + * peer stay — findContestedWriters' freshness test still honors them for + * PEER_FILE_FRESH_MS. */ +export function reapStaleWriterMarkers(dir: string, now: number): void { + let entries: fs.Dirent[]; + try { + entries = fs.readdirSync(dir, { withFileTypes: true }); + } catch { + return; + } + for (const entry of entries) { + if (!entry.isFile() || !entry.name.endsWith(".json")) continue; + const file = path.join(dir, entry.name); + try { + const raw = JSON.parse(fs.readFileSync(file, "utf8")) as Partial; + const dead = typeof raw.pid !== "number" || !isPidAlive(raw.pid); + if (!dead) continue; + if (now - fs.statSync(file).mtimeMs <= STALE_REAP_MS) continue; + fs.rmSync(file, { force: true }); + } catch { + // unreadable file — leave it; a later boot retries + } + } +} diff --git a/src/server.ts b/src/server.ts index 7e0daeff80..f960d90726 100644 --- a/src/server.ts +++ b/src/server.ts @@ -94,11 +94,12 @@ import { renderUI, handleConfigGet, handleConfigPut, handleSummaryCredentialPut, import { reapOrphanBlocks } from "./orphan-gc.js"; import { conflictScanEnabled, isDesignBenign, scanClientPlugins, sniffScanClient } from "./thirdparty-scan.js"; import { clearConflictEvents, recordConflict, summarizeConflicts } from "./conflict-watch.js"; -import { getStore } from "./persist.js"; +import { getStore, setGlobalSessionCommitListener } from "./persist.js"; import { log as loggerLog, configureLogger, getLogPath, closeLogger, isStreamWriteError, isBenignSocketRaceError, enterSessionContext, muteStderrLeg, onFileLegLive } from "./logger.js"; import { queryLogLines } from "./web/logs-query.js"; import { configFile, defaultLogFile, stateDir } from "./paths.js"; import { atomicWriteInstanceFile, clearProxyInstanceFile, classifySameLanePeer, entryScriptFingerprint, entryShape, findSameLanePredecessor, isPidAlive, listInstances, readProxyInstanceFile, registerInstanceAndWarn, sameInstanceShape, unregisterInstance, warnOnNewPeers, writeSpawnPortLog, type InstanceShape, type ProxyInstanceFile, type SameLanePeer } from "./instance.js"; +import { createServedMarkerWriter, reapStaleWriterMarkers, writerMarkerDir, type ServedMarkerWriter } from "./served-markers.js"; import { mayClobberPointer, probeOwnerInfo, resolvePortClaim } from "./port-owner.js"; import { hoistTrappedToolItems } from "./tool-pair-order.js"; import { computeAnthropicMessageMarks, stampAnthropicSystemCacheControl, anthropicToolsCarryCacheControl } from "./loop/cache-control.js"; @@ -1062,6 +1063,9 @@ export async function startServer(opts: ProxyOptions): Promise { const PREDECESSOR_WAIT_MS = 500; let predecessorTicksLeft = PREDECESSOR_WAIT_TICKS; let predecessorWaitAnnounced = false; + // #3923: per-session writer marker (session-level split-brain visibility) — + // wired in announceListening, disposed on graceful shutdown. + let servedWriter: ServedMarkerWriter | undefined; const sameLanePredecessorHolds = (port: number): boolean => { const pred = findSameLanePredecessor(listInstances(), port, launcherLane, ownFingerprint); if (pred && !predecessorWaitAnnounced) { @@ -1152,6 +1156,21 @@ export async function startServer(opts: ProxyOptions): Promise { }, peerRescanMs); peerTimer.unref?.(); } + // #3923: per-session writer marker — the machine-level warnings above + // stay as-is; this adds SESSION-level visibility: when another live + // instance commits state for the same conversation, both sides are + // named in a [split-brain] error line instead of requiring a manual + // [v=…] diff of the shared log. Pure observability (served-markers.ts). + { + const markerDir = writerMarkerDir(stateDir()); + servedWriter = createServedMarkerWriter( + { instanceId, pid: process.pid, origin, version: VERSION }, + markerDir, + (level, msg) => log(level, msg), + ); + reapStaleWriterMarkers(markerDir, Date.now()); + setGlobalSessionCommitListener((id) => servedWriter?.noteServed(id)); + } const nOverrides = Object.keys(opts.routes).length; // #2864: stamp pid + instance id into the banner — 24 identical // indistinguishable banners in 13s (short-lived instances rebinding @@ -1405,6 +1424,7 @@ export async function startServer(opts: ProxyOptions): Promise { const finishShutdown = (): void => { flushPrefixAffinity(); flushOrderSkeletons(); + servedWriter?.dispose(); clearProxyInstanceFile(instanceId); unregisterInstance(instanceId); closeLogger(); @@ -2687,6 +2707,64 @@ function preflightDeadEndCooldownMs(): number { return knobPreflightDeadEndCooldownMs(); } +// #3846: a local 413 ("can never fit, NOT forwarded") is independent evidence +// that this origin/model/window cannot accept the payload — but it only ever +// lived in the #726 dead-end marker's body-hash dimension: the moment the +// baseline source demotes to estimate (the #2313 probe re-arms at +// baselineFloor === 0) or the body changes between turns (the hash misses), +// the refusal evaporates and every forward-once escape hatch (estimate-only +// probe / image arbitration / #1839 rebuilt-view divergence / #3146 +// fresh-session) re-bets the same estimator guess — production shape: three +// consecutive full-bill forwards of 606K/620K/633K-token payloads bili itself +// had already refused locally. Persist refusals as their own metadata channel +// — NEVER as usage (that field is upstream billing evidence only) — scoped to +// the route/model/window that saw them; an equal-or-wider refused window +// covers the current equal-or-tighter one, anything else evaluates fresh. +interface LocalRefusalRecord { + origin: string; + model: string; + window: number; + at: number; +} + +const LOCAL_REFUSAL_MAX_RECORDS = 16; + +function readLocalRefusals(metadata: Record): LocalRefusalRecord[] { + const raw = metadata.localRefusals; + if (!Array.isArray(raw)) return []; + return raw.filter((r): r is LocalRefusalRecord => + !!r && typeof r === "object" + && typeof (r as LocalRefusalRecord).origin === "string" + && typeof (r as LocalRefusalRecord).model === "string" + && typeof (r as LocalRefusalRecord).window === "number", + ); +} + +/** Record a local 413 refusal for (origin, model, window). A wider refusal in + * the same scope already dominates (it covers every tighter window); a + * narrower one replaces it so the record tracks the widest verdict seen. */ +function noteLocalRefusal(session: Session, origin: string | undefined, model: string | undefined, window: number): void { + if (!origin || !model || !Number.isFinite(window) || window <= 0) return; + const records = readLocalRefusals(session.metadata); + const existing = records.find((r) => r.origin === origin && r.model === model); + if (existing !== undefined && existing.window >= window) return; + session.metadata.localRefusals = [ + ...records.filter((r) => !(r.origin === origin && r.model === model)), + { origin, model, window: Math.floor(window), at: Date.now() }, + ].slice(-LOCAL_REFUSAL_MAX_RECORDS); + markDirty(session); +} + +/** True when this session was locally refused (413) by the SAME origin+model + * at an equal or WIDER window — the current request may then not reopen any + * forward-once escape hatch (#3846). Both sides must be known + * (evidence-permitlist discipline: an unprovenanced refusal proves nothing + * about an unprovenanced request, and vice versa). */ +function localRefusalCovers(metadata: Record, origin: string | undefined, model: string | undefined, window: number): boolean { + if (!origin || !model) return false; + return readLocalRefusals(metadata).some((r) => r.origin === origin && r.model === model && r.window >= window); +} + /** #568: commit the response early so a long preflight cannot lose the client * to its header timeout. Streaming clients get an SSE stream with keep-alive * comment lines (`: bili-preflight` — a spec-mandated no-op for every SSE @@ -3165,6 +3243,13 @@ export async function preflightCompressIfNeeded( log("debug", `[${session.id}] [usage-vs-est] upstream-billed=${billed} est=${Math.round(textEstimate)} k̂=${kFactor !== undefined ? kFactor.toFixed(2) : "n/a"} route=${currentOrigin === undefined ? "?" : maskUrlForLog(currentOrigin)} model=${model ?? "?"} baseline-grade=${session.stats.lastInputTokensSource ?? "none"} (#2283)`); } if (limit <= 0 || !model || !triggerFires) return prepared; + // #3846: an earlier local 413 for THIS origin+model at an equal or wider + // window is independent evidence the payload cannot fit here — the + // forward-once escapes below must not re-bet the estimator guess. Fitting + // payloads are unaffected (they never reach an escape hatch); another + // origin/model or a larger window judges fresh, exactly like the #1933 F2 + // baseline demotion does. + const localRefusalHere = localRefusalCovers(session.metadata, currentOrigin, model, limit); // #2313: an estimate may not block the forward. The trigger above can // fire on the calibrated chars/4 estimate alone; when the current // upstream's billing scale diverges from the estimator's caliber @@ -3223,10 +3308,17 @@ export async function preflightCompressIfNeeded( // convention and is terminal for every mainstream host. Same doom // math, same message, same no-forward behavior — status only. log("error", `[${session.id}] preflight fail-fast 413 (retryable=false): ${doomedMessage}`); + noteLocalRefusal(session, currentOrigin, model, limit); return { failFast: true, status: 413, message: doomedMessage, retryable: false, respond: !res.writableEnded }; } - log("info", `[${session.id}] preflight trigger fired on estimate only (~${Math.round(textChannel)} text + ~${imageTokens} image vs window ${limit}); baseline ${session.stats.lastInputTokens} (${session.stats.lastInputTokensSource ?? "unprovenanced"}) carries no current-route upstream evidence — forwarding once to acquire usage/overflow evidence before compressing (#2313)`); - return prepared; + if (localRefusalHere) { + // #3846: bili already refused this scope locally — the estimate-only + // bet is spent. Fall through to fold/fail-fast instead of re-forwarding. + log("warn", `[${session.id}] preflight trigger fired on estimate only (~${Math.round(textChannel)} text + ~${imageTokens} image vs window ${limit}) but this origin/model was locally refused (413) at an equal or wider window — NOT re-forwarding for evidence; falling through to fold/fail-fast (#3846)`); + } else { + log("info", `[${session.id}] preflight trigger fired on estimate only (~${Math.round(textChannel)} text + ~${imageTokens} image vs window ${limit}); baseline ${session.stats.lastInputTokens} (${session.stats.lastInputTokensSource ?? "unprovenanced"}) carries no current-route upstream evidence — forwarding once to acquire usage/overflow evidence before compressing (#2313)`); + return prepared; + } } // #3146: judge fit on the calibrated CJK-aware estimate for EVERY regime — // the char-count bound (tokenCount) is the TRIGGER arm only. Judging fit on @@ -3264,7 +3356,8 @@ export async function preflightCompressIfNeeded( // arbitration and let preflightCompress fold the text portion first, re-applying // this same forward-instead-of-fail-fast decision after compression (below). const imageArbitration = imageTokens > 0 && textChannel < limit && textChannel + imageTokens >= limit && noOverflowEvidence; - if (imageArbitration && (prepared.nudge?.compressibleRanges ?? []).length === 0) { + // #3846: a scoped local 413 outranks the image-arbitration bet too. + if (imageArbitration && !localRefusalHere && (prepared.nudge?.compressibleRanges ?? []).length === 0) { log("warn", `[${session.id}] image-dominated payload (~${textEstimate} text + ~${imageTokens} image tokens) exceeds window ${limit} by estimate only, nothing compressible, no upstream overflow evidence — forwarding for the upstream to arbitrate billing (#496/#1800)`); return prepared; } @@ -3533,10 +3626,19 @@ export async function preflightCompressIfNeeded( if (growthArmed) { if (queueDroppedAll) { log("warn", `[${session.id}] auto-fold made no progress (0 range(s) folded) — no summary attempt was ever dispatched (shared pool saturated / deadline expired before dispatch); congestion, not a chain failure — NOT backing off, auto-fold retries next turn (#2662); forwarding as-is (estimate fits the model window ${limit})`); - } else { + } else if (result.hadFoldableCandidates) { log("warn", `[${session.id}] auto-fold made no progress (0 range(s) folded) — forwarding as-is (estimate fits the model window ${limit}); backing off auto-fold for ${Math.round(AUTO_FOLD_BACKOFF_MS / 60_000)}m so classic nudges resume`); armAutoFoldBackoff(session); markDirty(session); + } else { + // #2638: lean steady state — no candidate ever passed the list gates + // (advertised spans carry remembered structural dead verdicts / are + // sub-minimum), so zero progress is EXPECTED and a dead chain is not + // the cause. Arming the cooldown here cycled classic nudges at the + // same dead spans every AUTO_FOLD_BACKOFF_MS (the issue's nudge + // churn); forward quietly — the next growth turn re-evaluates from + // live state. + log("info", `[${session.id}] auto-fold steady state: payload fits and no foldable candidates remained — forwarding as-is without backoff (#2638)`); } } else { log("warn", `[${session.id}] preflight made no progress but the payload fits; forwarding as-is`); @@ -3553,7 +3655,7 @@ export async function preflightCompressIfNeeded( // and armOverflowShrink recovers with real evidence; a fitting // payload is no longer refused on a stale measurement. One forward // only — the #330 relaxed-zone caveat still bounds the risk. - if (result.compressedRanges > 0 && !unknownBaseline && result.payloadEstimate < limit) { + if (result.compressedRanges > 0 && !unknownBaseline && result.payloadEstimate < limit && !localRefusalHere) { log("warn", `[${session.id}] preflight view fits (~${result.payloadEstimate}/${limit}) but the rebuilt payload measures over — forwarding once for upstream arbitration (#1839)`); return outbound; } @@ -3569,10 +3671,15 @@ export async function preflightCompressIfNeeded( if (growthArmed && payloadFitsWindow) { if (queueDroppedAll) { log("warn", `[${session.id}] auto-fold did not reach the growth target ${compressionTarget} (${result.compressedRanges} range(s) folded) — no summary attempt was ever dispatched (shared pool saturated / deadline expired before dispatch); congestion, not a chain failure — NOT backing off, auto-fold retries next turn (#2662); forwarding anyway (payload fits the model window ${limit})`); - } else { + } else if (result.hadFoldableCandidates) { log("warn", `[${session.id}] auto-fold did not reach the growth target ${compressionTarget} (${result.compressedRanges} range(s) folded) — forwarding anyway (payload fits the model window ${limit}); backing off auto-fold for ${Math.round(AUTO_FOLD_BACKOFF_MS / 60_000)}m so classic nudges resume`); armAutoFoldBackoff(session); markDirty(session); + } else { + // #2638: same lean-steady-state distinction as the fits branch above — no + // foldable candidate existed, so the unreachable target is a property of the + // session state, not of the chain; do not cycle nudges at dead spans. + log("info", `[${session.id}] auto-fold steady state: growth target ${compressionTarget} unreachable while nothing foldable remained (${result.compressedRanges} range(s) folded) — forwarding anyway (payload fits the model window ${limit}) without backoff (#2638)`); } return outbound; } @@ -3581,7 +3688,7 @@ export async function preflightCompressIfNeeded( // upstream overflow evidence. Forward for the upstream to arbitrate billing // instead of fail-fasting a payload whose real bill likely fits (#496). The text // portion was already folded above when foldable; we do NOT re-loop. - if (imageArbitration) { + if (imageArbitration && !localRefusalHere) { // Same fallback as the fit-gate measurement above: processedMessages // empty ⇒ kernel transform failure ⇒ the raw body rides. const outMsgs = outbound.processedMessages.length > 0 ? outbound.processedMessages : outbound.originalMessages; @@ -3609,10 +3716,17 @@ export async function preflightCompressIfNeeded( const byteFloorTokens = Math.ceil(doomBytes / DOOMED_PROBE_BYTES_PER_TOKEN); const doomedMessage = `payload core text alone is ~${doomBytes} bytes even after preflight compression — at the densest tokenizer density observed on a live route (~${DOOMED_PROBE_BYTES_PER_TOKEN} bytes/token, #2122) that is ≥ ~${byteFloorTokens} tokens vs the model window ${limit} (model=${model}); no upstream can accept it. Remove or trim the oversized content (e.g. a giant paste or tool result) and retry.`; log("error", `[${session.id}] preflight fail-fast 413 (retryable=false): ${doomedMessage}`); + noteLocalRefusal(session, currentOrigin, model, limit); return { failFast: true, status: 413, message: doomedMessage, retryable: false, respond: !res.writableEnded }; } - log("info", `[${session.id}] fresh-session payload still over window ${limit} by local estimate (~${Math.round(result.payloadEstimate)} tok, ${result.compressedRanges} range(s) folded) with no upstream evidence — forwarding once for the upstream to arbitrate size before failing fast (#3146)`); - return outbound; + if (localRefusalHere) { + // #3846: bili already refused this scope locally — do not re-bet on + // the estimator; fall through to the fail-fast below. + log("warn", `[${session.id}] fresh-session payload still over window ${limit} by local estimate (~${Math.round(result.payloadEstimate)} tok, ${result.compressedRanges} range(s) folded) but this origin/model was locally refused (413) at an equal or wider window — NOT re-forwarding for size arbitration (#3846)`); + } else { + log("info", `[${session.id}] fresh-session payload still over window ${limit} by local estimate (~${Math.round(result.payloadEstimate)} tok, ${result.compressedRanges} range(s) folded) with no upstream evidence — forwarding once for the upstream to arbitrate size before failing fast (#3146)`); + return outbound; + } } // The payload still overflows the window: fail fast with a diagnostic // error instead of forwarding a guaranteed-400 payload (#301). @@ -3633,11 +3747,27 @@ export async function preflightCompressIfNeeded( // lanes keep their 502 fail-fast contract where NOT re-forwarding IS the // protection, not a client spin. const status = f?.kind === "upstream" && f.status === 429 ? 503 : f?.kind === "exhausted" && f.deterministic && !payloadFitsWindow ? 413 : 502; + // #3918: non-deterministic exhaustion WITH committed progress is not terminal — + // the folds persist in the session, so the next attempt starts from a smaller + // baseline and continues the fold. Reporting retryable=false told hosts to stop + // spinning exactly where each retry makes real progress (incident: 64 effective + // folds, every round declared terminal). Deterministic exhaustion keeps today's + // verdict: nothing more can fold from this view, so retries are genuinely doomed. + const progressContinues = f?.kind === "exhausted" && !f.deterministic && result.compressedRanges > 0; + // #3846: a deterministic can-never-fit verdict is local refusal evidence for + // this scope — persist it so the forward-once escapes above stay closed on + // retry even after the baseline demotes to estimate or the body changes. + if (status === 413) noteLocalRefusal(session, currentOrigin, model, limit); // #2189: an explicit retryable=false from preflight (credential-shape // rejection — deterministic for the identical payload) overrides the // status-based heuristic below; every other path keeps today's mapping. - const retryable = f?.retryable === false ? false : f?.retryable === true || (f?.kind === "upstream" && f.status !== undefined && (f.status === 429 || f.status >= 500)); - const ff = failFast(status, f?.detail ?? "the payload still exceeds the window after preflight compression", retryable, result.compressedRanges > 0 ? foldedCalibratedSize : undefined, result.rangesRemaining); + const retryable = progressContinues ? true : f?.retryable === false ? false : f?.retryable === true || (f?.kind === "upstream" && f.status !== undefined && (f.status === 429 || f.status >= 500)); + let ffDetail = f?.detail ?? "the payload still exceeds the window after preflight compression"; + if (progressContinues) { + const shortfall = foldedCalibratedSize !== undefined ? Math.max(0, limit - foldedCalibratedSize) : undefined; + ffDetail += ` ${result.compressedRanges} range(s) folded this turn persist in the session — the next attempt starts from a smaller baseline and continues the fold${shortfall !== undefined ? ` (~${shortfall.toLocaleString("en-US")} tokens short of the target)` : ""}.`; + } + const ff = failFast(status, ffDetail, retryable, result.compressedRanges > 0 ? foldedCalibratedSize : undefined, result.rangesRemaining); const contentDeadEnd = f?.kind === "exhausted" || (f?.kind === "upstream" && f.status !== undefined && f.status >= 400 && f.status < 500 && !retryable); if (contentDeadEnd && result.compressedRanges === 0) { const cooldownMs = preflightDeadEndCooldownMs(); @@ -3806,5 +3936,5 @@ function logMsg(opts: ProxyOptions, level: string, msg: string): void { export { getUnrecognizedPathStats, logDumpFailure, logUnrecognizedPath } from "./server/observability.js"; export { BILI_HOP_HEADER, parseLauncherModelWindows, anthropicBetaContextWindow, capRegistryWindowByStandard, expandedContextSuffixWindow } from "./server/context-window.js"; -export { BILI_TOOL_NAMES, isSideRequest, outputBudgetField, restoreOutputBudget, sideRequestGuard, stripLeakedBiliTools, _resetNoOutputCeilingWarningsForTest, OUTPUT_RESTORE_ABSOLUTE_FLOOR, type OutputBudgetField } from "./server/side-request.js"; +export { BILI_TOOL_NAMES, isSideRequest, outputBudgetField, restoreOutputBudget, sideRequestGuard, stripLeakedBiliTools, _resetNoOutputCeilingWarningsForTest, _resetDshMidBandWarningsForTest, OUTPUT_RESTORE_ABSOLUTE_FLOOR, type OutputBudgetField } from "./server/side-request.js"; export { countSystemAndToolsTokens, estimateInputTokens, estimateWireOverhead, clampOutputBudget, emergencyNudge, projectThinkingMass, type ThinkingMassInput } from "./server/budget.js"; diff --git a/src/server/handle.ts b/src/server/handle.ts index f07dbc420a..7f19954fe3 100644 --- a/src/server/handle.ts +++ b/src/server/handle.ts @@ -1821,7 +1821,13 @@ async function handleRequest( // (title requests arrive FIRST, at session start). Starved all-bili requests // never reach here demoted: stripLeakedBiliTools vetoes them per #546. if (!demotedSide && !serverToolUtility) { - restoreOutputBudget(parsed, session, log, resolveKnownOutputCeiling(req.headers, parsed as Record, opts.routes, route?.rewrittenUrl, opts.sessionHeader, requestModel)); + // #3438: the dsh host contract (pi-ai buildBaseOptions) stamps the model-declared + // maxTokens on every main turn, so on this lane a below-ceiling tool-carrying + // budget is a raw-history collapse, not intent — arm the mid-band rescue. + // The plugin-stamped lane is the live path; the clientHint arm is defensive — + // it matches only if some dsh build's wire identity is literally "dsh". + const dshRestoreLane = pluginAgent === "dsh" || session.metadata.clientHint === "dsh"; + restoreOutputBudget(parsed, session, log, resolveKnownOutputCeiling(req.headers, parsed as Record, opts.routes, route?.rewrittenUrl, opts.sessionHeader, requestModel), dshRestoreLane); } // #896: the per-scope output-headroom cap (compress.outputHeadroomMaxPct, // three-level merge; default 0.25, aligned with billion-context-pi). diff --git a/src/server/prepare-anthropic.ts b/src/server/prepare-anthropic.ts index 7703ef0315..aafb3fb290 100644 --- a/src/server/prepare-anthropic.ts +++ b/src/server/prepare-anthropic.ts @@ -21,6 +21,7 @@ import { applyAbsorbView, storeEffectiveAbsorb } from "../absorb.js"; import { adoptContentStore, ccrEnabled, ccrLoopConfig, contentStoreOf, dropRetrievals, imageRescaleToolArmed, pruneExpiredRetrievals, reconcileReloadedRetrievals, renderRetrievalNotes, retrieveToolName, snapshotPendingRetrievals, snapshotRetrievalNotes } from "../store.js"; import { rulesEnabled, storeEffectiveRules } from "../rules-feature.js"; import { autoFoldEngaged, externalSummaryEnabled, growthFoldingArmed } from "../external-summary-surface.js"; +import { filterDeadRanges } from "../dead-ranges.js"; import { attachSubagentSessions } from "../subagent-sessions.js"; import { estimateCoreMessages, estimateCoreMessagesUpper, extractBillingAttributionBlock } from "../preflight.js"; import { dshNativeCompactionWitness, recordConflict } from "../conflict-watch.js"; @@ -320,7 +321,10 @@ export async function prepareAnthropic( // every downstream consumer (decision prompt + directive, prose nudge, // emergency override, preflight length gates) sees only spans the // submit gate can accept — no surface advertises a doomed span. - if (turn.nudge) turn.nudge.compressibleRanges = excludeDeadRefRanges(viableRanges(turn.nudge.compressibleRanges), session.state.deadRefs); + // #2638: also drop remembered structurally-dead spans (kernel preview + // returned "no new compressible messages") — a nudge advertising them + // steers the model into an atomically-refused compress call. + if (turn.nudge) turn.nudge.compressibleRanges = filterDeadRanges(session, excludeDeadRefRanges(viableRanges(turn.nudge.compressibleRanges), session.state.deadRefs)); nudge = turn.nudge; session.stats.contextTokens = tokenCount; session.stats.contextTokensSource = tokenCountSource; diff --git a/src/server/prepare-google.ts b/src/server/prepare-google.ts index cde34d9915..0cd9ed03c9 100644 --- a/src/server/prepare-google.ts +++ b/src/server/prepare-google.ts @@ -20,6 +20,7 @@ import { absorbToolName, applyAbsorbView, storeEffectiveAbsorb } from "../absorb import { adoptContentStore, ccrEnabled, ccrLoopConfig, contentStoreOf, imageRescaleToolArmed, retrieveToolName } from "../store.js"; import { rulesEnabled, storeEffectiveRules } from "../rules-feature.js"; import { autoFoldEngaged, externalSummaryEnabled, growthFoldingArmed } from "../external-summary-surface.js"; +import { filterDeadRanges } from "../dead-ranges.js"; import { attachSubagentSessions } from "../subagent-sessions.js"; import { reconcileSystemAnchor } from "../system-anchor.js"; import { logRawStripCounts, markerArtifactCount, stripRawStage } from "../marker/engine.js"; @@ -194,14 +195,14 @@ export async function prepareGoogle( storeEffectiveRules(session, config); turn.messages = applyAbsorbView(turn.messages, session.state, loopConfig, tokenCount); turn.messages = attachSubagentSessions(turn.messages, session); - // Drop sub-viability fragments before any consumer sees them: a tiny - // range in the list makes batched compress attempts fail atomically - // (kernel validates the whole batch). Mirrors billion-context-pi. // #3204: also drop DEAD-ref-bracketing spans at this single point so // every downstream consumer (decision prompt + directive, prose nudge, // emergency override, preflight length gates) sees only spans the // submit gate can accept — no surface advertises a doomed span. - if (turn.nudge) turn.nudge.compressibleRanges = excludeDeadRefRanges(viableRanges(turn.nudge.compressibleRanges), session.state.deadRefs); + // #2638: also drop remembered structurally-dead spans (kernel preview + // returned "no new compressible messages") — a nudge advertising them + // steers the model into an atomically-refused compress call. + if (turn.nudge) turn.nudge.compressibleRanges = filterDeadRanges(session, excludeDeadRefRanges(viableRanges(turn.nudge.compressibleRanges), session.state.deadRefs)); nudge = turn.nudge; session.stats.contextTokens = tokenCount; session.stats.contextTokensSource = tokenCountSource; diff --git a/src/server/prepare-openai.ts b/src/server/prepare-openai.ts index 4ffe373488..6b825e22e9 100644 --- a/src/server/prepare-openai.ts +++ b/src/server/prepare-openai.ts @@ -21,6 +21,7 @@ import { absorbToolName, applyAbsorbView, storeEffectiveAbsorb } from "../absorb import { adoptContentStore, ccrEnabled, ccrLoopConfig, contentStoreOf, dropRetrievals, imageRescaleToolArmed, pruneExpiredRetrievals, reconcileReloadedRetrievals, renderRetrievalNotes, retrieveToolName, snapshotPendingRetrievals, snapshotRetrievalNotes } from "../store.js"; import { rulesEnabled, storeEffectiveRules } from "../rules-feature.js"; import { autoFoldEngaged, externalSummaryEnabled, growthFoldingArmed } from "../external-summary-surface.js"; +import { filterDeadRanges } from "../dead-ranges.js"; import { attachSubagentSessions } from "../subagent-sessions.js"; import { estimateCoreMessages, estimateCoreMessagesUpper } from "../preflight.js"; import { dshNativeCompactionWitness, recordConflict } from "../conflict-watch.js"; @@ -213,7 +214,10 @@ export async function prepareOpenai( // every downstream consumer (decision prompt + directive, prose nudge, // emergency override, preflight length gates) sees only spans the // submit gate can accept — no surface advertises a doomed span. - if (turn.nudge) turn.nudge.compressibleRanges = excludeDeadRefRanges(viableRanges(turn.nudge.compressibleRanges), session.state.deadRefs); + // #2638: also drop remembered structurally-dead spans (kernel preview + // returned "no new compressible messages") — a nudge advertising them + // steers the model into an atomically-refused compress call. + if (turn.nudge) turn.nudge.compressibleRanges = filterDeadRanges(session, excludeDeadRefRanges(viableRanges(turn.nudge.compressibleRanges), session.state.deadRefs)); nudge = turn.nudge; session.stats.contextTokens = tokenCount; session.stats.contextTokensSource = tokenCountSource; diff --git a/src/server/prepare-responses.ts b/src/server/prepare-responses.ts index 1cdaa83818..1bbd183c6e 100644 --- a/src/server/prepare-responses.ts +++ b/src/server/prepare-responses.ts @@ -24,6 +24,7 @@ import { absorbToolName, applyAbsorbView, storeEffectiveAbsorb } from "../absorb import { adoptContentStore, ccrEnabled, ccrLoopConfig, contentStoreOf, imageRescaleToolArmed, retrieveToolName } from "../store.js"; import { rulesEnabled, storeEffectiveRules } from "../rules-feature.js"; import { autoFoldEngaged, externalSummaryEnabled, growthFoldingArmed } from "../external-summary-surface.js"; +import { filterDeadRanges } from "../dead-ranges.js"; import { attachSubagentSessions } from "../subagent-sessions.js"; import { estimateCoreMessages, estimateCoreMessagesUpper } from "../preflight.js"; import { hoistTrappedToolItems } from "../tool-pair-order.js"; @@ -345,7 +346,10 @@ export async function prepareResponses( // every downstream consumer (decision prompt + directive, prose nudge, // emergency override, preflight length gates) sees only spans the // submit gate can accept — no surface advertises a doomed span. - if (turn.nudge) turn.nudge.compressibleRanges = excludeDeadRefRanges(viableRanges(turn.nudge.compressibleRanges), session.state.deadRefs); + // #2638: also drop remembered structurally-dead spans (kernel preview + // returned "no new compressible messages") — a nudge advertising them + // steers the model into an atomically-refused compress call. + if (turn.nudge) turn.nudge.compressibleRanges = filterDeadRanges(session, excludeDeadRefRanges(viableRanges(turn.nudge.compressibleRanges), session.state.deadRefs)); nudge = turn.nudge; session.stats.contextTokens = tokenCount; session.stats.contextTokensSource = tokenCountSource; diff --git a/src/server/side-request.ts b/src/server/side-request.ts index 4286eedc64..511daf8bdc 100644 --- a/src/server/side-request.ts +++ b/src/server/side-request.ts @@ -354,6 +354,22 @@ function noCeilingWarnText(sessionId: string, model: string, from: number, to: n return `[${sessionId}] output-budget restore has NO known output ceiling for model=${model} (raised ${from} -> ${to}) — the target was the client's own raw-history estimate, which decays unbounded; while it stays pathologically small every turn truncates at max-tokens. Declare the model's max output in the bili config (providers..models."${model}".output) or have the client report it to floor the restore higher (#1840/#3071)`; } +// #3438: one-shot per SESSION+MODEL warn for the UNRESCUABLE dsh-lane mid-band — +// a tool-carrying budget above the <=200 starve gate with NO known output ceiling +// cannot be told apart from a deliberate cap, so it forwards verbatim; this warn +// gives the previously silent high-water branch observability without spamming. +const warnedDshMidBandNoCeiling = new Set(); +function noteDshMidBandNoCeilingWarning(sessionId: string, model: string): boolean { + const key = `${sessionId}|${model}`; + if (warnedDshMidBandNoCeiling.has(key)) return false; + if (warnedDshMidBandNoCeiling.size >= WARNED_NO_OUTPUT_CEILING_CAP) warnedDshMidBandNoCeiling.clear(); + warnedDshMidBandNoCeiling.add(key); + return true; +} +export function _resetDshMidBandWarningsForTest(): void { + warnedDshMidBandNoCeiling.clear(); +} + // #2904: the no-ceiling degraded state surfaced by the /acp panel, the // /__bili/plugin/status?conversationId= JSON and the web session page while it holds. #1840's // once-per-model log warn is invisible to GUI hosts (stderr swallowed), so the @@ -398,12 +414,27 @@ function clearNoCeilingDegraded(session: { metadata: Record }, * providers that cap maxOutputTokens below 32K (availability over ideology, * #3128). A non-starved value is always trusted and recorded verbatim as the * high-water. The no-ceiling condition warns once per SESSION+model (the - * degradation is session-scoped). */ + * degradation is session-scoped). + * #3438: on lanes whose host stamps the model-declared maxTokens onto every + * main turn by contract (`dshLane` — the dsh host's pi-ai buildBaseOptions + * default), a tool-carrying budget BELOW the known ceiling is a raw-history + * collapse, not intent: the client's estimator sees zero real usage on + * replayed history and falls back to chars/4 over the unfolded history, which + * shrinks toward zero as the raw history approaches the window. Such mid-band + * values (above the <=200 starve gate) are treated as starved too — restored + * through the same target logic — and are NEVER recorded as the high-water + * (they are collapse evidence, not proven health). With NO known ceiling the + * lane cannot judge collapse from intent: the value forwards verbatim but + * warns once per SESSION+model so the operator can declare the model's output + * cap to arm the rescue. Non-dsh lanes keep the old behavior byte-exact — a + * deliberately small budget elsewhere is intent, not collapse. */ export function restoreOutputBudget( parsed: unknown, session: { id: string; metadata: Record }, log: (level: string, msg: string) => void, outputCeiling?: number, + /** #3438: true when this lane's host stamps the model-declared maxTokens onto every main turn by contract (dsh/pi-ai buildBaseOptions default) — see the doc above. */ + dshLane?: boolean, ): void { const field = outputBudgetField(parsed); if (!field) return; @@ -414,16 +445,25 @@ export function restoreOutputBudget( // A usable ceiling is known from some source (runtime-info / launcher / // configured route / registry); a value at/below the side threshold is not usable. const ceiling = typeof outputCeiling === "number" && outputCeiling > SIDE_REQUEST_MAX_TOKENS ? outputCeiling : undefined; - if (value > SIDE_REQUEST_MAX_TOKENS) { + // #3438: on the dsh lane a below-ceiling tool-carrying budget IS the collapse + // signature (see the doc above) — it must not pass through as "healthy", and + // it must not seed the high-water as if it were proven health. + const collapsed = dshLane === true && ceiling !== undefined && value < ceiling; + if (value > SIDE_REQUEST_MAX_TOKENS && !collapsed) { session.metadata.outputBudgetHighWater = value; // #2904: a healthy (non-starved) budget means the client's own estimate // recovered — the active no-ceiling pin for THIS model is gone, so drop // its marker (other models' markers are untouched). clearNoCeilingDegraded(session, modelName); + // #3438: dsh-lane mid-band with NO known ceiling — bili cannot judge + // collapse from intent here, so forward verbatim but surface the gap once. + if (dshLane === true && Array.isArray(p.tools) && p.tools.length > 0 && noteDshMidBandNoCeilingWarning(session.id, modelName)) { + log("warn", `[${session.id}] dsh-lane main request carries output budget ${value} with NO known output ceiling for model=${modelName} — cannot tell a raw-history collapse from deliberate intent, forwarding verbatim; declare the model's max output (providers..models."${modelName}".output) to arm the mid-band collapse rescue (#3438)`); + } return; } - // Starved main request (tiny budget + tools present). A side request (no tools) - // never reaches here — restoring it would break its side status. + // Starved or #3438-collapsed main request (budget + tools present). A side + // request (no tools) never reaches here — restoring it would break its side status. if (!Array.isArray(p.tools) || p.tools.length === 0) return; const highWaterRaw = session.metadata.outputBudgetHighWater; const highWater = typeof highWaterRaw === "number" && highWaterRaw > SIDE_REQUEST_MAX_TOKENS ? highWaterRaw : undefined; @@ -470,7 +510,12 @@ export function restoreOutputBudget( : (target === highWater ? "; no known output ceiling — restored to the session's own recorded budget, never raised above it (#3406)" : "; no known output ceiling — floored to the absolute capability minimum (#3071)"); - log("info", `[${session.id}] output budget restored ${value} -> ${target} (#546: client shrank it from its raw-history estimate${note})`); + // #3438: the <=200 starve keeps its original wording; a mid-band + // collapse names itself so the incident log distinguishes the two entries. + const origin = collapsed && value > SIDE_REQUEST_MAX_TOKENS + ? "#3438: dsh-lane mid-band collapse above the <=200 starve gate" + : "#546: client shrank it from its raw-history estimate"; + log("info", `[${session.id}] output budget restored ${value} -> ${target} (${origin}${note})`); } // #2904: persist the no-ceiling degraded state for the user-visible surfaces. // With the #3071 absolute floor a ceiling-less starve is ALWAYS restored (never diff --git a/src/session.ts b/src/session.ts index 25d0c08ddf..8a0b5d1e89 100644 --- a/src/session.ts +++ b/src/session.ts @@ -838,6 +838,25 @@ let MAX_SESSIONS = knobMaxSessions(); let MAX_RESIDENT_BYTES = sessionResidentMaxBytes(); const residentWeights = new WeakMap(); +// #3918: last-known LIVE weight of evicted sessions, stamped at eviction time. +// The live weight (transient message caches included) is only measurable while +// the object is resident — a disk reload carries persisted state alone and +// under-measures by exactly the mass that made the eviction necessary. The map +// is advisory (it only shapes refusal wording), never load-bearing; bounded so +// a long-lived process cannot accumulate unbounded entries. +const evictedWeights = new Map(); +const EVICTED_WEIGHT_CAP = 256; + +function stampEvictedWeight(id: string, bytes: number): void { + if (evictedWeights.has(id)) evictedWeights.delete(id); + evictedWeights.set(id, bytes); + while (evictedWeights.size > EVICTED_WEIGHT_CAP) { + const oldest = evictedWeights.keys().next().value; + if (oldest === undefined) break; + evictedWeights.delete(oldest); + } +} + /** Estimate retained objects without serializing a second giant copy. Include * originals and last-message caches; counting files alone misses both. */ function deepSizeOf(value: unknown): number { @@ -940,6 +959,42 @@ function residentBytes(): number { return bytes; } +/** #3918: can this session's message view survive BETWEEN model requests? The + * empty-view compress refusal (#3126) self-heals on the next model request ONLY + * when the reloaded session stays resident afterwards. When the session's + * last-known live weight plus the other residents exceeds the byte budget, + * every request cycle ends with it evicted again (releaseInFlight discharge) + * and the refusal repeats identically forever — the incident's 444 MiB image + * session ping-ponged against an 87.5 MiB peer under the 512 MiB default. + * Returns the measured pressure so the refusal receipt says so plainly instead + * of promising a heal that will not come. */ +interface ResidentPressure { + /** True when retrying cannot restore a usable view under the CURRENT pool state. */ + permanent: boolean; + /** Last-known LIVE weight of this session (eviction-time measurement; falls + * back to its resident weight when it is resident now). Undefined = unknown. */ + weightBytes?: number; + /** Resident bytes held by all OTHER sessions (this one excluded). */ + otherBytes: number; + residentBytes: number; + budgetBytes: number; +} + +export function residentPressure(id: string): ResidentPressure { + const resident = sessions.get(id); + const self = resident ? sessionWeight(resident) : 0; + const total = residentBytes(); + const known = evictedWeights.get(id); + const weightBytes = known !== undefined ? Math.max(known, self) : resident ? self : undefined; + return { + permanent: weightBytes !== undefined && weightBytes + (total - self) > MAX_RESIDENT_BYTES, + weightBytes, + otherBytes: total - self, + residentBytes: total, + budgetBytes: MAX_RESIDENT_BYTES, + }; +} + function makeResidentRoom(incoming = 0): void { while (sessions.size >= MAX_SESSIONS || (sessions.size > 0 && residentBytes() + incoming > MAX_RESIDENT_BYTES)) { if (!evictOldest()) { @@ -1644,6 +1699,7 @@ function evictOldest(): boolean { return false; } sessions.delete(oldestId); + stampEvictedWeight(oldestId, sessionWeight(s)); // #3126: eviction is the trigger for the empty-view compress window (the // session's message bytes live only here; state reloads from disk but the // view self-heals only on the next model request). Log it so operators can @@ -1667,6 +1723,7 @@ export function _resetSessionsForTest(max?: number, maxBytes?: number): void { if (s.inFlight > 0) s.inFlight = 0; } sessions.clear(); + evictedWeights.clear(); } export function _sessionsSizeForTest(): number { diff --git a/src/stream.ts b/src/stream.ts index 74a8c6bf21..0e0cf80c4a 100644 --- a/src/stream.ts +++ b/src/stream.ts @@ -2,7 +2,7 @@ import { createHash, randomBytes } from "node:crypto"; import { activeBlockSpans, collectBlockContent, countMessageTokens, defaultCountTokens, formatRanges, hideConsumedCompressCalls, storeCoveredOriginals, viableRanges, type CompressionCore, type Config, type CoreMessage, type CompressionState, type NudgeDecision, type CompressParseDiagnostics } from "acp-kernel"; import { excludeDeadRefRanges, handleAcpStatus, liveCompressionRanges } from "./acp-status.js"; import { handleAcpCache, recordCacheFoldsFromBlocks } from "./cache-ledger.js"; -import { type Session, cacheBlockContent, markDirty, scrubTurnSeparatorIds, statusInputBaseline } from "./session.js"; +import { type Session, cacheBlockContent, markDirty, residentPressure, scrubTurnSeparatorIds, statusInputBaseline } from "./session.js"; import { COMPRESS_TOOL_NAME, parseCompressInput, ABSORB_TOOL_NAME, type ParsedRange } from "./compress-tool.js"; import { effectiveAbsorbConfig, executeAbsorb, isProxyToolFor } from "./absorb.js"; import { executeSearchContextTarget, resolveDecompress } from "./decompress-shared.js"; @@ -633,14 +633,34 @@ export function noVisibleContextRefusal( }); if (touchable) return null; const detail = ranges.map((r) => `${r.startRef}–${r.endRef}`).join(", "); - ctx.log(`[acp-proxy: compress REFUSED ${detail} — no visible context (view=0 message(s), ${mappedRefs} ref(s) mapped); refs left untouched and nothing recorded as dead — the view self-heals on the next model request (#3126)]`); + // #3918: the "self-heals on the next model request" promise holds ONLY when + // the reloaded session survives between requests. When this session's + // last-known live weight cannot fit alongside the current residents within + // the byte budget, every request cycle ends with it evicted again and this + // exact refusal repeats forever — say so plainly instead of advising an + // impossible retry (the incident: a ~444 MiB image session ping-ponging + // against an ~87.5 MiB peer under the 512 MiB default budget). + const pressure = residentPressure(ctx.session.id); + ctx.log(pressure.permanent + ? `[acp-proxy: compress REFUSED ${detail} — no visible context (view=0 message(s), ${mappedRefs} ref(s) mapped); refs left untouched and nothing recorded as dead — PERMANENT: last-known weight ${formatMiB(pressure.weightBytes)} + other residents ${formatMiB(pressure.otherBytes)} > budget ${formatMiB(pressure.budgetBytes)}; the session cannot stay resident between model requests (#3918)]` + : `[acp-proxy: compress REFUSED ${detail} — no visible context (view=0 message(s), ${mappedRefs} ref(s) mapped); refs left untouched and nothing recorded as dead — the view self-heals on the next model request (#3126)]`); const specKey = normalizedSpecKey(ranges); - const repeatGuard = recordCompressFailure(ctx.session, specKey, "This failure is the proxy's missing in-memory message view, not your refs — it clears on the next model request. Wait for that turn instead of re-issuing."); + const repeatGuard = recordCompressFailure(ctx.session, specKey, pressure.permanent + ? "This failure repeats until the resident pool can hold this session alongside the others — closing another open session or raising sessions.residentMaxBytes clears it; waiting for the next turn does NOT." + : "This failure is the proxy's missing in-memory message view, not your refs — it clears on the next model request. Wait for that turn instead of re-issuing."); const loopGuard = opts?.loopTracking === false ? "" : noteCompressLoopFailure(ctx, specKey, undefined, "no-visible-context"); - const receipt = `[Compression FAILED: no visible context — the proxy holds no live message view of this conversation right now (its in-memory copy was dropped between model requests, e.g. by memory eviction or a restart). Your refs are intact and NOTHING was marked dead: ${mappedRefs} ref(s) remain mapped, and this is NOT a ref problem. Do NOT retry now — every retry fails identically until the next model request re-sends the conversation and restores the view. Continue your task; after the next turn you can compress the same ranges again. [cause: no-visible-context]${repeatGuard}${loopGuard}]`; + const healClause = pressure.permanent + ? ` THIS REFUSAL IS NOT TRANSIENT: your session's last measured memory footprint (~${formatMiB(pressure.weightBytes)}) plus the other resident sessions (~${formatMiB(pressure.otherBytes)}) exceed the resident pool budget (${formatMiB(pressure.budgetBytes)}), so every reload of this session is evicted again and no usable view survives between model requests — retrying now or after the next turn fails identically. To recover: finish or close the other open session(s), or raise sessions.residentMaxBytes in ~/.config/billion-context/billion-context.json (env twin BILI_SESSION_RESIDENT_MAX_BYTES) above ~${formatMiB((pressure.weightBytes ?? 0) + pressure.otherBytes)} and restart bili.` + : "Do NOT retry now — every retry fails identically until the next model request re-sends the conversation and restores the view. Continue your task; after the next turn you can compress the same ranges again."; + const receipt = `[Compression FAILED: no visible context — the proxy holds no live message view of this conversation right now (its in-memory copy was dropped between model requests, e.g. by memory eviction or a restart). Your refs are intact and NOTHING was marked dead: ${mappedRefs} ref(s) remain mapped, and this is NOT a ref problem.${healClause} [cause: no-visible-context]${repeatGuard}${loopGuard}]`; return compressResult(receipt, "refused", 0, "gate:no-visible-context"); } +function formatMiB(bytes: number | undefined): string { + if (bytes === undefined) return "?"; + return `${(bytes / (1024 * 1024)).toFixed(1)} MiB`; +} + // #2749: display-stub re-submission guard. In plugin mode the model's only // persistent record of a folded block's summary is the ~200-char display stub // that hideConsumedCompressCalls writes into the re-sent call args @@ -792,6 +812,41 @@ export function applyRanges(parsed: ReturnType, ctx: const beforeIds = new Set(ctx.session.state.blocks.map((b) => b.blockId)); const beforeSummaries = new Map(ctx.session.state.blocks.map((b) => [b.blockId, b.summary] as const)); const r = res.result; + // #3322: coverage-monotonicity backstop. A committed fold must never + // leave a previously-covered message without ANY covering block: on + // full-resend hosts the kernel's turn-integrity gate once judged splits + // against the fold's own set instead of post-fold coverage, so ids + // carried by consumed blocks were withdrawn wholesale — the processed + // view doubled (253→519), the local estimate blew past the window and + // preflight fail-fasted forever while upstream had headroom. Measure + // active-block coverage before vs after on the clone; any loss means + // this fold GROWS the live view, refuse it before commit (state and + // prefix cache untouched). Healthy folds never lose coverage: fresh + // folds add it, refolds preserve it, integrity withdrawals only touch + // direct ids of the range being folded. + { + const coveredAfter = new Set(); + for (const b of res.state.blocks) { + if (!b.active) continue; + for (const id of b.effectiveMessageIds) coveredAfter.add(id); + } + const lostCoverage = new Set(); + for (const b of ctx.session.state.blocks) { + if (!b.active) continue; + for (const id of b.effectiveMessageIds) { + if (!coveredAfter.has(id)) lostCoverage.add(id); + } + } + if (lostCoverage.size > 0) { + const detail = ranges.map((rg) => `${rg.startRef}–${rg.endRef}`).join(", "); + const sample = [...lostCoverage].slice(0, 5).map((id) => saltedMsgIdForLog(id)).join(", "); + const specKey = normalizedSpecKey(ranges); + const repeatGuard = recordCompressFailure(ctx.session, specKey); + ctx.log(`[acp-proxy: compress REFUSED ${detail} — coverage regression: ${lostCoverage.size} previously-folded message(s) would lose ALL covering block(s) (${sample}); committing it grows the live view and can push the local estimate past the model window (#3322). State left uncommitted, prefix cache untouched.]`); + const receipt = `[Compression refused: coverage regression — folding these ranges would leave ${lostCoverage.size} already-folded message(s) without any covering block, so the live context view would GROW instead of shrink (and the size estimate can blow past the model window, bricking the session behind permanent preflight rejections). Nothing was folded: no block was created and the context prefix/cache were left untouched. Retry with ranges aligned to whole turns (do not cut through a reasoning/tool-call/result group), target fewer blocks, or continue your task without compressing.${repeatGuard !== "" ? " " + repeatGuard : ""}]`; + return compressResult(receipt, "refused", 0, "gate:coverage-regression"); + } + } // #2748: host-side image notes append a ref-carrying footer to a new // block's summary AFTER the kernel's length validation — count that // wrapped mass in the net-benefit decision too. Moved above the commit diff --git a/tests/dead-ranges.test.ts b/tests/dead-ranges.test.ts new file mode 100644 index 0000000000..129afd7638 --- /dev/null +++ b/tests/dead-ranges.test.ts @@ -0,0 +1,90 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import type { CompressionBlock } from "acp-kernel"; +import type { Session } from "../src/session.ts"; +import { deadSkipKeys, filterDeadRanges, parseStructuralEmptyVerdict, recordDeadRange } from "../src/dead-ranges.ts"; + +const LIVELOCK = "range m00016..m00036: Range m00016..m00036 contains no new compressible messages — every message in it is already covered by active block(s) b3. Nothing was compressed. To rewrite or merge those blocks, reference them by block ID (b3..b3); otherwise run acp_st"; +const LIVELOCK_MULTI = "Range m00010..m00242 contains no new compressible messages — every message in it is already covered by active block(s) b2, b5. Nothing was compressed."; +const UNNAMED = "Range m00001..m00099 contains no compressible messages — all are already covered by active blocks or protected."; + +test("parseStructuralEmptyVerdict recognizes both structural wordings and names the covering blocks", () => { + assert.deepEqual(parseStructuralEmptyVerdict(LIVELOCK), ["b3"]); + assert.deepEqual(parseStructuralEmptyVerdict(LIVELOCK_MULTI), ["b2", "b5"]); + assert.deepEqual(parseStructuralEmptyVerdict(UNNAMED), []); +}); + +test("parseStructuralEmptyVerdict rejects non-structural verdicts (they stay retryable)", () => { + assert.equal(parseStructuralEmptyVerdict("Range m00001..m00005 is below the minimum compressible range"), null); + assert.equal(parseStructuralEmptyVerdict("Range intersects the protected recent zone; relax first"), null); + assert.equal(parseStructuralEmptyVerdict("unknown ref m99999"), null); + assert.equal(parseStructuralEmptyVerdict("summary length 12 below minSummaryLength 64"), null); + assert.equal(parseStructuralEmptyVerdict(""), null); +}); + +test("parseStructuralEmptyVerdict survives kernel truncation of the name list", () => { + const truncated = "Range m00001..m00900 contains no new compressible messages — every message in it is already covered by active block(s) b2, b5,"; + assert.deepEqual(parseStructuralEmptyVerdict(truncated), ["b2", "b5"]); +}); + +const block = (id: string): CompressionBlock => ({ blockId: id, runId: "r0", tier: 1, summary: "", directMessageIds: [], effectiveMessageIds: [], directBlockIds: [], compressedTokens: 0, createdAt: Date.now(), survivedCount: 0, generation: "young", active: true }); + +function makeSession(blockIds: string[]): Session { + return { + id: "dead-ranges-unit", meta: {}, metadata: {}, + state: { blocks: blockIds.map(block), messageRefs: { byRaw: {}, byRef: {} }, nudge: {}, stats: {}, nextBlockId: blockIds.length + 1 }, + createdAt: Date.now(), lastSeen: Date.now(), blockContents: new Map(), inFlight: 0, persisted: false, pendingRetrievals: [], + } as unknown as Session; +} + +test("recorded dead ranges persist in deadSkipKeys until a block lifecycle change", () => { + const session = makeSession(["b1", "b2", "b3"]); + recordDeadRange(session, "m00016", "m00036", ["b3"]); + recordDeadRange(session, "m00001", "m00005", []); + assert.deepEqual([...deadSkipKeys(session)].sort(), ["m00001:m00005", "m00016:m00036"]); + // Append-only growth never invalidates: only block lifecycle does. + session.state.blocks.push(block("b9")); + assert.ok(deadSkipKeys(session).has("m00016:m00036"), "a NEW unrelated block must not wake entries"); + // A named covering block dying wakes its entry (coverage left the span). + session.state.blocks.find((b) => b.blockId === "b3")!.active = false; + const after = deadSkipKeys(session); + assert.ok(!after.has("m00016:m00036"), "entry must die when a covering block goes inactive"); + // Subsumption: ANY recorded-active block dying wakes every entry conservatively. + assert.ok(!after.has("m00001:m00005"), "snapshot subsumption wakes all entries on any block death"); +}); + +test("re-recording upserts and the cap evicts oldest first", () => { + const session = makeSession(["b1"]); + recordDeadRange(session, "m00016", "m00036", ["b1"]); + recordDeadRange(session, "m00016", "m00036", ["b1", "b2"]); + const store = session.metadata!.preflightDeadRanges as Array<{ blocks: string[] }>; + assert.equal(store.length, 1, "same key must upsert, not duplicate"); + assert.deepEqual(store[0].blocks, ["b1", "b2"], "re-record refreshes the diagnostics"); + for (let i = 0; i < 70; i++) recordDeadRange(session, `m${String(i).padStart(5, "0")}`, `m${String(i + 1).padStart(5, "0")}`, []); + const capped = session.metadata!.preflightDeadRanges as Array<{ key: string }>; + assert.equal(capped.length, 64, "store must stay bounded"); + assert.ok(!capped.some((e) => e.key === "m00000:m00001"), "oldest entry evicted"); + assert.ok(capped.some((e) => e.key === "m00069:m00070"), "newest entry kept"); +}); + +test("filterDeadRanges drops exact keys only — wider containing ranges keep their live members", () => { + const session = makeSession(["b1"]); + recordDeadRange(session, "m00016", "m00036", ["b1"]); + const out = filterDeadRanges(session, [ + { startRef: "m00016", endRef: "m00036" }, + { startRef: "m00001", endRef: "m00040" }, + { startRef: "m00020", endRef: "m00030" }, + ]); + assert.deepEqual(out, [{ startRef: "m00001", endRef: "m00040" }, { startRef: "m00020", endRef: "m00030" }]); +}); + +test("the store survives a JSON persistence round-trip", () => { + const session = makeSession(["b1", "b2"]); + recordDeadRange(session, "m00016", "m00036", ["b2"]); + const revived = JSON.parse(JSON.stringify({ metadata: session.metadata })) as { metadata: Record }; + const restored: Session = makeSession(["b1", "b2"]); + restored.metadata = revived.metadata; + assert.deepEqual([...deadSkipKeys(restored)], ["m00016:m00036"]); + restored.state.blocks.find((b) => b.blockId === "b2")!.active = false; + assert.deepEqual([...deadSkipKeys(restored)], [], "revived entries invalidate like fresh ones"); +}); diff --git a/tests/issue3322-coverage-regression-guard.test.ts b/tests/issue3322-coverage-regression-guard.test.ts new file mode 100644 index 0000000000..2c25a13769 --- /dev/null +++ b/tests/issue3322-coverage-regression-guard.test.ts @@ -0,0 +1,120 @@ +import assert from "node:assert/strict"; +import { randomUUID } from "node:crypto"; +import { afterEach, test } from "node:test"; +import { createCore, defaultConfig, type Config, type CoreMessage } from "acp-kernel"; +import { applyRanges } from "../src/stream.ts"; +import { parseCompressInput } from "../src/compress-tool.ts"; +import { getSession } from "../src/session.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { _resetForTest as resetRegistryForTest } from "../src/registry.ts"; + +process.env.NODE_ENV = "test"; +_setStoreForTest(new SessionStore({ enabled: false })); + +afterEach(() => { + resetRegistryForTest(); +}); + +// #3322: a committed fold must never leave a previously-covered message +// WITHOUT ANY covering block. On full-resend hosts (dsh plugin lane compresses +// against the raw history, #2627) the kernel's turn-integrity gate once judged +// splits against the fold's OWN set instead of post-fold coverage: turns whose +// pieces sat in non-consumed sibling blocks were misread as split, their +// inherited ids withdrew wholesale, the consumed blocks deactivated — ~305 +// messages lost ALL coverage, the processed view doubled (253→519), the local +// estimate blew past the model window and preflight fail-fasted forever while +// upstream had headroom. applyRanges now measures active-block coverage before +// vs after on the clone and refuses the commit when it shrank (state and prefix +// cache untouched). The kernel root fix lands separately (Kernel Boundary); +// this guard is the host-side backstop pinning the commit contract either way. + +function makeCtx(prefix: string) { + const core = createCore(); + const config = defaultConfig(200_000) as Config; + config.compress.maxSummaryLength = 0; + // Fold ANY message: the guard under test must not depend on where the + // token-based recent-zone boundary falls inside this small fixture. + config.preserveRecentMessages = 0; + config.preserveRecentTokens = 0; + const session = getSession(`issue3322-${prefix}-${randomUUID()}`); + const msgs: CoreMessage[] = []; + for (let i = 0; i < 12; i++) { + msgs.push({ + id: `h_${i}`, + role: i % 2 === 0 ? "user" : "assistant", + contentType: "text", + text: `\x3cacp tokens="2K" type="text"\x3em${String(i + 1).padStart(5, "0")}\x3c/acp\x3e\nHistorical detail ${i}. ${"x".repeat(3000)}`, + }); + } + const turn = core.processTurn({ messages: msgs, state: session.state, config, tokenCount: 9999, renderTags: "text-only" }); + session.state = turn.state; + const logs: string[] = []; + const ctx = { core, config, messages: turn.messages, session, log: (msg: string) => { logs.push(msg); } }; + return { core, session, logs, ctx }; +} + +function coveredIds(state: { blocks: Array<{ active: boolean; effectiveMessageIds: string[] }> }): Set { + const out = new Set(); + for (const b of state.blocks) { + if (!b.active) continue; + for (const id of b.effectiveMessageIds) out.add(id); + } + return out; +} + +test("#3322 a fold that regresses coverage is refused BEFORE commit (state + cache untouched)", () => { + const { core, session, logs, ctx } = makeCtx("guard"); + + // Land a real fold first: b1 covers h_0/h_1. + const first = applyRanges(parseCompressInput({ content: [{ startId: "m00001", endId: "m00002", summary: "First summary: messages 1-2 covered the initial phase in detail." }] }), ctx as never); + assert.equal(first.outcome, "applied", `base fold must land: ${first.text}`); + const source = session.state.blocks.find((b) => b.active); + assert.ok(source, "the base fold materialized a block"); + assert.deepEqual([...coveredIds(session.state)].sort(), ["h_0", "h_1"]); + + // Simulate the incident on the NEXT fold: the kernel hands back a state in + // which the consuming block deactivated while the NEW block lost every + // inherited id (the pre-fix integrity gate's withdrawal shape). The guard + // must catch ANY such regression regardless of producer. + const realApply = core.applyCompression.bind(core); + core.applyCompression = (args) => { + const out = realApply(args); + const beforeIds = new Set(session.state.blocks.map((b) => b.blockId)); + for (const b of out.state.blocks) { + if (!beforeIds.has(b.blockId)) b.effectiveMessageIds = []; + } + return out; + }; + + const stateRefBefore = session.state; + const blocksBefore = session.state.blocks.length; + const distill = applyRanges(parseCompressInput({ content: [{ startId: source!.blockId, endId: source!.blockId, summary: "Distilled: the initial phase details merged into the running context." }] }), ctx as never); + assert.equal(distill.outcome, "refused", `a coverage-regressing fold must be refused: ${distill.text}`); + assert.equal(distill.reason, "gate:coverage-regression", `machine-readable reason: ${distill.reason}`); + assert.equal(distill.blocksCreated, 0, "no block is reported as created"); + assert.match(distill.text, /Compression refused: coverage regression/, `refusal is explicit: ${distill.text}`); + assert.match(distill.text, /Nothing was folded/, "the receipt states nothing was folded"); + assert.ok(!distill.text.includes("[Compressed "), `no fake success line: ${distill.text.slice(0, 120)}`); + assert.equal(session.state.blocks.length, blocksBefore, "no new block materialized"); + assert.equal(session.state, stateRefBefore, "state object uncommitted — prefix cache left untouched"); + assert.deepEqual([...coveredIds(session.state)].sort(), ["h_0", "h_1"], "previously-folded messages stay covered"); + assert.ok(logs.some((line) => line.includes("compress REFUSED") && line.includes("coverage regression")), "the refusal reaches the server log copy"); +}); + +test("#3322 legitimate nested consumption still applies (no false rejection)", () => { + const { session, ctx } = makeCtx("healthy"); + + const first = applyRanges(parseCompressInput({ content: [{ startId: "m00001", endId: "m00002", summary: "First summary: messages 1-2 covered the initial phase in detail." }] }), ctx as never); + assert.equal(first.outcome, "applied", `base fold must land: ${first.text}`); + + // Production shape of the incident batch: a later distillation whose span + // crosses the existing block's anchor consumes it — the new block INHERITS + // the old coverage, so the guard must let it through. + const second = applyRanges(parseCompressInput({ content: [{ startId: "m00001", endId: "m00006", summary: "Distilled span 1-6: initial phase plus the following exchange, details merged." }] }), ctx as never); + assert.equal(second.outcome, "applied", `nested consumption must not be refused: ${second.text}`); + assert.ok((second.blocksCreated ?? 0) >= 1, `a block landed: ${second.text}`); + const covered = coveredIds(session.state); + for (const id of ["h_0", "h_1", "h_2", "h_3", "h_4", "h_5"]) { + assert.ok(covered.has(id), `${id} must stay covered after the nested fold`); + } +}); diff --git a/tests/issue3846-local-refusal-evidence.test.ts b/tests/issue3846-local-refusal-evidence.test.ts new file mode 100644 index 0000000000..c6b869cd3b --- /dev/null +++ b/tests/issue3846-local-refusal-evidence.test.ts @@ -0,0 +1,320 @@ +// #3846: a locally refused oversized request (preflight 413, NOT forwarded) +// could be forwarded on retry once the usage baseline demoted to +// "estimate": the refusal lived only in the #726 dead-end marker's +// body-hash dimension, so (a) the #2313 estimate-only probe re-armed at +// baselineFloor === 0 and ran BEFORE the dead-end check, and (b) every +// turn's changed body missed the hash — production shape: three consecutive +// full-bill forwards of 606K/620K/633K-token payloads bili itself had +// already refused locally. The fix persists local refusal evidence in +// session metadata scoped to origin/model/window and keeps every +// forward-once escape hatch closed at the same or tighter window. Pinned +// here end-to-end, both directions (#3617 gate discipline): +// (a) positive — the issue's deterministic repro: fitting seed → oversized +// non-foldable system text → local 413 → baseline demotion + restart +// simulation → three identical retries stay 413 with ZERO additional +// upstream calls; a trimmed payload still forwards; +// (b) positive — production shape: retries with CHANGED bodies defeat the +// dead-end hash, so only the persisted evidence can hold the line; +// (c) negative — another model on the same origin, and another origin +// entirely, are evaluated independently (the escape hatches stay open +// where bili never refused); +// (d) hygiene — the refusal is its own metadata channel: it must never be +// labeled upstream usage (lastInputTokens/Source untouched). +import assert from "node:assert/strict"; +import http from "node:http"; +import { once } from "node:events"; +import test from "node:test"; + +process.env.NODE_ENV = "test"; + +import { defaultConfig } from "acp-kernel"; +import { startServer } from "../src/server.ts"; +import type { ProxyOptions } from "../src/config.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { _setForTest as setRegistryForTest } from "../src/registry.ts"; +import { getSession } from "../src/session.ts"; +import { normalizeUpstreamOrigin } from "../src/util.ts"; + +const WINDOW = 20_000; +const SUMMARY_TEXT = + "PREFLIGHT SUMMARY: the segment held deterministic load-growth payloads across turns; every marker derives from the turn index, so folding loses nothing."; + +// ~400KB of varying ASCII core text: ~100K tokens at the chars/4 estimate +// (5x the 20K window, robust across the whole k̂ clamp [0.25, 4]) yet far +// under the 2MB absolute doom bar, so the pre-fix probe would forward it. +function bigSystem(): string { + let out = ""; + for (let i = 0; i < 400_000; i += 64) { + out += `oversized non-foldable system load segment ${Math.floor(i / 64)}: ` + "x".repeat(40); + } + return out.slice(0, 400_000); +} + +type Call = { summary: boolean; stream: boolean; contentChars: number }; + +function chatSse(text: string, usage?: { prompt_tokens: number; completion_tokens: number }): string { + const chunk = (delta: Record, finish: string | null, extra?: Record): string => + `data: ${JSON.stringify({ id: "chatcmpl-1", object: "chat.completion.chunk", created: 1, model: "gpt-test", choices: [{ index: 0, delta, finish_reason: finish }], ...extra })}\n\n`; + const u = usage ? { usage } : {}; + return chunk({ role: "assistant" }, null) + chunk({ content: text }, null) + chunk({}, "stop", u) + "data: [DONE]\n\n"; +} + +// Counting fake upstream: bills what it receives at ~40 B/token and reports +// it as usage; answers summarization calls deterministically. +function makeUpstream(): { server: http.Server; calls: Call[] } { + const calls: Call[] = []; + const server = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (c: Buffer) => chunks.push(c)); + req.on("end", () => { + const raw = Buffer.concat(chunks).toString("utf8"); + const isSummary = /TASK: The conversation segment below/.test(raw); + try { + const parsed = JSON.parse(raw) as { stream?: boolean; messages?: Array<{ content?: unknown }> }; + const chars = (parsed.messages ?? []).reduce((n, m) => n + (typeof m.content === "string" ? m.content.length : 0), 0); + calls.push({ summary: isSummary, stream: !isSummary && !!parsed.stream, contentChars: chars }); + if (isSummary) { + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ id: "chatcmpl-sum", object: "chat.completion", model: "gpt-test", choices: [{ index: 0, message: { role: "assistant", content: SUMMARY_TEXT }, finish_reason: "stop" }] })); + return; + } + res.writeHead(200, { "content-type": "text/event-stream" }); + res.end(chatSse("forwarded answer", { prompt_tokens: Math.max(1, Math.round(chars / 40)), completion_tokens: 3 })); + } catch { + res.writeHead(400, { "content-type": "application/json" }); + res.end('{"error":{"message":"unparseable"}}'); + } + }); + }); + return { server, calls }; +} + +type Upstream = ReturnType; + +function proxyOptions(routes: Record): ProxyOptions { + return { + port: 0, + host: "127.0.0.1", + upstream: "http://127.0.0.1", + routes: routes as ProxyOptions["routes"], + modelContextLimit: WINDOW, + kernelConfig: defaultConfig(WINDOW, { + preserveRecentMessages: 2, + preserveRecentTokens: 2000, + compress: { minCompressRange: 1000, maxSummaryLength: 20000, minSummaryLength: 50 }, + }), + compress: { injectTool: true, injectNudge: true }, + sessionHeader: "x-acp-session", + log: false, + debug: false, + passthrough: false, + autoUpdate: false, + compat: { roles: {} }, + streamErrorShape: "protocol", + passthroughSource: null, + autoRestartOnUpdate: false, + updateTag: "latest", + advisoryCheck: false, + releaseNotesCheck: false, + promptCache: { routing: "auto" }, + mitm: { enabled: false, domains: [] }, + } as ProxyOptions; +} + +type Ctx = { url: string; headers: Record; upstreamPort: number }; + +async function listen(u: Upstream): Promise { + u.server.listen(0, "127.0.0.1"); + await once(u.server, "listening"); + return (u.server.address() as { port: number }).port; +} + +async function setup(routes: Record): Promise<{ ctx: Ctx; close: () => Promise }> { + _setStoreForTest(new SessionStore({ enabled: false })); + setRegistryForTest({}); + const firstKey = Object.keys(routes)[0]!; + const port = Number(new URL(firstKey).port); + const proxy = await startServer(proxyOptions(routes)); + await once(proxy, "listening"); + const proxyPort = (proxy.address() as { port: number }).port; + return { + ctx: { + url: `http://127.0.0.1:${proxyPort}/bili/${firstKey}/chat/completions`, + headers: { "content-type": "application/json", "x-acp-session": "" }, + upstreamPort: port, + }, + close: async () => { + proxy.close(); + proxy.closeIdleConnections?.(); + await Promise.allSettled([once(proxy, "close")]); + }, + }; +} + +const post = (ctx: Ctx, body: Record) => + fetch(ctx.url, { method: "POST", headers: { ...ctx.headers }, body: JSON.stringify(body) }); + +const streamCalls = (calls: Call[]) => calls.filter((c) => c.stream).length; +const summaryCalls = (calls: Call[]) => calls.filter((c) => c.summary).length; + +// The issue's repro step 3: the baseline demotion a preflight rebuild performs +// (lastInputTokens anchored to the rebuilt size, source "estimate"), followed +// by a restart simulation — only the persisted metadata survives the round trip. +function demoteToEstimate(sessionId: string): void { + const s = getSession(sessionId); + assert.ok(s, "session exists"); + s.stats.lastInputTokens = 60_000; + s.stats.lastInputTokensSource = "estimate"; + s.metadata = JSON.parse(JSON.stringify(s.metadata)); +} + +// Repro steps 1-2: a fitting seed settles a usage baseline; then an oversized +// NON-FOLDABLE system text (system is wire overhead — folding never removes +// it) must come back as a local 413 without touching the upstream. +async function seedAndRefuse(ctx: Ctx, sessionId: string, model: string): Promise { + ctx.headers["x-acp-session"] = sessionId; + const rSeed = await post(ctx, { model, stream: true, messages: [{ role: "user", content: "hello" }] }); + assert.equal(rSeed.status, 200, "the fitting seed forwards"); + await rSeed.text(); + + const rBig = await post(ctx, { model, stream: true, messages: [{ role: "system", content: bigSystem() }, { role: "user", content: "q" }] }); + assert.equal(rBig.status, 413, "oversized non-foldable payload is refused locally"); + const body = await rBig.text(); + assert.match(body, /NOT forwarded/, "the verdict says the payload did not ride"); +} + +test("e2e #3846 a: identical oversized retries stay local 413 after the baseline demotes to estimate — zero extra upstream calls", async () => { + const up = makeUpstream(); + const port = await listen(up); + const { ctx, close } = await setup({ [`http://127.0.0.1:${port}`]: { models: { "gpt-test": { context: WINDOW } } } }); + try { + await seedAndRefuse(ctx, "i3846-a", "gpt-test"); + assert.equal(streamCalls(up.calls), 1, "only the seed reached the upstream"); + assert.equal(summaryCalls(up.calls), 0, "the refusal spent no summarization call"); + + demoteToEstimate("i3846-a"); + + // Three byte-identical retries: pre-fix the FIRST one sailed through + // the estimate-only probe (which sits ahead of the dead-end check) and + // billed the full 400KB payload. + for (let i = 1; i <= 3; i++) { + const r = await post(ctx, { model: "gpt-test", stream: true, messages: [{ role: "system", content: bigSystem() }, { role: "user", content: "q" }] }); + assert.equal(r.status, 413, `retry ${i} stays a local 413`); + await r.text(); + } + assert.equal(streamCalls(up.calls), 1, "no oversized retry may reach the upstream"); + assert.equal(summaryCalls(up.calls), 0, "no summarization call may run either"); + + // Hygiene: the refusal is its own channel, never labeled usage. + const s = getSession("i3846-a")!; + assert.equal(s.stats.lastInputTokens, 60_000, "refusal evidence never rewrites the baseline value"); + assert.equal(s.stats.lastInputTokensSource, "estimate", "refusal evidence never relabels the baseline source"); + const refusals = s.metadata.localRefusals as Array<{ origin: string; model: string; window: number }>; + assert.ok(Array.isArray(refusals) && refusals.length === 1, `one refusal record, got ${JSON.stringify(refusals)}`); + assert.equal(refusals[0]!.origin, normalizeUpstreamOrigin(`http://127.0.0.1:${port}`)); + assert.equal(refusals[0]!.model, "gpt-test"); + assert.equal(refusals[0]!.window, WINDOW); + + // A fitting payload still forwards — the evidence closes escapes, not + // the lane. + const rTrim = await post(ctx, { model: "gpt-test", stream: true, messages: [{ role: "user", content: "small question" }] }); + assert.equal(rTrim.status, 200, "the trimmed request succeeds"); + await rTrim.text(); + assert.equal(streamCalls(up.calls), 2, "the trimmed request is the only new forward"); + } finally { + up.server.close(); + up.server.closeIdleConnections?.(); + await close(); + } +}); + +test("e2e #3846 b: CHANGED-body retries defeat the dead-end hash — only the persisted evidence holds the line", async () => { + const up = makeUpstream(); + const port = await listen(up); + const { ctx, close } = await setup({ [`http://127.0.0.1:${port}`]: { models: { "gpt-test": { context: WINDOW } } } }); + try { + await seedAndRefuse(ctx, "i3846-b", "gpt-test"); + demoteToEstimate("i3846-b"); + + // Production shape: each turn's body differs, so the #726 dead-end key + // (model+window+body hash) misses every time. Pre-fix each retry + // re-entered the estimate-only probe and forwarded once. + const tails = ["q-A", "q-B-tail", "q-C-even-longer-tail"]; + for (let i = 0; i < tails.length; i++) { + const r = await post(ctx, { model: "gpt-test", stream: true, messages: [{ role: "system", content: bigSystem() }, { role: "user", content: tails[i]! }] }); + assert.equal(r.status, 413, `changed-body retry ${i + 1} stays a local 413`); + await r.text(); + } + assert.equal(streamCalls(up.calls), 1, "zero additional upstream stream calls across all changed-body retries"); + assert.equal(summaryCalls(up.calls), 0, "zero summarization calls — nothing was foldable"); + } finally { + up.server.close(); + up.server.closeIdleConnections?.(); + await close(); + } +}); + +test("e2e #3846 c: another model on the same origin is evaluated independently — the probe stays open", async () => { + const up = makeUpstream(); + const port = await listen(up); + const { ctx, close } = await setup({ + [`http://127.0.0.1:${port}`]: { models: { "gpt-test": { context: WINDOW }, "gpt-other": { context: WINDOW } } }, + }); + try { + await seedAndRefuse(ctx, "i3846-c", "gpt-test"); + demoteToEstimate("i3846-c"); + + // Same origin, same window, DIFFERENT model: the refusal record does + // not cover it — the estimate-only bet is still worth one request. + const r = await post(ctx, { model: "gpt-other", stream: true, messages: [{ role: "system", content: bigSystem() }, { role: "user", content: "q" }] }); + assert.equal(r.status, 200, "an uncovered model still probes once"); + await r.text(); + assert.equal(streamCalls(up.calls), 2, "seed + exactly one probe forward for the other model"); + + const s = getSession("i3846-c")!; + const refusals = s.metadata.localRefusals as Array<{ model: string }>; + assert.equal(refusals.length, 1, "the probe on the other model adds no refusal record"); + assert.equal(refusals[0]!.model, "gpt-test"); + } finally { + up.server.close(); + up.server.closeIdleConnections?.(); + await close(); + } +}); + +test("e2e #3846 d: another ORIGIN is evaluated independently — the probe stays open", async () => { + const upA = makeUpstream(); + const upB = makeUpstream(); + const portA = await listen(upA); + const portB = await listen(upB); + const { ctx, close } = await setup({ + [`http://127.0.0.1:${portA}`]: { models: { "gpt-test": { context: WINDOW } } }, + [`http://127.0.0.1:${portB}`]: { models: { "gpt-test": { context: WINDOW } } }, + }); + try { + // Refuse on origin A (ctx.url points at A). + await seedAndRefuse(ctx, "i3846-d", "gpt-test"); + assert.equal(streamCalls(upA.calls), 1); + demoteToEstimate("i3846-d"); + + // Same session, same model, DIFFERENT origin B: no covering record → + // the probe fires again (a fresh route may bill on a different scale). + const urlB = ctx.url.replace(`:${portA}`, `:${portB}`); + const r = await fetch(urlB, { method: "POST", headers: { ...ctx.headers }, body: JSON.stringify({ model: "gpt-test", stream: true, messages: [{ role: "system", content: bigSystem() }, { role: "user", content: "q" }] }) }); + assert.equal(r.status, 200, "an uncovered origin still probes once"); + await r.text(); + assert.equal(streamCalls(upB.calls), 1, "origin B saw exactly the one probe forward"); + assert.equal(streamCalls(upA.calls), 1, "origin A saw nothing more"); + + const s = getSession("i3846-d")!; + const refusals = s.metadata.localRefusals as Array<{ origin: string }>; + assert.equal(refusals.length, 1, "the probe on the other origin adds no refusal record"); + assert.equal(refusals[0]!.origin, normalizeUpstreamOrigin(`http://127.0.0.1:${portA}`)); + } finally { + upA.server.close(); + upA.server.closeIdleConnections?.(); + upB.server.close(); + upB.server.closeIdleConnections?.(); + await close(); + } +}); diff --git a/tests/issue3918-preflight-progress-verdict.test.ts b/tests/issue3918-preflight-progress-verdict.test.ts new file mode 100644 index 0000000000..802a98c15b --- /dev/null +++ b/tests/issue3918-preflight-progress-verdict.test.ts @@ -0,0 +1,184 @@ +// #3918: a preflight that EXHAUSTS its summarization budget after committing +// real folds was reported terminal (502 retryable=false) even though the folds +// persist in the session and the NEXT attempt starts from a smaller baseline — +// the incident's 64 effective folds each got a "doomed" verdict. Non- +// deterministic exhaustion WITH progress must now be retryable=true with a +// progress note; zero-progress exhaustion keeps today's terminal verdict +// (pinned here AND in tests/preflight-multi-range.test.ts), and deterministic +// exhaustion keeps its 413 path (tests/issue3461-overlimit-exits.test.ts). +import assert from "node:assert/strict"; +import http from "node:http"; +import { once } from "node:events"; +import test from "node:test"; + +process.env.NODE_ENV = "test"; + +import { defaultConfig } from "acp-kernel"; +import { startServer } from "../src/server.ts"; +import type { ProxyOptions } from "../src/config.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { _setForTest as setRegistryForTest } from "../src/registry.ts"; +import { listSessions } from "../src/session.ts"; + +const SUMMARY_TEXT = + "PREFLIGHT SUMMARY: the segment covered a multi-step debugging session. " + + "Key decisions: chose the multi-range walk over lossy truncation because the payload must stay coherent. " + + "Files touched: src/a.ts:10, src/b.ts:20. Outcome: fixed and verified by tests."; + +function okSse(inputTokens: number): string { + return ( + `event: message_start\ndata: ${JSON.stringify({ type: "message_start", message: { id: "m1", role: "assistant", usage: { input_tokens: inputTokens } } })}\n\n` + + `event: content_block_start\ndata: ${JSON.stringify({ type: "content_block_start", index: 0, content_block: { type: "text", text: "" } })}\n\n` + + `event: content_block_delta\ndata: ${JSON.stringify({ type: "content_block_delta", index: 0, delta: { type: "text_delta", text: "ok" } })}\n\n` + + `event: content_block_stop\ndata: ${JSON.stringify({ type: "content_block_stop", index: 0 })}\n\n` + + `event: message_delta\ndata: ${JSON.stringify({ type: "message_delta", delta: { stop_reason: "end_turn", stop_sequence: null }, usage: { output_tokens: 3 } })}\n\n` + + `event: message_stop\ndata: ${JSON.stringify({ type: "message_stop" })}\n\n` + ); +} + +function conversation(n: number, tag: string): Array<{ role: string; content: string }> { + const msgs: Array<{ role: string; content: string }> = []; + for (let i = 0; i < n; i++) { + const role = i % 2 === 0 ? "user" : "assistant"; + msgs.push({ role, content: `${tag} Message ${i} of the long conversation. ` + `MARKER_${i}_content_`.repeat(250) }); + } + return msgs; +} + +type Call = { stream: boolean; body: string }; + +function makeUpstream(usable: (idx: number) => boolean): { server: http.Server; calls: Call[] } { + const calls: Call[] = []; + let summaryIdx = 0; + const server = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (c: Buffer) => chunks.push(c)); + req.on("end", () => { + const raw = Buffer.concat(chunks).toString("utf8"); + let parsed: { stream?: boolean } = {}; + try { parsed = JSON.parse(raw); } catch { /* keep {} */ } + calls.push({ stream: !!parsed.stream, body: raw }); + if (parsed.stream) { + res.writeHead(200, { "content-type": "text/event-stream" }); + res.end(okSse(1000)); + } else { + summaryIdx += 1; + const text = usable(summaryIdx) ? SUMMARY_TEXT : ""; + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ + id: "msg_summary", type: "message", role: "assistant", model: "claude-small", + content: [{ type: "text", text }], stop_reason: "max_tokens", + usage: { input_tokens: 500, output_tokens: 50 }, + })); + } + }); + }); + return { server, calls }; +} + +function startProxy(upstreamPort: number, contextWindow: number): Promise { + _setStoreForTest(new SessionStore({ enabled: false })); + setRegistryForTest({}); + return startServer({ + port: 0, + host: "127.0.0.1", + upstream: "http://127.0.0.1", + routes: { [`http://127.0.0.1:${upstreamPort}`]: { models: { "claude-small": { context: contextWindow } } } }, + modelContextLimit: 400_000, + kernelConfig: defaultConfig(400_000), + compress: { injectTool: true, injectNudge: true }, + promptCache: { routing: "auto" }, + sessionHeader: "x-acp-session", + log: false, + debug: false, + passthrough: false, + compat: { roles: {} }, streamErrorShape: "protocol", passthroughSource: null, autoRestartOnUpdate: false, updateTag: "latest", advisoryCheck: false, releaseNotesCheck: false, + autoUpdate: false, + mitm: { enabled: false, domains: [] }, + } as ProxyOptions); +} + +test("#3918: budget exhaustion WITH committed folds → 502 retryable=true + progress note (not terminal)", async () => { + // First two summarization draws are usable (two real folds land), every + // later draw is unusable: the walk burns the rest of the budget on failed + // ranges while the payload stays over-window — the incident's shape. + // Window 100K vs ~230K raw payload: even generous two-fold savings leave + // it well over-window, and the deficit stays below the remaining foldable + // pool so the #1841 futility bail never pre-empts the budget. + const { server: upstream, calls } = makeUpstream((idx) => idx <= 2); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const upstreamPort = (upstream.address() as { port: number }).port; + + const proxy = await startProxy(upstreamPort, 100_000); + await once(proxy, "listening"); + const proxyPort = (proxy.address() as { port: number }).port; + + try { + const r = await fetch(`http://127.0.0.1:${proxyPort}/bili/http://127.0.0.1:${upstreamPort}/v1/messages`, { + method: "POST", + headers: { "content-type": "application/json", "x-acp-session": "issue3918-progress-sess" }, + body: JSON.stringify({ model: "claude-small", max_tokens: 1024, stream: true, messages: conversation(200, "t3918-prog") }), + }); + assert.equal(r.status, 502, "still over-window after the budget → fail-fast 502"); + const json = JSON.parse(await r.text()) as { error?: { code?: string; retryable?: boolean; message?: string } }; + assert.equal(json.error?.code, "preflight_compress_failed"); + assert.equal(json.error?.retryable, true, `progress persists → the next attempt continues, so the verdict must be retryable (got: ${json.error?.retryable})`); + assert.match(json.error?.message ?? "", /summarization budget/i, `the budget variant is reported (got: ${json.error?.message})`); + assert.match(json.error?.message ?? "", /smaller baseline/i, `the progress note says the next attempt continues (got: ${json.error?.message})`); + assert.equal(calls.filter((c) => c.stream).length, 0, "the over-window payload was NOT forwarded"); + + const s = listSessions().find((x) => x.id === "issue3918-progress-sess") ?? listSessions()[0]; + const activeBlocks = (s?.state.blocks ?? []).filter((b) => b.active); + assert.ok(activeBlocks.length >= 1, `the committed folds persist in the session (got ${activeBlocks.length} active block(s))`); + } finally { + proxy.close(); + proxy.closeIdleConnections?.(); + await once(proxy, "close"); + upstream.close(); + upstream.closeIdleConnections?.(); + await once(upstream, "close"); + } +}); + +test("#3918 negative direction: budget exhaustion with ZERO progress stays terminal (retryable=false, no progress note)", async () => { + // Every draw unusable → nothing folds → today's terminal verdict stands. + // Window 230K reuses the proven sizing of + // tests/preflight-multi-range.test.ts (deficit far below the foldable + // pool, so the budget — not the futility bail — is the binding bound). + const { server: upstream, calls } = makeUpstream(() => false); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const upstreamPort = (upstream.address() as { port: number }).port; + + const proxy = await startProxy(upstreamPort, 230_000); + await once(proxy, "listening"); + const proxyPort = (proxy.address() as { port: number }).port; + + try { + const r = await fetch(`http://127.0.0.1:${proxyPort}/bili/http://127.0.0.1:${upstreamPort}/v1/messages`, { + method: "POST", + headers: { "content-type": "application/json", "x-acp-session": "issue3918-noprogress-sess" }, + body: JSON.stringify({ model: "claude-small", max_tokens: 1024, stream: true, messages: conversation(200, "t3918-none") }), + }); + assert.equal(r.status, 502, "nothing folded → fail-fast 502"); + const json = JSON.parse(await r.text()) as { error?: { code?: string; retryable?: boolean; message?: string } }; + assert.equal(json.error?.code, "preflight_compress_failed"); + assert.equal(json.error?.retryable, false, `zero progress → retries are genuinely doomed (got: ${json.error?.retryable})`); + assert.match(json.error?.message ?? "", /summarization budget/i, `the budget variant is reported (got: ${json.error?.message})`); + assert.doesNotMatch(json.error?.message ?? "", /smaller baseline/, `no progress note when nothing was folded (got: ${json.error?.message})`); + assert.ok(calls.filter((c) => !c.stream).length >= 2, `the walk ran before giving up (got ${calls.filter((c) => !c.stream).length} summary call(s))`); + assert.equal(calls.filter((c) => c.stream).length, 0, "the over-window payload was NOT forwarded"); + + const s = listSessions().find((x) => x.id === "issue3918-noprogress-sess") ?? listSessions()[0]; + const activeBlocks = (s?.state.blocks ?? []).filter((b) => b.active); + assert.equal(activeBlocks.length, 0, "no blocks created when nothing could be compressed"); + } finally { + proxy.close(); + proxy.closeIdleConnections?.(); + await once(proxy, "close"); + upstream.close(); + upstream.closeIdleConnections?.(); + await once(upstream, "close"); + } +}); diff --git a/tests/issue3918-resident-eviction-refusal.test.ts b/tests/issue3918-resident-eviction-refusal.test.ts new file mode 100644 index 0000000000..69ec1c2b2f --- /dev/null +++ b/tests/issue3918-resident-eviction-refusal.test.ts @@ -0,0 +1,116 @@ +// #3918: a session whose live footprint cannot fit the resident pool alongside +// other active sessions is evicted after EVERY model request (releaseInFlight +// discharge), so the empty-view compress refusal (#3126) repeats forever — yet +// its receipt kept promising "the next model request restores the view". These +// tests pin the honest diagnosis: when the session's last-known live weight +// plus the current residents exceeds the byte budget, the refusal names the +// permanent case and gives the actionable exits (close the peer / raise +// sessions.residentMaxBytes); when the pool CAN hold it (peer closed, restart, +// small session), the original transient self-heal wording stands. The gate +// itself (refusal, nothing marked dead) stays pinned in +// tests/issue3126-no-visible-context.test.ts. +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { mkdtempSync, rmSync } from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import type { CoreMessage } from "acp-kernel"; +import { createCore, defaultConfig, assignRefs, emptyRefMap } from "acp-kernel"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { getSession, acquireInFlight, releaseInFlight, markDirty, dropSessionForGc, residentPressure, _resetSessionsForTest } from "../src/session.ts"; +import { parseCompressInput } from "../src/compress-tool.ts"; +import { applyRanges, type RewriteCtx } from "../src/stream.ts"; + +const BUDGET = 300_000; + +const MESSAGES = Array.from({ length: 80 }, (_, i) => ({ + id: `raw${i}`, + role: (i % 2 ? "assistant" : "user") as "assistant" | "user", + text: "lorem ipsum dolor sit amet conseq ".repeat(8) + i, +})) as unknown as CoreMessage[]; + +function makeCtx(session: RewriteCtx["session"], logs: string[]): RewriteCtx { + return { + core: createCore(), + // Explicit protect-zone override (same recipe as + // tests/issue3126-no-visible-context.test.ts): the bare default would + // fence m00010..m00040 under recent-message protection and mask the + // gate behavior under test. + config: defaultConfig(200000, { preserveRecentMessages: 5, preserveRecentTokens: 0 }), + messages: [], + session, + log: (m: string) => { logs.push(m); }, + }; +} + +const SPEC = () => parseCompressInput({ content: [{ startId: "m00010", endId: "m00040", summary: "y".repeat(100) }] }); + +async function fixture(run: () => Promise): Promise { + const root = mkdtempSync(path.join(os.tmpdir(), "bili-3918-")); + const store = new SessionStore({ dir: path.join(root, "sessions"), enabled: true, debounceMs: 0 }); + _resetSessionsForTest(256, BUDGET); + _setStoreForTest(store); + try { await run(); } finally { + store.closeQuietly(); + _resetSessionsForTest(256, 512 * 1024 * 1024); + _setStoreForTest(new SessionStore({ enabled: false })); + rmSync(root, { recursive: true, force: true }); + } +} + +/** Grow `big-sess` past what BUDGET can hold alongside `peer-sess`, then run + * one request cycle over it: releaseInFlight's discharge evicts the oldest + * idle session (big, touched first) and stamps its live weight. Returns the + * evicted object — alive in JS, out of the pool, refs intact on its state. */ +function growAndEvictBig(): RewriteCtx["session"] { + const big = getSession("big-sess"); + const peer = getSession("peer-sess"); + big.state.messageRefs = assignRefs(MESSAGES, { existing: emptyRefMap(), nextIndex: 0 }).map; + big.metadata.blob = "x".repeat(80_000); + peer.metadata.blob = "y".repeat(60_000); + markDirty(big); + markDirty(peer); + acquireInFlight(big); + releaseInFlight(big); + return big; +} + +test("#3918: eviction stamps the live weight; pressure is permanent while the peer holds the budget, transient once it is gone", async () => fixture(async () => { + const big = growAndEvictBig(); + const p = residentPressure(big.id); + assert.ok(p.weightBytes !== undefined && p.weightBytes > 100_000, `live weight stamped at eviction (80K-char blob ≈ 160KiB): ${p.weightBytes}`); + assert.equal(p.budgetBytes, BUDGET); + assert.ok(p.otherBytes > 50_000, `the peer still occupies budget space: ${p.otherBytes}`); + assert.ok(p.permanent, `weight ${p.weightBytes} + others ${p.otherBytes} > budget ${BUDGET} → permanent`); + assert.equal(dropSessionForGc("peer-sess"), true); + const freed = residentPressure(big.id); + assert.equal(freed.permanent, false, "without the peer the session fits the budget → transient again"); + assert.equal(residentPressure("never-seen").permanent, false, "unknown session: no permanent claim"); + assert.equal(residentPressure("never-seen").weightBytes, undefined, "unknown session: no weight to quote"); +})); + +test("#3918: the refusal receipt names the PERMANENT case and gives the exits instead of an impossible retry", async () => fixture(async () => { + const big = growAndEvictBig(); + const logs: string[] = []; + const res = applyRanges(SPEC(), makeCtx(big, logs)); + assert.equal(res.outcome, "refused"); + assert.equal(res.reason, "gate:no-visible-context"); + assert.ok(res.text.includes("NOT TRANSIENT"), `receipt names the permanent case: ${res.text}`); + assert.ok(res.text.includes("sessions.residentMaxBytes"), `receipt gives the config exit: ${res.text}`); + assert.ok(!res.text.includes("after the next turn you can compress the same ranges again"), `impossible-retry advice removed: ${res.text}`); + assert.ok(logs.some((l) => l.includes("PERMANENT")), `operator log carries the marker: ${JSON.stringify(logs)}`); + assert.equal(res.blocksCreated, 0); + assert.equal(big.state.deadRefs, undefined, "still nothing marked dead"); +})); + +test("#3918: with the peer gone the same empty-view shape keeps the transient self-heal advice (negative direction)", async () => fixture(async () => { + const big = growAndEvictBig(); + assert.equal(dropSessionForGc("peer-sess"), true); + const logs: string[] = []; + const res = applyRanges(SPEC(), makeCtx(big, logs)); + assert.equal(res.outcome, "refused"); + assert.equal(res.reason, "gate:no-visible-context"); + assert.ok(res.text.includes("Do NOT retry now"), `transient wording intact: ${res.text}`); + assert.ok(!res.text.includes("NOT TRANSIENT"), `no permanent claim when the pool fits: ${res.text}`); + assert.ok(!logs.some((l) => l.includes("PERMANENT")), "log stays on the #3126 self-heal line"); +})); diff --git a/tests/native-intercept.test.ts b/tests/native-intercept.test.ts index d3f244bddd..2f9548fe7f 100644 --- a/tests/native-intercept.test.ts +++ b/tests/native-intercept.test.ts @@ -1,5 +1,8 @@ import test from "node:test"; import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; import { createLiveOriginResolver, installNativeFetchIntercept, isModelApiUrl, liveStateSlotEffect, noteRoutedOrigin, observeRoutedOrigin, _resetForTest, type NativeInterceptState } from "../src/agent/native-intercept.ts"; import { makeRecordingFetch, runHotReplace, type RecordedCall } from "./helpers/hot-replace.ts"; @@ -1140,6 +1143,42 @@ test("#1365 noteRoutedOrigin: records origin, sticky per origin, hook fires on t assert.equal(seen.length, 2); }); +// #3923: spawn-mode lanes (no owner hook) used to absorb routed-channel +// evidence SILENTLY — a `/bili/` URL baked against a foreign origin kept +// passing through to its baked instance while raw URLs followed state.origin, +// splitting one conversation across two live instances with zero signal until +// someone manually diffed [v=…] tags in the shared log. Now: one error line +// per foreign origin into bili.log. Warn-only — spawn mode has no recovery +// policy, so it never rebinds; attach lanes are untouched (their owner hook +// handles rebinds and the fallback only fires when no hook is armed). + +test("#3923 noteRoutedOrigin spawn-mode fallback: foreign origin warns once per origin, never rebinds", () => { + const saved = process.env.XDG_STATE_HOME; + const root = fs.mkdtempSync(path.join(os.tmpdir(), "bili-3923-routed-")); + process.env.XDG_STATE_HOME = root; + // XDG_STATE_HOME override replaces the base dir entirely — no .local/state + const logFile = path.join(root, "billion-context", "bili.log"); + try { + const state: NativeInterceptState = { origin: "http://127.0.0.1:18802", ready: Promise.resolve("http://127.0.0.1:18802") }; + noteRoutedOrigin(state, "http://127.0.0.1:18991/bili/https://api.anthropic.com/v1/messages"); + noteRoutedOrigin(state, "http://127.0.0.1:18991/bili/http://127.0.0.1:8199/v1/chat/completions"); + assert.equal(state.routedOrigin, "http://127.0.0.1:18991"); + assert.equal(state.origin, "http://127.0.0.1:18802", "warn-only: spawn mode never rebinds"); + let lines = fs.readFileSync(logFile, "utf8").split("\n").filter((l) => l.includes("(#3923)")); + assert.equal(lines.length, 1, "one line per foreign origin, not per request"); + assert.match(lines[0], /\[error\]/); + assert.ok(lines[0].includes("http://127.0.0.1:18991") && lines[0].includes("http://127.0.0.1:18802"), "names both sides"); + + const slashState: NativeInterceptState = { origin: "http://127.0.0.1:18993/", ready: Promise.resolve("http://127.0.0.1:18993/") }; + noteRoutedOrigin(slashState, "http://127.0.0.1:18993/bili/http://127.0.0.1:8199/v1/messages"); + lines = fs.readFileSync(logFile, "utf8").split("\n").filter((l) => l.includes("(#3923)")); + assert.equal(lines.length, 1, "same origin modulo trailing slash is not a mismatch"); + } finally { + if (saved === undefined) delete process.env.XDG_STATE_HOME; else process.env.XDG_STATE_HOME = saved; + fs.rmSync(root, { recursive: true, force: true }); + } +}); + test("#1365 pre-gate proof: first routed request records evidence while toolsReady is still pending", async () => { let releaseTools: () => void = () => {}; const toolsReady = new Promise((r) => { releaseTools = r; }); diff --git a/tests/preflight-dead-ranges.test.ts b/tests/preflight-dead-ranges.test.ts new file mode 100644 index 0000000000..981803b0f9 --- /dev/null +++ b/tests/preflight-dead-ranges.test.ts @@ -0,0 +1,170 @@ +import assert from "node:assert/strict"; +import http from "node:http"; +import { once } from "node:events"; +import test from "node:test"; +import { createCore, createInitialState, defaultConfig, defaultPrompts, type CoreMessage } from "acp-kernel"; +import { preflightCompress } from "../src/preflight.ts"; +import type { Session } from "../src/session.ts"; + +process.env.NODE_ENV = "test"; +const SUMMARY = "The preceding work is summarized here with all decisions and remaining tasks preserved for the next turn."; + +// #2638 minimal geometry (full-log host resends the ENTIRE history every turn; +// mimics a days-old autoFold session whose middle is already fully folded): +// b_head = m1..m6 (anchor m1) <- everything before the dead span is covered +// A@7, B@17, M22@22, M37@37 = checkpoints (summaryOfBlockId=b_head, live carriers) +// b_mid = m8..m21 minus carriers (anchor m8) -> covered raws sit BETWEEN B and M22 +// b_tailblock = m23..m36 (anchor m23) -> nests inside the advertised span +// The carrier cluster [A,B,M22] looks like a plain compressible group in the pruned view +// (covered raws invisible there), so the kernel advertises m00007..m00022 every turn; at +// apply time every message of that positional span is either covered by an active block or +// carved out as a live carrier -> structurally empty -> permanent "no new compressible +// messages" rejection loop (#2638). +async function buildDeadSpanFixture() { + const core = createCore(); + const config = defaultConfig(1_000_000); + config.preserveRecentMessages = 0; + config.preserveRecentTokens = 0; + config.compress.minCompressRange = 1000; // chars — merge batch gate: carrier cluster must clear it alone to be advertised + const H: CoreMessage[] = []; + let n = 0; + const pushU = (text: string) => { n++; const id = `msg-${n}`; H.push({ id, role: "user", contentType: "text", text }); return id; }; + const pushA = (text: string) => { n++; const id = `msg-${n}`; H.push({ id, role: "assistant", contentType: "text", text }); return id; }; + let state = createInitialState(); + const refresh = () => { + const t = core.processTurn({ messages: H, state, config, tokenCount: 9000, renderTags: "text-only" }); + state = t.state; + return state.messageRefs.byRaw; + }; + // Block ids are kernel-minted (b1, b2, ...) in creation order — capture them: the + // checkpoint carriers must declare summaryOfBlockId against the REAL id to count as live. + const fold = (a: string, z: string, label: string): { result: ReturnType["result"]; blockId: string } => { + const refs = refresh(); + const before = state.blocks.length; + const r = core.applyCompression({ messages: H, state, config, ranges: [{ startRef: refs[a], endRef: refs[z], summary: `${label} SUMMARY `.repeat(40) }] }); + state = r.state; + assert.equal(state.blocks.length, before + 1, `fold ${label} must create exactly one block`); + return { result: r.result, blockId: state.blocks[before]!.blockId }; + }; + refresh(); + pushU("FIRST USER ".repeat(40)); // m1 + pushA("S0a ".repeat(200)); pushA("S0b ".repeat(200)); pushA("S0c ".repeat(200)); // m2..m4 + pushU("continue stage 0"); // m5 + pushA("w6 ".repeat(200)); // m6 + const A = pushU("CHECKPOINT A: stage 0 complete; decisions recorded; proceed to stage 1 as agreed. ".repeat(6)); // m7 (~500 chars — below the merge gate alone) + const head = fold("msg-1", "msg-6", "HEAD"); // covers m1..m6 (anchor idx 0) + (H.find((m) => m.id === A) as CoreMessage).summaryOfBlockId = head.blockId; + for (let i = 0; i < 9; i++) pushA(`S1 ${i} `.repeat(200)); // m8..m16 + const B = pushU("CHECKPOINT B: stage 1 complete; remaining tasks listed; proceed to stage 2 with the agreed plan. ".repeat(18)); // m17 (~1500 chars) + (H.find((m) => m.id === B) as CoreMessage).summaryOfBlockId = head.blockId; + for (let i = 0; i < 4; i++) pushA(`S1t ${i} `.repeat(200)); // m18..m21 + const M22 = pushU("follow-up: confirm stage 1 outputs landed and stage 2 inputs are ready before proceeding further ".repeat(24)); // m22 (~2300 chars) + (H.find((m) => m.id === M22) as CoreMessage).summaryOfBlockId = head.blockId; + const mid = fold("msg-8", "msg-21", "MID"); // carriers A,B,M22 excluded from coverage + for (let i = 0; i < 14; i++) pushA(`S2 ${i} `.repeat(200)); // m23..m36 + const M37 = pushU("keep going with stage 2"); // m37 + (H.find((m) => m.id === M37) as CoreMessage).summaryOfBlockId = head.blockId; + fold("msg-23", "msg-36", "TAILBLOCK"); + const bHead = state.blocks.find((b) => b.blockId === head.blockId)!; + assert.equal(bHead.effectiveMessageIds.length, 6, "geometry precondition: head block covers the whole head"); + const bMid = state.blocks.find((b) => b.blockId === mid.blockId)!; + assert.ok(bMid.active && !bMid.effectiveMessageIds.includes(A) && !bMid.effectiveMessageIds.includes(B) && !bMid.effectiveMessageIds.includes(M22), "geometry precondition: mid block must exclude its live carriers"); + assert.ok(mid.result.warnings.some((w) => w.includes("checkpoint")), "setup must show the carrier exclusion"); + return { core, config, H, state }; +} + +function makeSession(state: Awaited>["state"]): Session { + return { + id: "dead-range-preflight", meta: {}, metadata: {}, state, + stats: { requests: 1, tokensSaved: 0, inputTokens: 0, cachedTokens: 0, outputTokens: 0, cacheSamples: 0, lastInputTokens: 0, compressCreditTokens: 0, contextTokens: 0, retrieveCalls: 0, retrieveHits: 0, retrieveMisses: 0, storedBytes: 0, storeBytesSaved: 0, rangeRestores: 0 }, + createdAt: Date.now(), lastSeen: Date.now(), blockContents: new Map(), inFlight: 0, persisted: false, pendingRetrievals: [], + } as unknown as Session; +} + +test("#2638 a structurally-dead span is preview-rejected once, remembered, and never re-nominated", async () => { + const { core, config, H, state } = await buildDeadSpanFixture(); + const session = makeSession(state); + const summaries: string[] = []; + const upstream = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (chunk: Buffer) => chunks.push(chunk)); + req.on("end", () => { + const body = JSON.parse(Buffer.concat(chunks).toString("utf8")) as { input: { content: string }[] }; + summaries.push(body.input[0].content); + res.setHeader("content-type", "application/json"); + res.end(JSON.stringify({ output_text: SUMMARY })); + }); + }); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const port = (upstream.address() as { port: number }).port; + try { + // Target sits between the post-tail-fold payload (~1.8K tok) and the pre-fold + // payload (~2.1K tok) so each invocation's walk fires once and exits after folding. + // NOTE: master's kernel no longer advertises the structurally-dead carrier cluster — + // recommend now shares apply's unit-projection judgment (#3236) and screens + // consumed units out of recommendations (#3371), so the dead span never reaches the + // server from THIS kernel any more. #2638's server-side dead-range memory is + // defense-in-depth for kernels/geometries that still advertise such spans + // (downgraded kernels, future recommend regressions): force the nomination through a + // processTurn wrapper — preview, apply, and state stay on the real kernel, so the + // structural rejection verdict and the tail folds remain genuine. + const refs = state.messageRefs.byRaw; + const deadStart = refs["msg-7"]!; + const deadEnd = refs["msg-22"]!; + const injectedCore: typeof core = Object.create(core); + injectedCore.processTurn = ((args: Parameters[0]) => { + const turn = core.processTurn(args); + const ranges = turn.nudge?.compressibleRanges; + if (!ranges || ranges.some((r) => r.startRef === deadStart && r.endRef === deadEnd)) return turn; + return { ...turn, nudge: { ...turn.nudge!, compressibleRanges: [...ranges, { startRef: deadStart, endRef: deadEnd, count: 16, tokens: 1141, chars: 4566, toolPct: 0, textPct: 1 }] } }; + }) as typeof core.processTurn; + const runPreflight = async (logs: string[], cfg = config) => preflightCompress({ core: injectedCore, session, config: cfg, prompts: defaultPrompts, protocol: "responses", url: `http://127.0.0.1:${port}`, headers: {}, model: "test", unknownBaseline: false, compressionTarget: 2000, log: (_level, message) => logs.push(message) }, H); + // Turn 1: the dead span is nominated, previewed, rejected ONCE, remembered; the + // foldable tail still folds. Without the memory the same span is re-previewed in + // later rounds of this very invocation ("nominated twice within one request"). + H.push({ id: `msg-${H.length + 1}`, role: "user", contentType: "text", text: "NEW TURN 1 ".repeat(60) }); + H.push({ id: `msg-${H.length + 1}`, role: "assistant", contentType: "text", text: "REPLY 1 ".repeat(60) }); + const logs1: string[] = []; + const r1 = await runPreflight(logs1); + const rejects1 = logs1.filter((l) => l.includes("preview rejected") && l.includes("no new compressible messages")); + assert.equal(rejects1.length, 1, `expected exactly one structural rejection in invocation 1, got: ${logs1.join("\n")}`); + const rejectsKey = /preview rejected range (m\d+:m\d+)/.exec(rejects1[0])?.[1]; + assert.equal(rejectsKey, "m00007:m00022", `the dead span is the carrier cluster A..M22, got: ${logs1.join("\n")}`); + assert.ok(r1.hadFoldableCandidates, "the tail was foldable"); + assert.ok(r1.compressedRanges >= 1, "viable tail must still fold"); + const stored = session.metadata!.preflightDeadRanges as Array<{ key: string; blocks: string[] }>; + assert.ok(stored.some((e) => e.key === "m00007:m00022"), "the dead span must be remembered on the session"); + assert.ok(stored.some((e) => e.key === "m00007:m00022" && e.blocks.includes("b2")), "the named nested block (b2) must be captured for diagnostics"); + // Turn 2: full-log host resends everything plus a fresh pair. The dead span must NOT + // be re-nominated at all (no second rejection, no wasted preview), the new tail folds. + H.push({ id: `msg-${H.length + 1}`, role: "user", contentType: "text", text: "NEW TURN 2 ".repeat(60) }); + H.push({ id: `msg-${H.length + 1}`, role: "assistant", contentType: "text", text: "REPLY 2 ".repeat(60) }); + const logs2: string[] = []; + const r2 = await runPreflight(logs2); + const rejects2 = logs2.filter((l) => l.includes("preview rejected") && l.includes("no new compressible messages")); + assert.equal(rejects2.length, 0, `dead span must not be re-nominated in invocation 2, got: ${logs2.join("\n")}`); + assert.ok(r2.compressedRanges >= 1, "fresh tail must still fold on invocation 2"); + assert.ok(r2.hadFoldableCandidates); + // Carrier text must NEVER reach the summarizer through any plain-ref range. + assert.ok(summaries.every((c) => !c.includes("CHECKPOINT A") && !c.includes("CHECKPOINT B")), "live checkpoint carriers must never be resummarized"); + // Steady state: raise minCompressRange above the (small) fresh tail but below the + // (large) dead span — every above-min candidate is remembered-dead, so the walk must + // find NOTHING foldable and report it as such (this is what keeps the auto-fold + // backoff from arming in lean steady state). + H.push({ id: `msg-${H.length + 1}`, role: "user", contentType: "text", text: "NEW TURN 3 ".repeat(60) }); + H.push({ id: `msg-${H.length + 1}`, role: "assistant", contentType: "text", text: "REPLY 3 ".repeat(60) }); + const cfg3 = structuredClone(config); + cfg3.compress.minCompressRange = 4000; // below the dead span (4566 chars), above the fresh tail (~1140 chars) + const logs3: string[] = []; + const r3 = await runPreflight(logs3, cfg3); + assert.equal(r3.compressedRanges, 0, "nothing foldable remains"); + assert.equal(r3.hadFoldableCandidates, false, `steady state must report no foldable candidates, got: ${JSON.stringify(r3)}\n${logs3.join("\n")}`); + assert.ok(logs3.some((l) => l.includes("previously-dead structural verdicts")), "the suppression must leave a diagnosable trace"); + } finally { + const closed = once(upstream, "close"); + upstream.close(); + upstream.closeAllConnections(); + await closed; + } +}); diff --git a/tests/served-markers.test.ts b/tests/served-markers.test.ts new file mode 100644 index 0000000000..93f75d662b --- /dev/null +++ b/tests/served-markers.test.ts @@ -0,0 +1,142 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { createServedMarkerWriter, findContestedWriters, reapStaleWriterMarkers, writerMarkerDir, type WriterIdentity } from "../src/served-markers.ts"; + +// #3923: per-session writer markers — session-level split-brain visibility. +// Two live bili processes sharing one sessions directory already warn at +// MACHINE level (#394 registration, #2401 slow tick); these tests pin the +// SESSION level: who committed what when, both sides named, verdicts +// throttled, dead peers reaped, and the whole surface best-effort (no throw +// into the save chain). + +const SELF: WriterIdentity = { instanceId: "self-1", pid: process.pid, origin: "http://127.0.0.1:18802", version: "0.1.202" }; +const PEER: WriterIdentity = { instanceId: "peer-1", pid: process.pid, origin: "http://127.0.0.1:8787", version: "0.1.201" }; +// Beyond any sane pid_max (Linux default 4194304) — provably dead. +const DEAD_PID = 999_999_999; + +function freshDir(prefix: string): string { + return fs.mkdtempSync(path.join(os.tmpdir(), prefix)); +} + +function writePeerMarker(dir: string, peer: WriterIdentity, servedAt: Record): string { + fs.mkdirSync(dir, { recursive: true }); + const file = path.join(dir, `${peer.instanceId}.json`); + fs.writeFileSync(file, JSON.stringify({ v: 1, ...peer, servedAt })); + return file; +} + +function backdate(file: string, ageMs: number): void { + const t = (Date.now() - ageMs) / 1000; + fs.utimesSync(file, t, t); +} + +test("marker file lands with identity + served sessions after flush", () => { + const dir = writerMarkerDir(freshDir("bili-3923-marker-")); + const logs: Array<[string, string]> = []; + const w = createServedMarkerWriter(SELF, dir, (level, msg) => logs.push([level, msg])); + w.noteServed("sess-a"); + w.__flushForTest(); + const body = JSON.parse(fs.readFileSync(path.join(dir, "self-1.json"), "utf8")) as Record; + assert.equal(body.v, 1); + assert.equal(body.instanceId, "self-1"); + assert.equal(body.pid, process.pid); + assert.equal(body.origin, "http://127.0.0.1:18802"); + assert.equal(body.version, "0.1.202"); + const servedAt = body.servedAt as Record; + assert.equal(typeof servedAt["sess-a"], "number"); + assert.deepEqual(w.__servedForTest(), { "sess-a": servedAt["sess-a"] }); + assert.equal(logs.length, 0, "no contention — no log lines"); + w.dispose(); +}); + +test("dispose flushes pending entries; noteServed after dispose is a no-op", () => { + const dir = writerMarkerDir(freshDir("bili-3923-dispose-")); + const w = createServedMarkerWriter(SELF, dir, () => {}); + w.noteServed("sess-late"); + w.dispose(); + const body = JSON.parse(fs.readFileSync(path.join(dir, "self-1.json"), "utf8")) as Record; + assert.ok((body.servedAt as Record)["sess-late"] !== undefined, "final flush persisted the entry"); + w.noteServed("sess-after-dispose"); + assert.ok((w.__servedForTest()["sess-after-dispose"]) === undefined, "disposed writer records nothing"); +}); + +test("contested session logs [split-brain] with both sides named, once per verdict window", () => { + const dir = writerMarkerDir(freshDir("bili-3923-contest-")); + const logs: Array<[string, string]> = []; + writePeerMarker(dir, PEER, { "sess-a": Date.now() - 1000 }); + const w = createServedMarkerWriter(SELF, dir, (level, msg) => logs.push([level, msg])); + w.noteServed("sess-a"); + assert.equal(logs.length, 1, "first commit of a contested session names the conflict"); + assert.equal(logs[0][0], "error"); + assert.match(logs[0][1], /\[split-brain\] session sess-a/); + assert.ok(logs[0][1].includes("http://127.0.0.1:18802") && logs[0][1].includes("v0.1.202"), "names THIS side"); + assert.ok(logs[0][1].includes("http://127.0.0.1:8787") && logs[0][1].includes("v0.1.201"), "names the PEER side"); + assert.match(logs[0][1], /#3923/); + w.noteServed("sess-a"); + assert.equal(logs.length, 1, "verdict TTL suppresses repeats inside the window"); + w.dispose(); +}); + +test("non-contested session stays silent even while another session is contested", () => { + const dir = writerMarkerDir(freshDir("bili-3923-solo-")); + const logs: Array<[string, string]> = []; + writePeerMarker(dir, PEER, { "other-session": Date.now() - 1000 }); + const w = createServedMarkerWriter(SELF, dir, (level, msg) => logs.push([level, msg])); + w.noteServed("solo-session"); + assert.equal(logs.length, 0, "peer activity on a DIFFERENT session does not contest this one"); + w.dispose(); +}); + +test("peer commit outside the contest window is not contested", () => { + const dir = writerMarkerDir(freshDir("bili-3923-stale-window-")); + const logs: Array<[string, string]> = []; + writePeerMarker(dir, PEER, { "sess-a": Date.now() - 11 * 60_000 }); + const w = createServedMarkerWriter(SELF, dir, (level, msg) => logs.push([level, msg])); + w.noteServed("sess-a"); + assert.equal(logs.length, 0, "an 11-minute-old peer commit is not concurrent activity"); + w.dispose(); +}); + +test("dead-pid peer counts while its marker is fresh, stops counting once stale", () => { + const dir = writerMarkerDir(freshDir("bili-3923-freshness-")); + const deadPeer: WriterIdentity = { ...PEER, pid: DEAD_PID }; + const file = writePeerMarker(dir, deadPeer, { "sess-fresh": Date.now() - 1000 }); + let found = findContestedWriters(dir, SELF.instanceId, "sess-fresh", Date.now()); + assert.equal(found.length, 1, "fresh mtime covers platforms where the pid check cannot see the peer"); + backdate(file, 40_000); + found = findContestedWriters(dir, SELF.instanceId, "sess-fresh", Date.now()); + assert.equal(found.length, 0, "dead pid + stale mtime = gone"); +}); + +test("malformed or foreign-shape marker files are skipped, never fatal", () => { + const dir = writerMarkerDir(freshDir("bili-3923-malformed-")); + fs.mkdirSync(dir, { recursive: true }); + fs.writeFileSync(path.join(dir, "garbage.json"), "{not json"); + writePeerMarker(dir, { ...PEER, instanceId: "peer-v2" }, {}); + // v mismatch — future format must not be misread as v1 + fs.writeFileSync(path.join(dir, "peer-v2.json"), JSON.stringify({ v: 2, instanceId: "peer-v2", pid: process.pid, origin: "x", version: "y", servedAt: { "sess-m": Date.now() } })); + assert.deepEqual(findContestedWriters(dir, SELF.instanceId, "sess-m", Date.now()), []); +}); + +test("own marker file never contests itself", () => { + const dir = writerMarkerDir(freshDir("bili-3923-self-")); + const w = createServedMarkerWriter(SELF, dir, () => {}); + w.noteServed("sess-s"); + w.__flushForTest(); + assert.deepEqual(findContestedWriters(dir, SELF.instanceId, "sess-s", Date.now()), [], "self-instance excluded from its own scan"); + w.dispose(); +}); + +test("reapStaleWriterMarkers removes dead+stale files, keeps live and recent ones", () => { + const dir = writerMarkerDir(freshDir("bili-3923-reap-")); + const deadFile = writePeerMarker(dir, { ...PEER, instanceId: "peer-dead", pid: DEAD_PID }, {}); + backdate(deadFile, 2 * 60 * 60_000); + const liveFile = writePeerMarker(dir, { ...PEER, instanceId: "peer-live", pid: process.pid }, {}); + backdate(liveFile, 2 * 60 * 60_000); + reapStaleWriterMarkers(dir, Date.now()); + assert.equal(fs.existsSync(deadFile), false, "provably-dead + beyond reap horizon → removed"); + assert.equal(fs.existsSync(liveFile), true, "live pid → kept regardless of age"); +}); diff --git a/tests/side-request-isolation.test.ts b/tests/side-request-isolation.test.ts index a128ae3235..d27a087328 100644 --- a/tests/side-request-isolation.test.ts +++ b/tests/side-request-isolation.test.ts @@ -3,7 +3,7 @@ import assert from "node:assert/strict"; import http from "node:http"; import { once } from "node:events"; import { defaultConfig, createInitialState, defaultCountTokens } from "acp-kernel"; -import { startServer, isSideRequest, outputBudgetField, restoreOutputBudget, sideRequestGuard, resolveKnownOutputCeiling, _resetNoOutputCeilingWarningsForTest, OUTPUT_RESTORE_ABSOLUTE_FLOOR } from "../src/server.ts"; +import { startServer, isSideRequest, outputBudgetField, restoreOutputBudget, sideRequestGuard, resolveKnownOutputCeiling, _resetNoOutputCeilingWarningsForTest, _resetDshMidBandWarningsForTest, OUTPUT_RESTORE_ABSOLUTE_FLOOR } from "../src/server.ts"; import type { ProxyOptions } from "../src/config.ts"; import { recordPluginRuntimeInfo, _resetPluginStateForTest } from "../src/plugin.ts"; import { estimateRawBodyTokens } from "../src/preflight.ts"; @@ -349,6 +349,115 @@ test("restoreOutputBudget: #3071 leaves the ceiling-known behavior unchanged (re assert.equal(cappedSmall.max_tokens, 4096, "known ceiling below the absolute floor is honored, not over-floored (#3071 scope)"); }); +test("restoreOutputBudget: dsh-lane mid-band collapse is rescued when a ceiling is known (#3438)", () => { + _resetDshMidBandWarningsForTest(); + const lines: string[] = []; + const log = (_lvl: string, msg: string): void => { lines.push(msg); }; + const tools = [{ type: "function", function: { name: "bash" } }]; + const M = "cn:deepseek-v4.1-flash"; + const CEIL = 393_216; + // Incident replay (v0.1.197, DSH desktop, responses wire): the session ran + // healthy at the declared cap, then the client's raw-history estimate decayed + // THROUGH the mid-band toward zero. Before #3438 every >200 value passed + // verbatim AND re-seeded the high-water; the <=200 trigger was the only + // rescue entry and all three observed collapses (2260/872/204) slipped past it. + const s = metaSession("dsh-incident"); + const healthy = { max_output_tokens: CEIL, tools, model: M }; + restoreOutputBudget(healthy, s, noopLog, CEIL, true); + assert.equal(healthy.max_output_tokens, CEIL, "at-ceiling value passes untouched"); + assert.equal(s.metadata.outputBudgetHighWater, CEIL, "the healthy value seeds the high-water"); + for (const v of [2260, 872, 204]) { + const body = { max_output_tokens: v, tools, model: M }; + restoreOutputBudget(body, s, log, CEIL, true); + assert.equal(body.max_output_tokens, CEIL, `${v} restored to the known ceiling (#3438)`); + assert.equal(s.metadata.outputBudgetHighWater, CEIL, "a collapsed value must NOT re-seed the high-water"); + } + assert.ok(lines.some((l) => l.includes("output budget restored 2260 -> 393216")), "the restore is logged"); + assert.ok(lines.some((l) => l.includes("#3438: dsh-lane mid-band collapse")), "the mid-band origin names itself in the log"); + // A mid-band value far above the starve gate is the same signature. + const deep = { max_output_tokens: 25_484, tools, model: M }; + restoreOutputBudget(deep, s, log, CEIL, true); + assert.equal(deep.max_output_tokens, CEIL, "25484 (< ceiling) is a collapse too, not a healthy small budget"); + // The over-restoration surface is deliberate on this lane: even a plausible + // deliberate-looking 32K budget is raised to the declared cap — on the dsh + // contract the client's number is always its estimator, never intent. + const mid = { max_output_tokens: 32_768, tools, model: M }; + restoreOutputBudget(mid, s, log, CEIL, true); + assert.equal(mid.max_output_tokens, CEIL, "32768 < 393216 is still below the declared cap -> raised"); + // A side request (no tools) on the same lane is NEVER touched — restoring it + // would break its side status — and it must not seed the high-water either. + const side = { max_output_tokens: 1024, model: M }; + restoreOutputBudget(side, s, log, CEIL, true); + assert.equal(side.max_output_tokens, 1024, "tool-less mid-band passes verbatim"); + assert.equal(s.metadata.outputBudgetHighWater, CEIL, "the side request did not poison the high-water"); + // Below the starve gate the pre-existing rescue entry point still applies. + const starved = { max_output_tokens: 150, tools, model: M }; + restoreOutputBudget(starved, s, log, CEIL, true); + assert.equal(starved.max_output_tokens, CEIL, "the <=200 entry point still rescues on the dsh lane"); +}); + +test("restoreOutputBudget: non-dsh lanes keep trusting a below-ceiling budget (#3438 scope guard)", () => { + const tools = [{ name: "t" }]; + const s = metaSession("non-dsh"); + const body = { max_tokens: 8000, tools, model: "prov/model" }; + restoreOutputBudget(body, s, noopLog, 393_216); + assert.equal(body.max_tokens, 8000, "no lane flag: a below-ceiling value is intent, forwarded verbatim"); + assert.equal(s.metadata.outputBudgetHighWater, 8000, "recorded as the session's own proven budget"); + const explicit = { max_tokens: 4096, tools, model: "prov/model" }; + restoreOutputBudget(explicit, s, noopLog, 393_216, false); + assert.equal(explicit.max_tokens, 4096, "an explicit false behaves like absent"); + // The <=200 starve restores to the ceiling regardless of the lane flag. + const starved = { max_tokens: 16, tools, model: "prov/model" }; + restoreOutputBudget(starved, s, noopLog, 393_216, true); + assert.equal(starved.max_tokens, 393_216, "starve target logic is untouched by the lane flag"); +}); + +test("restoreOutputBudget: dsh-lane mid-band WITHOUT a ceiling forwards verbatim and warns once per session+model (#3438)", () => { + _resetDshMidBandWarningsForTest(); + const warns: string[] = []; + const log = (lvl: string, msg: string): void => { if (lvl === "warn") warns.push(msg); }; + const tools = [{ name: "t" }]; + const A = "prov/unknown-model"; + const s = metaSession("dsh-nocap"); + const body = { max_tokens: 2260, tools, model: A }; + restoreOutputBudget(body, s, log, undefined, true); + assert.equal(body.max_tokens, 2260, "no ceiling: bili cannot judge collapse from intent, forwards verbatim"); + assert.equal(s.metadata.outputBudgetHighWater, 2260, "no-ceiling learning is unchanged (trusted verbatim)"); + let hits = warns.filter((l) => l.includes("cannot tell a raw-history collapse from deliberate intent")); + assert.equal(hits.length, 1, "one warn for the unrescuable mid-band"); + assert.match(hits[0] ?? "", /model=prov\/unknown-model/, "names the model"); + assert.match(hits[0] ?? "", /providers\.\.models/, "points at the config escape hatch"); + // Deduped per session+model. + restoreOutputBudget({ max_tokens: 872, tools, model: A }, s, log, undefined, true); + assert.equal(warns.filter((l) => l.includes("deliberate intent")).length, 1, "deduped per session+model"); + // A different model in the same session warns of its own. + restoreOutputBudget({ max_tokens: 872, tools, model: "prov/other" }, s, log, undefined, true); + assert.equal(warns.filter((l) => l.includes("deliberate intent")).length, 2, "per session+model, not per session"); + // Tool-less mid-band: not a main turn, stays silent. + const side = { max_tokens: 1024, model: A }; + restoreOutputBudget(side, s, log, undefined, true); + assert.equal(side.max_tokens, 1024, "side request passes verbatim"); + assert.equal(warns.filter((l) => l.includes("deliberate intent")).length, 2, "the side request stays silent"); + // Non-dsh lane: silent passthrough, no new warn. + const other = { max_tokens: 2260, tools, model: A }; + restoreOutputBudget(other, metaSession("plain"), log); + assert.equal(other.max_tokens, 2260, "no lane flag: plain passthrough"); + assert.equal(warns.filter((l) => l.includes("deliberate intent")).length, 2, "no lane flag -> no new warn"); +}); + +test("restoreOutputBudget: the <=200 starve log keeps its original wording, the mid-band names #3438", () => { + const lines: string[] = []; + const log = (_lvl: string, msg: string): void => { lines.push(msg); }; + const tools = [{ name: "t" }]; + const s = metaSession("wording"); + const starved = { max_tokens: 16, tools, model: "m/x" }; + restoreOutputBudget(starved, s, log, 32_768, true); + assert.ok(lines.some((l) => l.includes("#546: client shrank it from its raw-history estimate")), "the starve wording is byte-identical to before"); + const mid = { max_tokens: 872, tools, model: "m/x" }; + restoreOutputBudget(mid, s, log, 32_768, true); + assert.ok(lines.some((l) => l.includes("#3438: dsh-lane mid-band collapse")), "the mid-band restore names itself"); +}); + test("resolveKnownOutputCeiling: runtime-info > launcher > declared > registry rank order (#1840)", () => { _resetPluginStateForTest(); setRegistryForTest({ "stepfun/step-5-preview": { limit: { context: 1_000_000, output: 1_000_000 } } }); @@ -817,6 +926,54 @@ test("e2e: poisoned high-water is floored by the known output ceiling when no ro } }); +test("e2e: dsh-lane mid-band budget is restored to the declared output ceiling on the wire (#3438)", async () => { + // Server-level wiring proof for the incident: a dsh plugin-stamped main turn + // whose raw-history estimate decayed into the mid-band (above the <=200 starve + // gate, below the declared ceiling) must leave the proxy at the ceiling. + // 16K stays well inside the default 200K rig window so the downstream clamp + // cannot touch the restored value. + const CEIL = 16_384; + const rig = await startRig({ routeModels: { [MODEL]: { output: CEIL } } }); + try { + const url = `http://127.0.0.1:${rig.proxyPort}/bili/http://127.0.0.1:${rig.upstreamPort}/v1/messages`; + const headers: Record = { + "content-type": "application/json", + "x-acp-session": SESSION, + "x-bili-plugin": "dsh", + "x-bili-plugin-conversation": SESSION, + }; + // Host-shaped tool names: a tool literally named "compress" is a LEAKED + // bili tool on this lane — #1897 strips it and demotes the request to + // side passthrough BEFORE restoreOutputBudget ever runs (#1897 gate). + const tools = [ + { name: "bash", description: "run a command", input_schema: { type: "object", properties: { command: { type: "string" } } } }, + { name: "read", description: "read a file", input_schema: { type: "object", properties: { path: { type: "string" } } } }, + ]; + // Healthy turn at the declared cap first (seeds the water mark, like the + // incident session before its estimate decayed). + const r1 = await fetch(url, { method: "POST", headers, body: JSON.stringify({ model: MODEL, max_tokens: CEIL, stream: true, tools, messages: mainConversation(8) }) }); + assert.equal(r1.status, 200); + await r1.text(); + assert.equal(rig.lastBody && rig.lastBody.max_tokens, CEIL, "at-ceiling value forwarded untouched"); + // The incident's three collapses, one by one through the real pipeline. + for (const v of [2260, 872, 204]) { + const r = await fetch(url, { method: "POST", headers, body: JSON.stringify({ model: MODEL, max_tokens: v, stream: true, tools, messages: mainConversation(9) }) }); + assert.equal(r.status, 200); + await r.text(); + assert.equal(rig.lastBody && rig.lastBody.max_tokens, CEIL, `${v} restored to the declared ceiling on the wire (#3438)`); + assert.equal(getSession(SESSION).metadata.outputBudgetHighWater, CEIL, "the collapsed value did not re-seed the high-water"); + } + // A non-dsh client on the SAME route keeps the old behavior: below-ceiling + // is intent there and forwards verbatim (the lane flag is the whole scope). + const rPlain = await fetch(url, { method: "POST", headers: { "content-type": "application/json", "x-acp-session": SESSION + "-plain" }, body: JSON.stringify({ model: MODEL, max_tokens: 2260, stream: true, tools, messages: mainConversation(8) }) }); + assert.equal(rPlain.status, 200); + await rPlain.text(); + assert.equal(rig.lastBody && rig.lastBody.max_tokens, 2260, "non-dsh lane: below-ceiling trusted verbatim (#3438 scope guard)"); + } finally { + await closeRig(rig); + } +}); + test("e2e: oversized side request is blocked locally (413), never reaches the upstream (#554)", async () => { const rig = await startRig({ modelContextLimit: 4_000, compressModelContextLimit: 4_000 }); try {