Repository navigation
[PR] fix: bound .corrupt-* quarantine files and stop OOM-driven state loss (#5779) - #5781
Conversation
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.8390-pr.5781.5db9d55This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5781.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重:不重复。同族仅 #3157/#3163( 标签: 代码核对完成(GitHub PR #5781,单提交 @8389ea0,基于 master@0aace2f20),中期结论:
CI 现状:static-gates / docs-sync / test-import-graph / lock-sync 已绿,11 个 test 分片运行中。本地定向验证(新套件 + issue3157 套件)正在跑,全部完成后补八问结论。 |
…decode failure (#5779) Two defects behind user disks filling with ~0.6 GiB .corrupt-<ts> session state copies (273 copies / ~157 GiB reported in #5779): 1. Unbounded forensic retention: quarantineCorruptState renamed a failed envelope to <file>.corrupt-<ts> and nothing ever deleted those copies — every repeated failure of a hot session added another full-size copy. pruneCorruptSiblings now keeps at most CORRUPT_SIBLING_CAP (1) copy per base file; applied to both the session store and plugin-conversations.json. 2. Resource exhaustion misclassified as corruption: when decoding a large envelope ran out of memory (V8 RangeError), readEnvelopeFile quarantined the healthy file and destroyed its fold state; the eviction/cold-load loop then rebuilt and re-quarantined it every cycle. Allocation failures now leave the file in place for retry (error log, no quarantine). Genuine decode failures still quarantine exactly as before.
8389ea0 to
ec60e23
Compare
(ai: vllm-qwen/qwen3.8-27b) CI 自动修复车道(attempt 1/3)收工:零代码改动,重跑两个红分片后 run 全绿。 七问:
|
(ai: vllm-qwen/qwen3.8-8.3) 审查完成:全绿,建议合并rebase + CI:PR base 之后 master 前进了 56 个提交(零个触碰 persist.ts/plugin.ts),我已把分支 rebase 到当前 master(be611a8)——单提交 8389ea0 → ec60e23,diff 逐字节相同(3 文件 +272/-14),push 前用 force-with-lease 校验过远端未动。rebase 后本地 typecheck + build 干净。新 head 上 12 条 workflow 全绿:主 CI(含 11 个 test 分片)+ 7 条 e2e 车道 + codex overlay + build artifact,0 失败。 本地定向验证(改动域收窄跑,非全量):新套件 + issue3157-persist-durability + instance-governance = 26/26 通过。透明说明:我第一次直跑时 issue3157 套件有 2 例假失败——原因是本地直跑没带 CI 的存储基线 preload( 结构化八问
auto-merge 门评估(§7.4):回归测试复现原 bug ✓ / rebased head 全绿 ✓ / 无配置·持久化格式·wire·kernel 变更 ✓ / 纯 fix ✓ / diff 干净 ✓ / 引用 #5779 ✓ / 不触 load-bearing infra(update.ts、ref-id、安全面)✓ —— 形式上满足自动合并条件;合并动作归你(agent 不合并 PR)。 记录的边界(非缺陷):plugin-conversations.json 只加了保留上限、未加 OOM 分类——它是路由表(每条目几百字节)、仅启动时加载一次、不在热循环里,parse OOM 现实不可达;事故证据全部来自 0.6 GiB 级会话信封。合理取舍,已在中期回复说明。 判定:无拦截项,可合并 — #5781 |
|
🔶 需要决策 PR #5781 (head ec60e23) review surfaced one residual touching save-path arbitration semantics — not fixable in-place under review-lane rules (cross-file behavior), routed here for owner call. No code changed by this review. Residual: first save can clobber a present-but-unreadable envelope (sustained OOM only). Mechanism (legacy storage tier — the tier the new tests pin, and the reality of default plain-JSON envelopes per #1080):
Sub-point (couples with the above): the classifier misses a third resource-limit family. Empirical (Node v22.23.2): producing >536,870,880 decoded chars throws plain Options:
Cite: PR #5781 head ec60e23; gated area = src/persist.ts |
ranxianglei
left a comment
There was a problem hiding this comment.
(ai: vllm-qwen/qwen3.8-27b)
APPROVE. Both halves of #5779 fixed at the root; the p1 disk explosion is definitively closed. One non-blocking residual (write-path clobber under sustained OOM) is routed through a separate 🔶 需要决策 comment — pre-existing in spirit, net improvement either way.
Direction (issue audit). Repro facts hold against code: nothing ever pruned <base>.corrupt-* copies (dir walkers collect only *.json/.tmp-*, src/persist.ts:4102-4115), and readEnvelopeFile's inner catch (src/persist.ts:1213) quarantined every decode failure, allocation exhaustion included — destroying a healthy envelope's fold state on each eviction/cold-load round. The fix targets both mechanisms, and I verified only two .corrupt- producers exist (src/persist.ts:1256, src/plugin.ts:385) — both covered. Base current (parent == master tip); CI 30/30 green via mirror cache.
Regression audit — existing behaviors on touched paths:
| behavior | verdict |
|---|---|
| genuine decode failure → quarantine + fresh start | preserved — negative-direction test pins it (SyntaxError still quarantines, exactly one copy kept) |
.corrupt-* retention unbounded |
intended change, disclosed: cap 1 per base file; newest kept; just-created copy always survives clock skew; locked files skipped + retried next round. No other consumer of these files exists (forensics-only by design) |
| OOM decode → quarantine | intended change, disclosed: file left in place, retried next load. Bonus self-heal verified in code: when the save-path re-read succeeds after a load-time miss, staleDiskPayload counter arbitration converges in-memory back onto the disk record (src/persist.ts:2892) — fold state recovers transparently |
| warn-log format | appends removed N older forensic copies (cap 1) — disclosed; no machine consumers of this line |
| plugin-conversations.json corrupt handling | same quarantine + prune added; routing-table semantics unchanged |
readEnvelopeFile null contract |
unchanged across all four caller probe sites (loadEnvelope ×3 + namespaced pool) |
| compression modes / wire / kernel / config surface | persistence layer only — mode-independent; no kernel/ diff (kernel-guard green); no config/env additions (CORRUPT_SIBLING_CAP hardcoded, deliberate; config-surface-guard green) |
Test validity. Real teeth: pre-seeded historical sweep keeps exactly the just-created copy (size-checked, not a stale twin); 3-round accumulation pin (the incident itself); bidirectional gate per #3617 discipline (OOM positive: file intact + zero siblings + explanatory error log; SyntaxError negative: quarantined + exactly one copy); classifier truth table incl. stack-overflow exclusion; plugin-lane cap. Deterministic (no ports, tmpdir + cleanup, env restored in finally). One coverage note, non-blocking: the OR branch where the plaintext attempt is the OOM side isn't directly exercised — forcing a real OOM inside JSON.parse(bytes.toString("utf8")) hermetically would require pulling that parse behind an injectable seam; the classifier table plus the codec-side wiring cover the rest.
Local empirical probe (Node v22.23.2): Buffer.alloc(540MiB).fill('a').toString('utf8') throws plain Error: Cannot create a string longer than 0x1fffffe8 characters — NOT a RangeError. Consequence for the classifier family analyzed in the 🔶 需要决策 comment (sub-point).
中文八问
- 是什么问题:热会话每次解码失败都新增一份全尺寸
.corrupt-*取证副本(单用户 273 份 / ~157 GiB),且内存不足导致的解码失败被误判为损坏——健康信封被隔离、折叠状态被毁、驱逐/冷加载循环每轮重建再隔离。 - 什么原因导致的:① #3157 隔离机制只有"改名保留"没有保留上限,目录扫描只收
*.json/.tmp-*,取证副本永不清理;② 解码失败只有"成功/损坏"两态,V8 资源耗尽(RangeError)与真实损坏混为一谈。 - 影响范围与严重程度:p1,所有长期大会话用户。磁盘可被取证副本撑爆(不可用级);OOM 误隔离丢折叠状态、压缩从零重启、成本回涨。触发条件:大会话状态文件 + 反复加载失败。
- 怎么修复的:双管——每次隔离改名后按基文件剪枝到 1 份(新建副本恒留、时钟偏移安全、锁文件跳过下轮重试、永不抛错);解码错误增加"资源耗尽"分类,此类文件原地保留、下轮重试。瞬态 OOM 另有自愈:保存路径重读成功时计数器仲裁把内存态收敛回磁盘态,折叠状态透明恢复。
- 是否有回归问题:上表既有行为逐项审过——真实损坏仍隔离(测试钉住)、null 契约不变、日志仅追加、wire/kernel/配置面零触碰。残留一处非阻塞:sustained OOM 下首轮保存会覆盖盘上未读信封(
staleDiskPayload把"存在但不可读"与"不存在"混同)——该缝隙 PR 之前即存在于 rename 失败场景(Windows AV 锁),本 PR 只是多一个入口;已走 🔶 需要决策,不拦本 PR。 - 是否破坏缓存:不涉及——prefix cache/重试/去重键零触碰;瞬态 OOM 场景折叠状态经收敛路径完整恢复。
- 是否改了设计方向:不改——在 #3157 自愈架构内加保留边界与错误分类,无新数据流。
- 是否引入新配置:无——
CORRUPT_SIBLING_CAP = 1硬编码常量,PR 已声明刻意不加配置面。
判定:可合。 残留项与处置选项见 🔶 需要决策 评论(推荐方向 A)。
Review lane done: PR #5781 @ ec60e23 reviewed end-to-end — verdict APPROVE (posted as a comment-state review; GitHub blocks approving one's own PR), CI 30/30 green via mirror cache, regression-audited, tests have teeth. One non-blocking residual (sustained-OOM first save can clobber a present-but-unreadable envelope — pre-existing in spirit via the rename-failure path) routed through a 🔶 需要决策 comment + |
…5779) 062e1dd raised the sessions.residentMaxBytes default from 512 MiB to 4 GiB and updated tests/knobs-config-precedence.test.ts, but missed tests/issue4788-resident-minimal.test.ts:210 which pins the concrete default hard cap (= pool soft cap fallback): 536_870_912 -> 4_294_967_296. The invariant assertion (hard == soft) directly above is unchanged.
(ai: vllm-qwen/qwen3.8-27b) 本轮处理(head ec60e23 → 062e1dd → 27c3256):作者推了新提交(把常驻池默认预算从 512 MiB 提到 4 GiB),随后 3 个测试分片全红。两个独立原因:一个是真实疏漏(已修并推送),一个是 Windows runner 的 Python 子进程超时(重跑即绿)。当前全套绿。 1. 是什么问题:新提交后 3 个测试分片同时变红(windows-22-s5 / ubuntu-22-s2 / ubuntu-24-s2);修完后又有 1 个 Windows 分片出现 Python 子进程超时。 2. 什么原因导致的:
3. 影响范围与严重程度:(a) 仅影响本 PR 的 CI 门禁,零运行时行为影响(该钉只是测试对默认常量的断言);(b) 瞬态,重跑自愈。两者都不影响用户。 4. 怎么修复的:
5. 是否有回归问题:无。我的改动是单测试文件一行字面量更新,零运行时代码变更。默认值 512MiB→4GiB 本身是作者的有意变更(commit message 与 CONFIGURATION.md / LIFECYCLE.md 文档均已披露),非我引入;我只对齐了该变更漏掉的测试。 6. 是否破坏缓存:不涉及。 7. 是否改了设计方向:无。 8. 是否引入了新的配置:我未引入。(作者把既有旋钮 现状与判定:run 42244117307(head 27c3256)26 个 job 全绿(red-signal 按设计跳过)。PR 可进入人工 review/合并。兜底:若 hermes python3 超时在其他干净 runner 上复现,按 #4599 先例升级(候选后续项: |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.1057-master.f66051d # this exact build
npm install -g billion-context@master # always the newest merged build
|
What
#5779 report: the sessions tree accumulates
.corrupt-<timestamp>copies of session state files (~0.61 GiB each, hundreds per tree — 273 copies / ~157 GiB on one user's disk), generated at up to ~36/hour.Two defects, both fixed here:
<file>.corrupt-<ts>([bug] 会话状态写入缺 fsync:内核崩溃后留下 size 正常但 VDL=0 的全零状态文件,loadSync 永久抛错 → 502 acp-proxy failure 且不自愈(含 7 处 tmp+rename 写路径盘点) #3157 self-heal) and nothing ever deletes those copies — directory walkers only collect*.json/.tmp-*. Every repeated failure of a hot session added another full-size copy forever.RangeError("Array buffer allocation failed" / "JavaScript heap out of memory").readEnvelopeFiletreated that as corruption: it quarantined a healthy file, destroying its fold state; the resident-pool eviction/cold-load cycle then rebuilt the session from scratch and re-quarantined it every round — the loop that produced ~0.6 GiB of forensic copies per failure.Plus one owner-directed default change found while analyzing the loop (in-thread decision on #5779):
Fix
pruneCorruptSiblings()(src/persist.ts): after each quarantine rename, keep at mostCORRUPT_SIBLING_CAP = 1copy per base file (newest survives; the just-created copy always survives even under clock skew). Applied to both the session store (quarantineCorruptState) andplugin-conversations.json(src/plugin.ts, same mechanism). Best-effort: locked files are skipped and retried on the next quarantine; never throws.isAllocationFailureError(): allocation-failure RangeErrors during decode now leave the file in place for the next load (error log explains the verdict); genuine decode failures still quarantine exactly as before. Both errors are classified — an OOM in the plaintext attempt surfaces only as the codec's format error in the inner catch.sessionResidentMaxBytes()default (src/knobs.ts):512 * 1024 * 1024→4 * 1024 * 1024 * 1024. The per-session hard cap (sessions.hardMaxBytes) defaults to the pool soft cap, so it follows automatically. Docs kept in sync through the yaml seed +gen-config-docsgenerator (CONFIGURATION.md en/zh, LIFECYCLE.md, website/config-reference/process.yaml); test asserts updated.Behavior changes (old → new)
.corrupt-*retention per base fileremoved N older forensic copies (cap 1)when the sweep runssessions.residentMaxBytes)sessions.residentMaxBytes/BILI_SESSION_RESIDENT_MAX_BYTESNo NEW config surface added: the quota constants are hardcoded by design (forensics need at most the latest copy), and item 3 changes the DEFAULT of an existing knob only — env/file overrides are unchanged.
Tests
New
tests/issue5779-corrupt-quarantine-cap.test.ts(pins the legacy-file tier like the CI non-storage baseline): pre-seeded old siblings pruned to cap, repeated quarantine rounds never accumulate, OOM decode leaves the file intact with zero siblings (gate positive), genuine SyntaxError still quarantines (gate negative), classifier unit table, plugin-conversations cap.tests/knobs-config-precedence.test.tsasserts the new 4 GiB default (both precedence tiers). Pre-existingissue3157-persist-durability,instance-governance,issue3918-resident-eviction-refusal,bounded-storage,issue3399-cross-route-loadsuites green alongside under the CI preload chain.Closes #5779