Repository navigation
[PR] fix: persist each committed preflight fold so a dead marathon leaves durable progress (#3978) - #4046
[PR] fix: persist each committed preflight fold so a dead marathon leaves durable progress (#3978)#4046ranxianglei wants to merge 18 commits into
Conversation
A hard process death mid-preflight (crash / OOM / restart on multi-million-token sessions) skipped every post-loop persistence path, so rounds already compressed lived only in memory; a client retry under a new session id found no parent state worth inheriting and re-compressed from zero — the permanent unavailability loop reported in #3978. commitFold now marks the session dirty as each fold lands; the persist debounce coalesces the calls into one write per quiet window, so durable progress accumulates regardless of how the request ends. A clean client abort already persists via the post-loop rebuild (src/server.ts:3554 runs before the :3607 fail-fast return); this closes the death-mid-request gap that path cannot cover. tests/issue3978-preflight-abort-persistence.test.ts pins it end-to-end: T1 guards the clean-abort ordering, T2 is the negative control (no fold committed => nothing fold-like persisted), T3 SIGKILLs a real child proxy mid-marathon and asserts a fresh instance's new-id retry inherits the persisted blocks instead of starting fresh. Without the fix the red run fails T3 exactly at the adopted-block count.
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.6193-pr.4046.48defc7This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr4046.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…3978) The 60s READY deadline lost the race on ubu22-1/win22-1 at head 9be2ec6: the child cold-transforms the whole src tree via tsx, which exceeds 60s under parallel-shard load. Raise the deadline to 180s, fail fast when the child exits before READY (diedP), and capture its stderr in-memory so a future crash surfaces in the assertion message instead of getting lost in interleaved shard logs.
…ts resolve (#3978) On CI the per-test tmp dir sits outside the repo tree, so Node finds no package.json walking up from the generated child script and defaults it to CJS. Its bare import of acp-kernel then resolves under the require condition, which acp-kernel's exports map does not define, killing the child at boot with ERR_PACKAGE_PATH_NOT_EXPORTED on every leg. Locally the tmp dir lives inside the repo (inheriting its type:module), which is why this only failed on CI. Writing a minimal {"type":"module"} package.json beside the child script pins the module format explicitly. Verified with an out-of-repo TMPDIR reproduction (red before, green after).
(ai: vllm-qwen/qwen3.8-27b) 查重:不重复——本票是 #3978 的 PR 票(持久化半)。同家族在途项:#3590(墙钟半,独立机制,零文本重叠已核)、#4049(master 既有 flake,与本 PR 无关)。仓库内无其他 open PR 触碰 commitFold/持久化路径。 标签:bug / scope/storage / p1(马拉松中断丢全部进度、超大上下文会话永久不可用——按 owner 2026-10-19 修订属 p1「不能用」类;无 kernel 改动故不打 scope/kernel;PR 本体不涉特定客户端,不打客户端标签) 初审进展(继续审查中):
后续:解冲突 + rebase 后定向测试 + 八问结论回复。 |
(#4046) resolve the single live-files.mjs conflict by keeping both new LIVE_FILES entries in alphabetical order (issue3978 before issue4341); src/preflight.ts auto-merged clean and the commitFold markDirty hunk is verified intact against master's changes.
… T3 child The blind sleep(1000) settle raced the store's trailing-debounce write under loaded CI runners: windows 22-1 at head 5722ac7 found NO parent record at all when the retry booted (T3 red; signature distinct from #4049's two #3769 flakes). Replace the fixed sleep with a wait on '[durable] N' — the child's own codec-aware store.loadSync disk read (envelopes are BILIENC1- framed at rest, so the test process cannot scan the file for the summary mark) — so the SIGKILL can never race an unwritten save. Failure messages now carry the child stderr tail and the store dir listing.
(ai: vllm-qwen/qwen3.8-27b) 本轮结论:T3 红根因定位并修复,分支已合并 master 并推送(head 12de55b)1. 独立复核 CI(逐份重读原 run 39491108431 的失败分片日志)
2. 根因判定(测试竞态,非产品缺陷)
3. 本轮改动(已推送,PR head = 12de55b)
结构化八问
行为变更披露(整 PR,重申)仅写频率:marathon 期间 store 每个 quiet window 收到一次 debounce save,而不只在 rebuild 点。无 wire 变更、无配置面、无持久化格式变更、无 kernel 变更。 Auto-merge gate(§7.4)评估条件全满足:单模块纯 fix、有回归测试、无 config/persistence/wire/kernel 变更、clean diff、引用 #3978(有意不关闭——wall-clock 半在 #3590)、不触碰 load-bearing infra。待办:新 CI 确认——T3 在其排到的 leg 绿;其余红必须全部可归因于 #4049 的两个 #3769 签名(A = tests/issue3769-pooled-bare-load.test.ts:115,B = tests/issue3769-upgrade-compat.test.ts:179)。 判定新 CI 确认上述后即可合入(合并动作请人工执行)。与 #3590 的协调说明仍然成立(零文本重叠,合入 master 后依旧)。另注:T3 在矩阵中只排一个 leg(LIVE_FILES 分片机制使然),CI 暴露面偏薄属既有设计,hermetic 且足够快,本轮不扩覆盖。 |
…4393 ci.yml already carried a concurrency block (workflow+event+ref groups, cancel-in-progress except schedule — #3772/#3331, owner-approved 2026-10-10) that implements exactly what #4393 intended: same-branch pushes cancel the branch's in-progress/queued main-CI run. #4393's premise ('the main CI did not [have a concurrency group]') missed that existing block and added a second top-level 'concurrency' key. A duplicate top-level YAML key makes the workflow unparseable: GitHub fails every ci.yml run instantly with zero jobs (master push 970bd4d at 08:32:25Z and PR pushes after 08:32Z all show the path-named '.github/workflows/ci.yml' instant-failure signature), killing the main CI leg repo-wide while the separate lane workflow files kept running. Remove #4393's 8-line block; the pre-existing concurrency semantics are unchanged, so same-ref supersede behavior #4393 set out to get still holds.
(ai: vllm-qwen/qwen3.8-27b) 状态更新:上一轮 CI 被批量取消(非测试失败)+ 发现并顺手修掉一个 master 级 CI 回归1. 上一轮 11 个 run 的去向head 12de55b 的 11 个 workflow 全部在 07:40:47Z 同一秒被批量 cancel(启动于 06:54Z,运行/排队 ~46 分钟)。不是测试红——是今天 CI 队列事故(458 个排队 run,#4393 的 commit message 里点名了这次 incident)期间的清理动作。该 head 上没有任何有效判读。 2. 重新触发时发现的 master 级回归(已修,独立标注 commit a46fd0d)把 master(仅 #4393 一个 CI-only commit)合入分支后推上去,主 CI run 秒失败、零 job,显示名退化成文件路径 根因:#4393 给 ci.yml 加了第二个顶层 处置:按几行级小修调度纪律①(CI 红的显而易见修复直接进当前 PR),在本分支删掉 #4393 的 8 行冗余块,其余不动——既有并发语义原样保留,same-ref supersede 行为照旧。其他在途 PR 若各自落地同样的 8 行删除属政策允许;本 PR 合入后 master 即恢复。 3. 当前状态
待办(唯一)等新 CI 判读:T3 在其排到的 leg 绿;其余红必须全部可归因于 #4049 的两个 #3769 签名。满足即可合入(合并动作请人工执行)。 |
…78 T3 child win22-1 @ a46fd0d: [durable] was observed yet the retry still started fresh (resume lines empty) — findResumeParent (#1486) matches on the parent's affinity-chain entry, which a fresh instance hydrates from the machine-shared state-dir snapshot (prefix-affinity.json), a SEPARATE artifact from the session record with its own 500ms-debounced single-shot persist whose write can lose a rename race or starve under event-loop pressure and never re-arms. The child now proves the FILE carries the t3978-p entry (re-arming the persist while it does not) and the test waits for that proof before SIGKILL; the final assertion also surfaces the child stderr tail.
… EBUSY observation Minor fix found while fixing this lane's CI (labeled per the small-fix rule): the #2771 corrupt-index proof spent the full ~9.6s retry ladder on EBUSY unlinking index.db (transient Windows AV/indexer lock, #2799 family) — extend the flat tail by two 3200ms rungs (~16s total) and align the budget unit test.
(ai: vllm-qwen/qwen3.8-27b) Fixed on the branch — pushed 2b8c26d (two test-only commits, zero product-code change). CI reruns on the new head. What the win22-1 shard @ a46fd0d actually showedTwo failures, different mechanisms: ① This PR's own T3, again — but a NEW failure mode. The Fix (test-side, same philosophy as last round — wait for the artifact itself, not a guessed settle time): the child now proves the file carries the ② Unrelated pre-existing flake (not caused by this PR): the #2771 corrupt-index proof spent the full ~9.6s Local validation (targeted, per local-testing discipline): 八问
判定:等新 head 的 CI 全腿绿即可合(合并由人执行)。若 win22-1 再红且签名仍是这两个之一,说明锁/竞争比预期更极端,下一轮我会转判定评论而不是继续加预算。 |
… union — keep issue3472 + issue3978 in sorted order)
(ai: vllm-qwen/qwen3.8-27b) 冲突已解决并推送(merge master → 结构化八问:
判定:冲突干净解决,可继续走 CI;待 #4603 合入后本 PR 的继承红自动消除,无需再动。 |
master window commits 293ae65/de5e03659 bumped kernel/package.json to 0.0.165 without regenerating the lockfiles; lock-sync re-resolution then rewrites the stale recorded versions (root 0.0.163, kernel 0.0.164) and fails. Pure recorded-version sync via npm install --package-lock-only; no dependency or semantic change.
What
#3978 reports a permanent unavailability loop on massively over-window sessions (~1.2M tokens vs a ~937k window): the preflight compression marathon outlives the client's hard wall timeout, the attempt dies, and every retry re-compresses from zero.
This PR implements the persistence half of the agreed fix direction (owner-approved in-thread): each committed preflight fold marks its session dirty as it lands (
markDirtyincommitFold,src/preflight.ts), so durable progress accumulates within the persist debounce no matter how the request ends. The wall-clock half is already implemented in open PR #3590 (client-patience wall budget) — hence this PR references #3978 without closing it; the two together close the loop.Diagnosis (verified at v0.1.202 and at master bc4a3f8)
src/server.ts:3554runPrepare → markDirty) now runs BEFORE the fail-fast return (:3607), so a CLEAN client disconnect already persists completed rounds (verified by md-trace instrumentation).Behavior change disclosure
None user-visible beyond write frequency: during a preflight marathon the store now receives one debounced save per quiet window instead of only at rebuild points. No wire change, no config surface added, no persistence-format change, no kernel change.
Verification
tests/issue3978-preflight-abort-persistence.test.ts(hermetic, ungated): T1 guards the clean-abort rebuild-before-failfast ordering; T2 negative control (no fold committed ⇒ nothing fold-like persisted, gate discipline Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617); T3 SIGKILLs a REAL child proxy mid-marathon and asserts a fresh instance's new-id retry inherits the persisted blocks instead of starting fresh.inherited 40 ref(s) up to m00040with no blocks clause); with the fix, 3/3 pass.Coordination notes
src/preflight.ts: open PR [PR] fix: stop preflight marathon rounds that would blow the client's patience wall clock (#3573) #3590 touches other regions (hunks at :146/:1580/:1628/:1797/:2715); this PR's hunks (:21 import, :2168–2174 commitFold tail) have zero textual overlap. If Root fix: server-side size-evidence ledger + preflight verdict state machine (fold-budget/eligibility B-class family) #3977's fold-budget integration branch lands first, expect a small semantic rebase here.Follow-up small fix in this branch (separately labelled commit a46fd0d)
While re-triggering CI after merging master, it turned out #4393 (merged to master 2026-10-23 08:32Z) added a SECOND top-level
concurrency:key to .github/workflows/ci.yml, which already carried one (workflow+event+ref groups, cancel-in-progress except schedule — #3772/#3331). A duplicate top-level YAML key makes the workflow unparseable: every ci.yml run fails instantly with zero jobs (master push 970bd4d shows the path-named instant-failure signature), killing the main CI leg repo-wide while the separate lane workflow files kept running. The pre-existing block already implements the same-branch-supersede behavior #4393 intended, so a46fd0d removes #4393's 8-line block and nothing else. Per the few-line CI-fix discipline it rides in this PR as its own labelled commit; other in-flight PRs landing the same 8-line removal independently is acceptable.Fixes #3978
(孤儿认领:标题本已指向 #3978,补上关闭关键字 —— 第三批清理)