Repository navigation
fix: fsync state writers before rename + self-heal corrupt session state (#3157) - #3163
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.3622-pr.3163.cb91fc8This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3163.tgz
npm install -g packageInstall instructions are refreshed on each push. |
tmp+rename guarantees rename atomicity but NOT content durability: the bytes sit in the page cache until the OS flushes them, so a power loss / kernel panic between write and rename leaves a size-correct, VDL=0 all-zero file on disk (NTFS journals metadata, not user data). That is exactly the Windows BugCheck 0x1A corruption reported in #3157. Add atomicWriteDurable (open/write/fsync/close) used by flushSync's canonical + spill writes, and fsyncFile (re-open + fsync, valid because fsync is inode-level) called after writeInner's async writeFile and before its fence-check+rename commit block. Bump acp-kernel to 0.0.123 per the Kernel Boundary version rule.
… state (#3157) Two-part fix for the permanent-502 class reported in #3157. (1) Durability: every tmp+rename STATE writer now fsyncs before the rename publishes the name, mirroring the repo's existing atomicWriteConfig/credential-store idiom (file-level fsync, no dir fsync). Covers the content-store sidecar + saveContentStore, launcher sqlite-origin/overlay/text-file, affinity snapshot, plugin-conversations, kimi registry, apig-resign pending-refusals, and dsh manifest. Ephemeral per-launch os.tmpdir files (claude --settings, opencode overlay json) are intentionally left as-is: they are not persistent user state. (2) Self-heal: readEnvelopeFile no longer throws forever on an undecodable/malformed session file. It quarantines it to <name>.corrupt-<ts> beside the original (preserved for forensics), logs one warn, and starts the session fresh — aligning with loadConversations' existing corrupt-file handling for plugin-conversations.json. Wrong-id and transient-read-error (EPERM) paths are unchanged. Behavior change (old -> new): a corrupt session state file used to make loadSync throw 'refusing an empty resume' on EVERY load -> permanent lane 502 with no recovery command. It is now isolated once + warned + the session resumes empty. The corrupt bytes are still never loaded as valid. Tests: new tests/issue3157-persist-durability.test.ts pins the zero-fill -> quarantine -> fresh-start path; bounded-storage #3/#6 updated to the no-longer-fail-closed contract.
…via #2970 Review fix on the #3157 PR branch (stale-base semantic collision, no textual conflict): this PR branched when master's kernel was 0.0.122 and bumped to 0.0.123, but master independently consumed 0.0.123 (#2970 foreign-name compress). Per the repo convention recorded in 9656fc0 ("bump past master's number so each changeset keeps its own marker"), the fsync changeset gets its own marker: 0.0.124. Also syncs root package-lock.json packages[kernel], which the original bump left stale at 0.0.122 (same convention: kernel/package.json + root lockfile packages[kernel] move together; kernel/package-lock.json stays as-is).
Review fix on the #3157 PR branch: the new durable-write sites used a single fs.writeSync(fd, data) whose return value was unchecked. writeSync may return fewer bytes than requested (short write), which would publish a silently truncated state file — the same corruption class this PR fixes. The repo's own idiom (external-summary-credentials.ts set(), web/api.ts atomicWriteConfig, instance.ts atomicWriteJson) writes through writeFileSync(fd), which loops to a full write before the fsync. Switch all eight new sites to it; this also removes the string/Buffer narrowing branches writeFileSync makes unnecessary.
Review fix on the #3157 PR branch: the new durability test only exercised the decode-failure half of readEnvelopeFile's self-heal (zero-filled bytes). The sibling branch — file decodes as JSON but fails isValidRecord — had no dedicated case. Add one with the same assertions (no throw, null result, exactly-one quarantine, warn naming the session).
27b72fa to
04ae5ef
Compare
…M on Windows (#3157) Review fix for the original PR's fsyncFile: it re-opened the written file with flag "r" before fsyncing. On POSIX that works (fsync is inode-level), but on Windows the flush goes through the handle (FlushFileBuffers) and a read-only handle fails with EPERM — breaking every async state write (writeNow/scheduleSave) on that platform. Caught by the static-gates lane's Windows kernel run: 23 persist.test.ts failures, all 'EPERM: operation not permitted, fsync' at StateStore.fsyncFile, while the sync path (fd opened "w") stayed green. Open with "r+" (write access, no truncate) so the flush succeeds on both platforms. Rides the 0.0.124 marker — not yet released.
(ai: vllm-qwen/qwen3.8-27b) 独立审阅完成(未照抄作者/issue 结论:逐文件读完 12 个文件完整 diff,对照 origin/master 核查既有行为,关键论断均本地实测)。总判定:方向正确、修法成立;review 中抓出并修掉 3 处问题(含 1 处 PR 自身引入的 Windows 回归);剩余 CI 红腿与 PR 无关,已单独立项。建议人工合并(触碰 kernel/,自动合并门禁禁止 agent 合并),等 master 侧测试缺陷修复后重跑全绿再合最稳。 Review 中发现并已修(4 个标注 commit,push 到 PR 分支,当前 head f0a9256)
本地定向验证:kernel 套件 1293/1293、issue3157 持久化测试、bounded-storage、typecheck 全绿;全量以 CI 为准。 八问
遗留事项
收工判定:代码可合(人工合并,kernel 门禁);建议等 #3170 修完、本 PR 重跑 CI 全绿后合并。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.429-master.757b551 # this exact build
npm install -g billion-context@master # always the newest merged build
|
What & why
Fixes the permanent-502 class reported in #3157: after a kernel crash / power loss, bili's session state file could be left size-correct but VDL=0 all-zero (NTFS journals metadata, not user data), and the read path then threw on every load → the whole lane 502'd until a human deleted the file.
Two-part fix:
① Durability — fsync before rename (root cause)
tmp+rename guarantees the rename is atomic, not that the content reached disk. The bytes sit in the page cache until the OS flushes them; a crash between write and rename publishes a name pointing at non-durable data. Every tmp+rename state writer now does
write → fsync(fd) → close → rename, mirroring the repo's existingatomicWriteConfig/ credential-store idiom (file-level fsync; no dir fsync — NTFS/journaled ext4 cover rename durability via the FS journal).Paths fixed (the 7 from the issue + adjacent state writers):
StateStore.flushSync(canonical + spill) — the direct victimStateStore.writeInner(async canonical + spill)saveContentStore(src/persist.ts)src/launcher.ts)src/affinity-persist.ts)src/plugin.ts)src/plugin-install.ts)src/apig-resign.ts)src/dsh-channel.ts)Ephemeral per-launch
os.tmpdir()files (claude--settings, opencode overlay json) are intentionally left as-is — they are not persistent user state.② Self-heal — isolate undecodable state instead of throwing forever
readEnvelopeFileno longer throwsrefusing an empty resumeon every load when a session file can't be decoded or fails validation. It now quarantines the file to<name>.corrupt-<ts>beside the original (preserved for forensics), logs one warn, and starts the session fresh — aligning withloadConversations' existing corrupt-file handling forplugin-conversations.json. Wrong-id and transient-read-error (EPERM) paths are unchanged.A corrupt session state file used to make
loadSyncthrow on every load → permanent lane 502, and no command could isolate the file. It is now isolated once + warned + the session resumes empty. The corrupt bytes are still never loaded as valid — the safety property that motivated "refusing an empty resume" is preserved; only the recovery mechanism changed from fail-forever to fail-once-with-recovery.Out of scope (deliberate, documented in code)
src/storage/engine-bootstrap.tscooldown marker + sqlite artifact publish: transient fail-open cache timer / sha512-verified one-shot copy — not persistent user state in the harmful-corruption class.src/update.tsinstall-backoff file: human-gated (§7.4), already fail-safe in the retry direction.Tests
tests/issue3157-persist-durability.test.ts: zero-fill a live state file → assertloadSyncdoes NOT throw, returns null, the file is quarantined to exactly one.corrupt-*sibling of the same size, a second load doesn't re-quarantine, and a warn fires.tests/persist.test.ts: new "every commit fsyncs the temp before rename" test (spiesfs.fsyncSync); adjusted the flushSync-retry test's spy fromwriteFileSynctoopenSync.tests/bounded-storage.test.tsUse kernel renderTags:"text-only" + forbid tag echoing in prompt #3/fix: post-merge review fixes + per-route context window + XDG config #6 updated to the no-longer-fail-closed contract.kernel-change
kernel-change: #3157 — StateStore.flushSync/writeInner tmp+rename wrote state without fsyncing, so a crash between write and rename left a VDL=0 zero file that permanently broke resume. Add atomicWriteDurable/fsyncFile so data is durable before rename publishes. Bump acp-kernel 0.0.122 → 0.0.123.Closes #3157