Skip to content

fix: fsync state writers before rename + self-heal corrupt session state (#3157) - #3163

Merged
ranxianglei merged 6 commits into
masterfrom
2026-10-14_persist-fsync-self-heal
Oct 14, 2026
Merged

ranxianglei merged 6 commits into
masterfrom
2026-10-14_persist-fsync-self-heal

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What & why

Fixes the permanent-502 class reported in #3157: after a kernel crash / power loss, bili's session state file could be left size-correct but VDL=0 all-zero (NTFS journals metadata, not user data), and the read path then threw on every load → the whole lane 502'd until a human deleted the file.

Two-part fix:

① Durability — fsync before rename (root cause)

tmp+rename guarantees the rename is atomic, not that the content reached disk. The bytes sit in the page cache until the OS flushes them; a crash between write and rename publishes a name pointing at non-durable data. Every tmp+rename state writer now does write → fsync(fd) → close → rename, mirroring the repo's existing atomicWriteConfig / credential-store idiom (file-level fsync; no dir fsync — NTFS/journaled ext4 cover rename durability via the FS journal).

Paths fixed (the 7 from the issue + adjacent state writers):

  • kernel StateStore.flushSync (canonical + spill) — the direct victim
  • kernel StateStore.writeInner (async canonical + spill)
  • host content-store sidecar + saveContentStore (src/persist.ts)
  • launcher sqlite-origin / overlay / text-file (src/launcher.ts)
  • affinity snapshot (src/affinity-persist.ts)
  • plugin-conversations (src/plugin.ts)
  • kimi registry (src/plugin-install.ts)
  • apig-resign pending-refusals (src/apig-resign.ts)
  • dsh manifest (src/dsh-channel.ts)

Ephemeral per-launch os.tmpdir() files (claude --settings, opencode overlay json) are intentionally left as-is — they are not persistent user state.

② Self-heal — isolate undecodable state instead of throwing forever

readEnvelopeFile no longer throws refusing an empty resume on every load when a session file can't be decoded or fails validation. It now quarantines the file to <name>.corrupt-<ts> beside the original (preserved for forensics), logs one warn, and starts the session fresh — aligning with loadConversations' existing corrupt-file handling for plugin-conversations.json. Wrong-id and transient-read-error (EPERM) paths are unchanged.

⚠️ Behavior change (old → new)

A corrupt session state file used to make loadSync throw on every load → permanent lane 502, and no command could isolate the file. It is now isolated once + warned + the session resumes empty. The corrupt bytes are still never loaded as valid — the safety property that motivated "refusing an empty resume" is preserved; only the recovery mechanism changed from fail-forever to fail-once-with-recovery.

Out of scope (deliberate, documented in code)

  • src/storage/engine-bootstrap.ts cooldown marker + sqlite artifact publish: transient fail-open cache timer / sha512-verified one-shot copy — not persistent user state in the harmful-corruption class.
  • src/update.ts install-backoff file: human-gated (§7.4), already fail-safe in the retry direction.

Tests

  • New tests/issue3157-persist-durability.test.ts: zero-fill a live state file → assert loadSync does NOT throw, returns null, the file is quarantined to exactly one .corrupt-* sibling of the same size, a second load doesn't re-quarantine, and a warn fires.
  • Kernel tests/persist.test.ts: new "every commit fsyncs the temp before rename" test (spies fs.fsyncSync); adjusted the flushSync-retry test's spy from writeFileSync to openSync.
  • tests/bounded-storage.test.ts Use kernel renderTags:"text-only" + forbid tag echoing in prompt #3/fix: post-merge review fixes + per-route context window + XDG config #6 updated to the no-longer-fail-closed contract.
  • Local targeted runs green: kernel persist (45), bounded-storage (12), issue3157 durability (1), persistence-core batch (112 pass / 1 skip), leaf-module batch (143). Full suite = remote CI.

kernel-change

kernel-change: #3157 — StateStore.flushSync/writeInner tmp+rename wrote state without fsyncing, so a crash between write and rename left a VDL=0 zero file that permanently broke resume. Add atomicWriteDurable/fsyncFile so data is durable before rename publishes. Bump acp-kernel 0.0.122 → 0.0.123.

Closes #3157

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-14_persist-fsync-self-heal (f0a9256)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3622-pr.3163.cb91fc8

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3163.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

收到,开始独立审阅 #3163(fsync 持久化 + 损坏状态自愈)。PR 当前 mergeable、base behind master,我会先 rebase 核对再出八问结论。初步范围:12 个文件(kernel StateStore 1 处 + host 状态写入器 8 处 + 测试 3 个)。

awork added 5 commits October 14, 2026 17:56
tmp+rename guarantees rename atomicity but NOT content durability: the bytes sit in the page cache until the OS flushes them, so a power loss / kernel panic between write and rename leaves a size-correct, VDL=0 all-zero file on disk (NTFS journals metadata, not user data). That is exactly the Windows BugCheck 0x1A corruption reported in #3157.

Add atomicWriteDurable (open/write/fsync/close) used by flushSync's canonical + spill writes, and fsyncFile (re-open + fsync, valid because fsync is inode-level) called after writeInner's async writeFile and before its fence-check+rename commit block. Bump acp-kernel to 0.0.123 per the Kernel Boundary version rule.
… state (#3157)

Two-part fix for the permanent-502 class reported in #3157.

(1) Durability: every tmp+rename STATE writer now fsyncs before the rename publishes the name, mirroring the repo's existing atomicWriteConfig/credential-store idiom (file-level fsync, no dir fsync). Covers the content-store sidecar + saveContentStore, launcher sqlite-origin/overlay/text-file, affinity snapshot, plugin-conversations, kimi registry, apig-resign pending-refusals, and dsh manifest. Ephemeral per-launch os.tmpdir files (claude --settings, opencode overlay json) are intentionally left as-is: they are not persistent user state.

(2) Self-heal: readEnvelopeFile no longer throws forever on an undecodable/malformed session file. It quarantines it to <name>.corrupt-<ts> beside the original (preserved for forensics), logs one warn, and starts the session fresh — aligning with loadConversations' existing corrupt-file handling for plugin-conversations.json. Wrong-id and transient-read-error (EPERM) paths are unchanged.

Behavior change (old -> new): a corrupt session state file used to make loadSync throw 'refusing an empty resume' on EVERY load -> permanent lane 502 with no recovery command. It is now isolated once + warned + the session resumes empty. The corrupt bytes are still never loaded as valid.

Tests: new tests/issue3157-persist-durability.test.ts pins the zero-fill -> quarantine -> fresh-start path; bounded-storage #3/#6 updated to the no-longer-fail-closed contract.
…via #2970

Review fix on the #3157 PR branch (stale-base semantic collision, no textual
conflict): this PR branched when master's kernel was 0.0.122 and bumped to
0.0.123, but master independently consumed 0.0.123 (#2970 foreign-name
compress). Per the repo convention recorded in 9656fc0 ("bump past
master's number so each changeset keeps its own marker"), the fsync
changeset gets its own marker: 0.0.124.

Also syncs root package-lock.json packages[kernel], which the original
bump left stale at 0.0.122 (same convention: kernel/package.json + root
lockfile packages[kernel] move together; kernel/package-lock.json stays
as-is).
Review fix on the #3157 PR branch: the new durable-write sites used a single
fs.writeSync(fd, data) whose return value was unchecked. writeSync may return
fewer bytes than requested (short write), which would publish a silently
truncated state file — the same corruption class this PR fixes. The repo's
own idiom (external-summary-credentials.ts set(), web/api.ts atomicWriteConfig,
instance.ts atomicWriteJson) writes through writeFileSync(fd), which loops to
a full write before the fsync. Switch all eight new sites to it; this also
removes the string/Buffer narrowing branches writeFileSync makes unnecessary.
Review fix on the #3157 PR branch: the new durability test only exercised
the decode-failure half of readEnvelopeFile's self-heal (zero-filled bytes).
The sibling branch — file decodes as JSON but fails isValidRecord — had no
dedicated case. Add one with the same assertions (no throw, null result,
exactly-one quarantine, warn naming the session).
@ranxianglei
ranxianglei force-pushed the 2026-10-14_persist-fsync-self-heal branch from 27b72fa to 04ae5ef Compare October 14, 2026 10:16
…M on Windows (#3157)

Review fix for the original PR's fsyncFile: it re-opened the written file
with flag "r" before fsyncing. On POSIX that works (fsync is inode-level),
but on Windows the flush goes through the handle (FlushFileBuffers) and a
read-only handle fails with EPERM — breaking every async state write
(writeNow/scheduleSave) on that platform. Caught by the static-gates lane's
Windows kernel run: 23 persist.test.ts failures, all
'EPERM: operation not permitted, fsync' at StateStore.fsyncFile, while the
sync path (fd opened "w") stayed green.

Open with "r+" (write access, no truncate) so the flush succeeds on both
platforms. Rides the 0.0.124 marker — not yet released.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

独立审阅完成(未照抄作者/issue 结论:逐文件读完 12 个文件完整 diff,对照 origin/master 核查既有行为,关键论断均本地实测)。总判定:方向正确、修法成立;review 中抓出并修掉 3 处问题(含 1 处 PR 自身引入的 Windows 回归);剩余 CI 红腿与 PR 无关,已单独立项。建议人工合并(触碰 kernel/,自动合并门禁禁止 agent 合并),等 master 侧测试缺陷修复后重跑全绿再合最稳。

Review 中发现并已修(4 个标注 commit,push 到 PR 分支,当前 head f0a9256)

  1. Windows 回归(本次审阅最重的发现):PR 新增的 fsyncFile() 用只读句柄 "r" reopen 后 fsync——Windows 上 FlushFileBuffers 要求句柄带写权限,否则 EPERM。后果:所有异步状态写入(writeNow/scheduleSave,即主热路径)在 Windows 上每次都会失败并降级 spill。linux 和本机测不出来(POSIX 允许任意 fd fsync),是 rebase 拉进新版 CI 后 static-gates 的 windows kernel lane 抓到的(23 fail)。已改 "r+" 并重写注释(原注释声称 fsync 与 fd 无关,在 Windows 上不成立)。修复后 windows static-gates 两条腿已绿。
  2. 版本号冲突 + lockfile 漏同步:master 已被 [Bug] MCP compress 历史调用未被清理,长会话累积大量旧摘要与回执 #2970 占用 acp-kernel 0.0.123,PR 的 0.0.122→0.0.123 会撞版本标记(同文本 hunk 能自动合并但破坏标记纪律)。按仓库惯例(9656fc0a 先例)改为 0.0.124,并补同步根 package-lock.json 的 packages["kernel"](PR 原本漏了;lock-sync 门禁确认此惯例)。
  3. writeSync → writeFileSync(fd):8 处 durable-write 点原为单次 writeSync,不保证全量写入——短写=静默截断,正是本 PR 要消灭的损坏类。对齐 credential-store 既有惯用法(writeFileSync 循环写满)。另补了 validation-failure 隔离分支的测试用例(decode-failure 分支原本有、validation 分支没有)。

本地定向验证:kernel 套件 1293/1293、issue3157 持久化测试、bounded-storage、typecheck 全绿;全量以 CI 为准。

八问

  1. 是什么问题:内核崩溃/掉电后 NTFS 只落元数据不落数据,bili 的 tmp+rename 状态文件可能留下「大小正确但内容全零」的文件,读路径每次 load 都抛错 → 整条 lane 永久 502,只能人工删文件恢复。
  2. 什么原因导致的:两层叠加。(a) 根因:状态写入只保证 rename 原子性、不保证数据持久化——字节停在 page cache,crash 落在 write 与 rename 之间就发布了指向非持久数据的文件名;(b) 放大因素:读路径对不可解码文件选择「永远抛错」而非隔离恢复,把一次瞬时损坏放大成永久故障。
  3. 影响范围与严重程度:触发面窄(仅 crash/掉电窗口内的用户),但一旦发生=该 lane 完全不可用且无任何自愈出口,严重度高。修复覆盖 kernel StateStore(canonical+spill、同步+异步共 4 个 tmp+rename 点)与 host 侧 7 个持久状态写入点;一次性临时文件(claude --settings 等)有意不动,分类理由已在代码注释。
  4. 怎么修复的:双管齐下——① 所有 tmp+rename 状态写入改为 write→fsync(fd)→close→rename(数据先于 rename 持久化,镜像仓库既有 atomicWriteConfig/credential-store 惯例);② 读路径遇不可解码或校验失败的状态文件时隔离为 <name>.corrupt-<ts>(保留现场供取证)+ 告警一次 + 空会话重启,与 plugin-conversations.json 既有的坏文件处理对齐。
  5. 是否有回归问题:审过的既有行为——loadSync 全部调用方(decompress-shared/fork-adoption/plugin/server/session/web 共 8 处)均 null-tolerant,throw→null 安全;walker 扫描规则确认 .corrupt-* 既不会被重新加载也不会被 stale-temp 清理误删;wrong-id 与 EPERM 路径不变;isValidRecord 无版本门,不存在升级误隔离;bounded-storage 契约变更(fail-closed→quarantine)已显式更新对应测试。发现并修掉一处真回归:fsyncFile 只读句柄在 Windows 上使异步写入全挂(见上第 1 条)。
  6. 是否破坏缓存:不涉及压缩缓存/增量游标/去重键。坏文件本来就无法加载,隔离机制不影响前缀缓存。
  7. 是否改了设计方向:不改架构。持久化语义从「rename 原子」收紧为「fsync 后 rename」;恢复语义从 fail-forever 变为 fail-once-with-recovery(PR 正文 ⚠️ 节已披露;安全属性保留——坏字节永远不会被当有效状态加载)。
  8. 是否引入了新的配置:无新 env/config/默认值。

遗留事项

收工判定:代码可合(人工合并,kernel 门禁);建议等 #3170 修完、本 PR 重跑 CI 全绿后合并。

@ranxianglei
ranxianglei merged commit 757b551 into master Oct 14, 2026
60 of 68 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.429-master.757b551 (rolling master tag).

npm install -g billion-context@0.0.429-master.757b551  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant