Repository navigation
[PR] fix: persist ledger rows before the model call (durability-before-render) - #6326
Conversation
…der) A turn's refs are minted in the prepare phase but their ledger rows land in the DEBOUNCED save that runs AFTER the model call (#5155). A crash in that window (power loss / disconnect / SIGKILL) leaves the rendered refs with no durable rows; on cold-load the refs and the ledger disagree, and a later turn that cites them hits "unknown ref / above the highest allocated ref" — the #3397 orphaned-generation gap. Make the rows durable BEFORE the refs are sent to the model: - add SessionStore.ingestRowsNow(): the debounced path's row-ingest legs (record build + stale-disk guard + shadow.record + own-commit stamp) with the JSON envelope write removed — just the small SQLite rows, synchronously. - call it in all four prepare phases (anthropic/google/openai/responses) between snapshotMessages() and markDirty() — after the refs are minted and the history snapshotted, before the body is built and forwarded. - call it again after forward() in the incremental lane as a safety net (idempotent); the debounced envelope save below it is unchanged (#5155). The JSON envelope stays debounced; only the small row ingest moves ahead of the model call. Verified with an isolated crash repro: a SIGKILL while the model call is in flight no longer loses the turn's rows (durable == rendered, gap=0; before the fix gap=5).
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.9262-pr.6326.4d92aa3This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr6326.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…bility path The durability-before-render pass (ingestRowsNow) converges a stale resident to the on-disk record BEFORE the debounced save does. The #405/#2401 'rejected stale snapshot' warning lived only in the debounced save's stale branch, so once this earlier pass converged the session the debounced save no longer saw a stale state and the warning (and the multi-instance test's assertion on it) went silent. Log the same throttled warning here so the guard stays observable regardless of which pass converges first.
|
ework |
…rd stays observable (#3397) Two legs: 1. ingestRowsNow alone (JSON writer off) makes the session durable in the unified index; a cold restart loads it with folds + ref ledger intact — the orphaned-generation gap the fix closes. 2. the multi-instance stale-disk guard through the durability path still emits the throttled 'rejected stale snapshot' warn (tagged '(durability-before-render)') and never rolls the shared index counters back.
ranxianglei
left a comment
There was a problem hiding this comment.
结论:可以合并(建议 CI 全绿后由人工点合并;本账号是 PR owner,GitHub 不允许自审 approve,故以 comment 形式发布)
这个 PR 做的事一句话:模型看到某轮内容之前,先把这一轮在 SQLite 索引里的账本行(refs/blocks)同步写盘,这样进程在「已渲染、但延迟 JSON 保存还没落」的窗口里崩溃(掉电/断连/SIGKILL)也不会留下「模型引用过、但索引里没有」的孤儿 ref(#3397 类故障)。大的 JSON 信封保存仍保持延迟,只有小的行 ingest 提前。
设计核验(都过)
- 不变式成立:ref 的铸造(fold)发生在 kernel transform,四个 prepare 阶段里都在插入点之前;
ingestRowsNow插在snapshotMessages()和markDirty()之间(prepare-anthropic.ts:611-618、prepare-google.ts:419、prepare-openai.ts:642、prepare-responses.ts:718)。第 N 轮响应里铸造的 ref 在第 N+1 轮 prepare 时铸造并同轮 ingest,都在模型调用前——模型永远看不到没有账本行支撑的 ref。incremental-lane.ts:1008 的调用是响应对齐后的兜底,幂等。 - 性能:
ingestRowsNow走staleDiskPayload的 ownership fast path(persist.ts:2853 起,索引行计数器 + 信封 stat 比对),正常请求不触发全量信封读取;steady-state 成本 = 一次 record build(boundedFoldedSnapshot是有界的)+ 一次小 ingest。注意:这把原来藏在流式输出期间的 ingest 挪到了请求关键路径(TTFB 上),对折叠尾部的全量 re-ingest 会话有一次性额外成本——量级小,可接受。 - 多实例 stale 守卫:commit b34e549 在 durable 路径的 stale 分支补了带节流(60s)的 "rejected stale snapshot … (durability-before-render)" 告警 + converge +
shadow.record(disk),守卫在持久化路径保持可观测,且不回滚共享索引计数器(见新增测试)。 - 失败安全:整个方法 try/catch,异常只告警,延迟保存照旧兜底。
发现并已修(小问题,按惯例直接追加了 fix commit)
- 6 处注释引用了错误的 issue 号:
#6150-fix(persist.ts:3165、incremental-lane.ts:1008)指向的是已关闭且无关的 #6150(删除 compressAtFoldPoint 实验特性的 PR);四个 prepare 文件里还留着没填的#6xxx占位。已改为 #3397 / PR #6326 → commit63bde1c0。 - PR 原本零测试:仓库惯例是每个修复都有钉死回归的测试。新增
tests/issue3397-durability-before-render.test.ts两条腿:- JSON writer 全关时,仅
ingestRowsNow就能让会话在索引里落盘——冷重启(模拟崩溃后)从索引加载,folds + ref 账本完整(#3397 缺口闭合的钉死); - stale 守卫在 durable 路径上仍告警 + 收敛 + 索引计数器不回滚。
→ commit47a542bb(本地 2/2 过,check:test-import-graph过)。
- JSON writer 全关时,仅
本地验证
- 相关套件全绿:incremental-lane 25/25、issue5155 7/7、storage-json-writer-off 4/4、multi-instance-persist 3/3、新测试 2/2;prepare 侧 loop-strict-echo/compaction-archive 等 41/41(见会话记录)。
- 全量
npm test里有 13 个失败(advisory、issue3304×2、issue5627×2、log-retention×1、omp-native×2、opencode-native×1、opencode-v2×3)——全部在 origin/master 基线上复现,与本 PR 无关(多为网络依赖/环境)。
观察项(不阻塞,记录在案)
- 「同步持久化」在 mysql 后端不成立(record 是 fire-and-forget 异步入队);sqlite(默认)成立。mysql 本就是实验模式,注释措辞建议后续统一。
- 掉电(非进程崩溃)下 WAL + synchronous=NORMAL 仍可能丢最新一个事务;进程崩溃(SIGKILL/断连)完全覆盖。PR 描述里的「掉电」说法略乐观。
- writerOff(无信封)模式下 stale 守卫读不到信封、无法检测旧实例回写——这是 #3049/#3406 之前就已记录的残余行为,本 PR 只是把 ingest 时机提前,不改变该类行为,更鲜的实例下一轮写会自愈。
CI
原 head b34e5499:全绿(61 pass / 31 skip,5 个测试分片 + 各 static gate)。新增两个 fix commit(63bde1c0、47a542bb):e2e 工作流已全绿,主 CI 的 static guard 已过,测试分片在 runner 队列里排队(内容仅为注释 + 一个本地验证过的测试文件)。
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.1177-master.9674cfa # this exact build
npm install -g billion-context@master # always the newest merged build
|
Master advanced 5 commits (durability-before-render #3397 / PR #6326) since the prior sync (c26c31a). The only overlapping file, src/server/prepare-anthropic.ts, auto-merged cleanly: #3397's getStore().ingestRowsNow(session) (line ~608, before markDirty) and #4974's if (!anthropicTitleGen) gate (line 669) sit in different regions, no conflict. typecheck clean; issue4974 + side-lane-invariants + issue3397 + issue2490 + issue2847 = 31/31 pass.
Base of PR #6299 switches to 2026-11-07_snapshot-table-foundation. Conflicts resolved 5901-side-first (ingestRowsNow durability-before-render from #6326 kept in all four prepare lanes; 2×2 stage-0/delta code kept above it). Carries the untagged-save guard in snapShadowSync (#6326's pre-render persist must not diff-delete the pinned-id plane) and the mint-scan event-type fallback for response.created on the event: line. Two e2e pins (B4, Residual-1a+2) still red on the merged tree — in work.
… rule) + mint tap spin Two regressions surfaced when the branch was re-based onto the #5901 head (f000197, which carries #6326 durability-before-render): 1. B4 e2e: #6326's ingestRowsNow persists the ledger rows BEFORE render as an UNTAGGED save; its final-state diff then deleted/re-homed the pinned plane rows (ids the tagged plane cannot address — R2 doctrine) and the next tagged save marked the plane sticky-dead. snapShadowSync now guards untagged saves against a sid that holds active plane rows with a PREFIX RULE: the save is allowed only when its ledger-id view prefix-identically extends the stage-0 l1 envelope (that save is the one that materializes the rows stage-0 declared — ids agree, idempotent); a stale view (re-derived kernel ids, pre-turn transcript, compaction) is skipped for the plane, the disk record still lands, and the plane converges at the next tagged save or stage-0 resync. The guard runs BEFORE the transaction (after it the destructive diff is already committed). 2. Residual-2 e2e: the mint-id scan (the record that lets the next chained round seed the delta frontier at the stored-world boundary) existed only in the compress-loop (injected) branch; plugin and non-injected responses turns never entered it, so the first chained round re-seeded the whole world. The scan is factored into MintIdTap and rides a tee'd twin of the plugin-lane stream and of the plain (non-rewrite) proxy stream — both branches where the client-facing bytes ARE the upstream bytes. The tap fix also repairs a latent infinite-spin in the old inline scan: the event-boundary recompute ran only at the loop bottom, which the non-marker skipped — a non-marker event preceding a marker event pinned the boundary on a shrinking tail and spun the loop forever (event loop blocked; no timers fired). The recompute is now unconditional on every pass. p3 42/42 (incl. B4 frontier-commit-on-accept + Residual 1a+2); p0 10/10, p1 4/4, fastpath 2/2, p2 17/17, responses-previous-response-id 7/7, snapshot-render 13/13, render-chain 6/6, shadow 5/5, storage-engine-proofs 12/12 (kernel rebuild for #5901's compress.ts), opencode-rolling 10/10.
… rule) + mint tap spin Two regressions surfaced when the branch was re-based onto the #5901 head (f000197, which carries #6326 durability-before-render): 1. B4 e2e: #6326's ingestRowsNow persists the ledger rows BEFORE render as an UNTAGGED save; its final-state diff then deleted/re-homed the pinned plane rows (ids the tagged plane cannot address — R2 doctrine) and the next tagged save marked the plane sticky-dead. snapShadowSync now guards untagged saves against a sid that holds active plane rows with a PREFIX RULE: the save is allowed only when its ledger-id view prefix-identically extends the stage-0 l1 envelope (that save is the one that materializes the rows stage-0 declared — ids agree, idempotent); a stale view (re-derived kernel ids, pre-turn transcript, compaction) is skipped for the plane, the disk record still lands, and the plane converges at the next tagged save or stage-0 resync. The guard runs BEFORE the transaction (after it the destructive diff is already committed). 2. Residual-2 e2e: the mint-id scan (the record that lets the next chained round seed the delta frontier at the stored-world boundary) existed only in the compress-loop (injected) branch; plugin and non-injected responses turns never entered it, so the first chained round re-seeded the whole world. The scan is factored into MintIdTap and rides a tee'd twin of the plugin-lane stream and of the plain (non-rewrite) proxy stream — both branches where the client-facing bytes ARE the upstream bytes. The tap fix also repairs a latent infinite-spin in the old inline scan: the event-boundary recompute ran only at the loop bottom, which the non-marker skipped — a non-marker event preceding a marker event pinned the boundary on a shrinking tail and spun the loop forever (event loop blocked; no timers fired). The recompute is now unconditional on every pass. p3 42/42 (incl. B4 frontier-commit-on-accept + Residual 1a+2); p0 10/10, p1 4/4, fastpath 2/2, p2 17/17, responses-previous-response-id 7/7, snapshot-render 13/13, render-chain 6/6, shadow 5/5, storage-engine-proofs 12/12 (kernel rebuild for #5901's compress.ts), opencode-rolling 10/10.
… rule) + mint tap spin Two regressions surfaced when the branch was re-based onto the #5901 head (f000197, which carries #6326 durability-before-render): 1. B4 e2e: #6326's ingestRowsNow persists the ledger rows BEFORE render as an UNTAGGED save; its final-state diff then deleted/re-homed the pinned plane rows (ids the tagged plane cannot address — R2 doctrine) and the next tagged save marked the plane sticky-dead. snapShadowSync now guards untagged saves against a sid that holds active plane rows with a PREFIX RULE: the save is allowed only when its ledger-id view prefix-identically extends the stage-0 l1 envelope (that save is the one that materializes the rows stage-0 declared — ids agree, idempotent); a stale view (re-derived kernel ids, pre-turn transcript, compaction) is skipped for the plane, the disk record still lands, and the plane converges at the next tagged save or stage-0 resync. The guard runs BEFORE the transaction (after it the destructive diff is already committed). 2. Residual-2 e2e: the mint-id scan (the record that lets the next chained round seed the delta frontier at the stored-world boundary) existed only in the compress-loop (injected) branch; plugin and non-injected responses turns never entered it, so the first chained round re-seeded the whole world. The scan is factored into MintIdTap and rides a tee'd twin of the plugin-lane stream and of the plain (non-rewrite) proxy stream — both branches where the client-facing bytes ARE the upstream bytes. The tap fix also repairs a latent infinite-spin in the old inline scan: the event-boundary recompute ran only at the loop bottom, which the non-marker skipped — a non-marker event preceding a marker event pinned the boundary on a shrinking tail and spun the loop forever (event loop blocked; no timers fired). The recompute is now unconditional on every pass. p3 42/42 (incl. B4 frontier-commit-on-accept + Residual 1a+2); p0 10/10, p1 4/4, fastpath 2/2, p2 17/17, responses-previous-response-id 7/7, snapshot-render 13/13, render-chain 6/6, shadow 5/5, storage-engine-proofs 12/12 (kernel rebuild for #5901's compress.ts), opencode-rolling 10/10.
…ng changed since the sync durability save (ranxianglei#6780) 0.1.214 runs two full unified-index ingests before the model call on every turn: the synchronous durability ingest (8849b66/ranxianglei#6326) and the writer-off coalesced build armed by prepare's own markDirty (5a80e2b removed the debounce on this lane). The second one is a no-op delta that still pays the three-table full reads, the whole-record re-serialization and a second commit round-trip (~1.3s of ranxianglei#6780's pre-forward cost on a 23k-message session; local=2315ms->7708ms). ingestRowsNow now arms a per-session proof (revision epoch + lastMessages array ref + metadata fingerprint) when it has made the current state durable; the writer-off coalesced build validates it against live state at run time and, on a full match, skips its heavy rebuild + full delta-ingest while still running the settle protocol unchanged. Any mismatch (extra markDirty, re-snapshot, decomposition payload, CCR re-key, pending ledger align, metadata mutation) falls through to the full build — the arm is a hint, never a commitment. Also document in the sessions upsert why a same-value skip does not help the big JSON columns: an unchanged OVERFLOW blob is copied onto fresh pages whenever ANY column of the row changes, so moving the blobs out of the row is the real byte-cost fix (write-side issue ranxianglei#6787). Tests: new tests/issue6780-pre-forward-dedup.test.ts pins the skip and its counterfactual (a changed state still takes the full build); issue3397-durability-before-render and storage-unified suites stay green.
问题
一个长会话反复出现
compress ... every ref is unknown to this session ... startId=mX is above this session's highest allocated ref (mY),压缩一直失败、浪费大量时间。根因
增量通道(
src/incremental-lane.ts)每轮的顺序是:铸编号 →forward把上下文发给模型 → 落盘在 forward 之后的延迟(debounce)里(#5155)。窗口 = 「发模型」到「落盘」。进程在这个窗口里崩溃(掉电/断连/SIGKILL)时,这一轮已经渲染出来的编号还没有账本行。冷加载后缓存里仍显示着那些编号,但账本里没有 → 编号和账本对不上。模型再去压/引用它们,网关就报
unknown ref / above the highest allocated ref(#3397 孤立代)。修复(先保存,再发给模型)
在发给模型之前就把这一轮的编号行同步落盘:
SessionStore.ingestRowsNow():取延迟落盘里「写行」那几步(构建 record + stale-disk 保护 +shadow.record+ own-commit 戳),去掉 JSON 大信封,只把小的 SQLite 行同步写盘。snapshotMessages()(编号已铸、历史已快照)和markDirty()(触发延迟落盘)之间调用它。forward()之后也调一次作安全网(幂等);下面的延迟落盘(大信封)不变。JSON 大信封仍走延迟落盘(#5155 的性能优化保留),只是把「小的行」提前到模型调用之前。
验证(隔离,不碰本机)
崩溃复现:seed 10 条落盘 → delta 5 条(共 15)→ fake 上游卡住响应(模型调用进行中)→ 此时 SIGKILL → 冷加载。
npm run build通过。改动
6 个文件,+69 行(纯 additive,不动老逻辑):
src/persist.ts(新增ingestRowsNow())+src/server/prepare-{anthropic,google,openai,responses}.ts(各加一次调用)+src/incremental-lane.ts(forward 后安全网)。关联 issue
order-persist.ts)关闭,但那修的是"重启后 persona key 漂移/路由错 fork",不是"崩溃窗口里渲染过的 ref 没落盘"。目标会话(2026-11-10 活跃,晚于 [PR] fix: persona conversation keys survive proxy restarts; management calls follow the fork holding cited refs (#3397) #3405)复现的正是这条没覆盖的路径,所以症状看起来和 Compression fails in long sessions while new sessions work fine | 压缩失败.新会话反而没事 #3397 一模一样,根因不同。