Skip to content

[PR] fix: persist ledger rows before the model call (durability-before-render) - #6326

Merged
ranxianglei merged 4 commits into
masterfrom
2026-11-12_durability-before-render
Nov 12, 2026
Merged

ranxianglei merged 4 commits into
masterfrom
2026-11-12_durability-before-render

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

问题

一个长会话反复出现 compress ... every ref is unknown to this session ... startId=mX is above this session's highest allocated ref (mY),压缩一直失败、浪费大量时间。

根因

增量通道(src/incremental-lane.ts)每轮的顺序是:铸编号 → forward 把上下文发给模型 → 落盘在 forward 之后的延迟(debounce)里(#5155)。窗口 = 「发模型」到「落盘」。

进程在这个窗口里崩溃(掉电/断连/SIGKILL)时,这一轮已经渲染出来的编号还没有账本行。冷加载后缓存里仍显示着那些编号,但账本里没有 → 编号和账本对不上。模型再去压/引用它们,网关就报 unknown ref / above the highest allocated ref(#3397 孤立代)。

修复(先保存,再发给模型)

在发给模型之前就把这一轮的编号行同步落盘:

  • 新增 SessionStore.ingestRowsNow():取延迟落盘里「写行」那几步(构建 record + stale-disk 保护 + shadow.record + own-commit 戳),去掉 JSON 大信封,只把小的 SQLite 行同步写盘。
  • 在 4 个 prepare 阶段(anthropic/google/openai/responses)的 snapshotMessages()(编号已铸、历史已快照)和 markDirty()(触发延迟落盘)之间调用它。
  • 增量通道 forward() 之后也调一次作安全网(幂等);下面的延迟落盘(大信封)不变。

JSON 大信封仍走延迟落盘(#5155 的性能优化保留),只是把「小的行」提前到模型调用之前。

验证(隔离,不碰本机)

崩溃复现:seed 10 条落盘 → delta 5 条(共 15)→ fake 上游卡住响应(模型调用进行中)→ 此时 SIGKILL → 冷加载。

  • 修复前:冷加载后 durable=10、rendered=15,gap=5(编号丢了)。
  • 修复后:模型调用进行中 durable 就已经=15(编号在发模型前落盘),冷加载后仍=15,gap=0。
  • 全量 npm run build 通过。

改动

6 个文件,+69 行(纯 additive,不动老逻辑):src/persist.ts(新增 ingestRowsNow())+ src/server/prepare-{anthropic,google,openai,responses}.ts(各加一次调用)+ src/incremental-lane.ts(forward 后安全网)。


关联 issue

…der)

A turn's refs are minted in the prepare phase but their ledger rows land in
the DEBOUNCED save that runs AFTER the model call (#5155). A crash in that
window (power loss / disconnect / SIGKILL) leaves the rendered refs with no
durable rows; on cold-load the refs and the ledger disagree, and a later turn
that cites them hits "unknown ref / above the highest allocated ref" — the
#3397 orphaned-generation gap.

Make the rows durable BEFORE the refs are sent to the model:
- add SessionStore.ingestRowsNow(): the debounced path's row-ingest legs
  (record build + stale-disk guard + shadow.record + own-commit stamp) with
  the JSON envelope write removed — just the small SQLite rows, synchronously.
- call it in all four prepare phases (anthropic/google/openai/responses)
  between snapshotMessages() and markDirty() — after the refs are minted and
  the history snapshotted, before the body is built and forwarded.
- call it again after forward() in the incremental lane as a safety net
  (idempotent); the debounced envelope save below it is unchanged (#5155).

The JSON envelope stays debounced; only the small row ingest moves ahead of
the model call. Verified with an isolated crash repro: a SIGKILL while the
model call is in flight no longer loses the turn's rows (durable == rendered,
gap=0; before the fix gap=5).
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-12_durability-before-render (47a542b)

Latest build — install this exact version

npm install -g billion-context@0.0.9262-pr.6326.4d92aa3

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
47a542b billion-context@0.0.9262-pr.6326.4d92aa3 build
b34e549 billion-context@0.0.9223-pr.6326.6fea884 build
8849b66 billion-context@0.0.9219-pr.6326.37296bd build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr6326.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: persist ledger rows before the model call (durability-before-render) [PR] fix: persist ledger rows before the model call (durability-before-render) Nov 12, 2026
…bility path

The durability-before-render pass (ingestRowsNow) converges a stale resident to
the on-disk record BEFORE the debounced save does. The #405/#2401 'rejected
stale snapshot' warning lived only in the debounced save's stale branch, so once
this earlier pass converged the session the debounced save no longer saw a stale
state and the warning (and the multi-instance test's assertion on it) went
silent. Log the same throttled warning here so the guard stays observable
regardless of which pass converges first.
@ranxianglei

Copy link
Copy Markdown
Owner Author

ework

… not the closed unrelated #6150

#6150 is the closed 'remove compressAtFoldPoint' PR — the comment refs
read as pointing at a retired feature. Fill the '#6xxx' placeholders with
the issue the fix actually closes.
…rd stays observable (#3397)

Two legs:
1. ingestRowsNow alone (JSON writer off) makes the session durable in the
   unified index; a cold restart loads it with folds + ref ledger intact —
   the orphaned-generation gap the fix closes.
2. the multi-instance stale-disk guard through the durability path still
   emits the throttled 'rejected stale snapshot' warn (tagged
   '(durability-before-render)') and never rolls the shared index counters
   back.

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

结论:可以合并(建议 CI 全绿后由人工点合并;本账号是 PR owner,GitHub 不允许自审 approve,故以 comment 形式发布)

这个 PR 做的事一句话:模型看到某轮内容之前,先把这一轮在 SQLite 索引里的账本行(refs/blocks)同步写盘,这样进程在「已渲染、但延迟 JSON 保存还没落」的窗口里崩溃(掉电/断连/SIGKILL)也不会留下「模型引用过、但索引里没有」的孤儿 ref(#3397 类故障)。大的 JSON 信封保存仍保持延迟,只有小的行 ingest 提前。

设计核验(都过)

  • 不变式成立:ref 的铸造(fold)发生在 kernel transform,四个 prepare 阶段里都在插入点之前;ingestRowsNow 插在 snapshotMessages() 和 markDirty() 之间(prepare-anthropic.ts:611-618、prepare-google.ts:419、prepare-openai.ts:642、prepare-responses.ts:718)。第 N 轮响应里铸造的 ref 在第 N+1 轮 prepare 时铸造并同轮 ingest,都在模型调用前——模型永远看不到没有账本行支撑的 ref。incremental-lane.ts:1008 的调用是响应对齐后的兜底,幂等。
  • 性能:ingestRowsNow 走 staleDiskPayload 的 ownership fast path(persist.ts:2853 起,索引行计数器 + 信封 stat 比对),正常请求不触发全量信封读取;steady-state 成本 = 一次 record build(boundedFoldedSnapshot 是有界的)+ 一次小 ingest。注意:这把原来藏在流式输出期间的 ingest 挪到了请求关键路径(TTFB 上),对折叠尾部的全量 re-ingest 会话有一次性额外成本——量级小,可接受。
  • 多实例 stale 守卫:commit b34e549 在 durable 路径的 stale 分支补了带节流(60s)的 "rejected stale snapshot … (durability-before-render)" 告警 + converge + shadow.record(disk),守卫在持久化路径保持可观测,且不回滚共享索引计数器(见新增测试)。
  • 失败安全:整个方法 try/catch,异常只告警,延迟保存照旧兜底。

发现并已修(小问题,按惯例直接追加了 fix commit)

  1. 6 处注释引用了错误的 issue 号:#6150-fix(persist.ts:3165、incremental-lane.ts:1008)指向的是已关闭且无关的 #6150(删除 compressAtFoldPoint 实验特性的 PR);四个 prepare 文件里还留着没填的 #6xxx 占位。已改为 #3397 / PR #6326 → commit 63bde1c0。
  2. PR 原本零测试:仓库惯例是每个修复都有钉死回归的测试。新增 tests/issue3397-durability-before-render.test.ts 两条腿:
    • JSON writer 全关时,仅 ingestRowsNow 就能让会话在索引里落盘——冷重启(模拟崩溃后)从索引加载,folds + ref 账本完整(#3397 缺口闭合的钉死);
    • stale 守卫在 durable 路径上仍告警 + 收敛 + 索引计数器不回滚。
      → commit 47a542bb(本地 2/2 过,check:test-import-graph 过)。

本地验证

  • 相关套件全绿:incremental-lane 25/25、issue5155 7/7、storage-json-writer-off 4/4、multi-instance-persist 3/3、新测试 2/2;prepare 侧 loop-strict-echo/compaction-archive 等 41/41(见会话记录)。
  • 全量 npm test 里有 13 个失败(advisory、issue3304×2、issue5627×2、log-retention×1、omp-native×2、opencode-native×1、opencode-v2×3)——全部在 origin/master 基线上复现,与本 PR 无关(多为网络依赖/环境)。

观察项(不阻塞,记录在案)

  • 「同步持久化」在 mysql 后端不成立(record 是 fire-and-forget 异步入队);sqlite(默认)成立。mysql 本就是实验模式,注释措辞建议后续统一。
  • 掉电(非进程崩溃)下 WAL + synchronous=NORMAL 仍可能丢最新一个事务;进程崩溃(SIGKILL/断连)完全覆盖。PR 描述里的「掉电」说法略乐观。
  • writerOff(无信封)模式下 stale 守卫读不到信封、无法检测旧实例回写——这是 #3049/#3406 之前就已记录的残余行为,本 PR 只是把 ingest 时机提前,不改变该类行为,更鲜的实例下一轮写会自愈。

CI

原 head b34e5499:全绿(61 pass / 31 skip,5 个测试分片 + 各 static gate)。新增两个 fix commit(63bde1c0、47a542bb):e2e 工作流已全绿,主 CI 的 static guard 已过,测试分片在 runner 队列里排队(内容仅为注释 + 一个本地验证过的测试文件)。

@ranxianglei ranxianglei added bug Something isn't working p1 scope/storage labels Nov 12, 2026
@ranxianglei
ranxianglei merged commit 9674cfa into master Nov 12, 2026
89 of 129 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1177-master.9674cfa (rolling master tag).

npm install -g billion-context@0.0.1177-master.9674cfa  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei pushed a commit that referenced this pull request Nov 12, 2026
Master advanced 5 commits (durability-before-render #3397 / PR #6326) since
the prior sync (c26c31a). The only overlapping file, src/server/prepare-anthropic.ts,
auto-merged cleanly: #3397's getStore().ingestRowsNow(session) (line ~608, before
markDirty) and #4974's if (!anthropicTitleGen) gate (line 669) sit in different
regions, no conflict. typecheck clean; issue4974 + side-lane-invariants +
issue3397 + issue2490 + issue2847 = 31/31 pass.
ranxianglei added a commit that referenced this pull request Nov 13, 2026
Base of PR #6299 switches to 2026-11-07_snapshot-table-foundation.
Conflicts resolved 5901-side-first (ingestRowsNow durability-before-render
from #6326 kept in all four prepare lanes; 2×2 stage-0/delta code kept
above it). Carries the untagged-save guard in snapShadowSync (#6326's
pre-render persist must not diff-delete the pinned-id plane) and the
mint-scan event-type fallback for response.created on the event: line.

Two e2e pins (B4, Residual-1a+2) still red on the merged tree — in work.
ranxianglei added a commit that referenced this pull request Nov 13, 2026
… rule) + mint tap spin

Two regressions surfaced when the branch was re-based onto the #5901 head
(f000197, which carries #6326 durability-before-render):

1. B4 e2e: #6326's ingestRowsNow persists the ledger rows BEFORE render as
   an UNTAGGED save; its final-state diff then deleted/re-homed the pinned
   plane rows (ids the tagged plane cannot address — R2 doctrine) and the
   next tagged save marked the plane sticky-dead. snapShadowSync now
   guards untagged saves against a sid that holds active plane rows with a
   PREFIX RULE: the save is allowed only when its ledger-id view
   prefix-identically extends the stage-0 l1 envelope (that save is the
   one that materializes the rows stage-0 declared — ids agree,
   idempotent); a stale view (re-derived kernel ids, pre-turn transcript,
   compaction) is skipped for the plane, the disk record still lands, and
   the plane converges at the next tagged save or stage-0 resync. The
   guard runs BEFORE the transaction (after it the destructive diff is
   already committed).

2. Residual-2 e2e: the mint-id scan (the record that lets the next chained
   round seed the delta frontier at the stored-world boundary) existed
   only in the compress-loop (injected) branch; plugin and non-injected
   responses turns never entered it, so the first chained round re-seeded
   the whole world. The scan is factored into MintIdTap and rides a tee'd
   twin of the plugin-lane stream and of the plain (non-rewrite) proxy
   stream — both branches where the client-facing bytes ARE the upstream
   bytes. The tap fix also repairs a latent infinite-spin in the old
   inline scan: the event-boundary recompute ran only at the loop bottom,
   which the non-marker  skipped — a non-marker event preceding
   a marker event pinned the boundary on a shrinking tail and spun the
   loop forever (event loop blocked; no timers fired). The recompute is
   now unconditional on every pass.

p3 42/42 (incl. B4 frontier-commit-on-accept + Residual 1a+2); p0 10/10,
p1 4/4, fastpath 2/2, p2 17/17, responses-previous-response-id 7/7,
snapshot-render 13/13, render-chain 6/6, shadow 5/5, storage-engine-proofs
12/12 (kernel rebuild for #5901's compress.ts), opencode-rolling 10/10.
@github-actions github-actions Bot mentioned this pull request Nov 13, 2026
ranxianglei added a commit that referenced this pull request Nov 14, 2026
… rule) + mint tap spin

Two regressions surfaced when the branch was re-based onto the #5901 head
(f000197, which carries #6326 durability-before-render):

1. B4 e2e: #6326's ingestRowsNow persists the ledger rows BEFORE render as
   an UNTAGGED save; its final-state diff then deleted/re-homed the pinned
   plane rows (ids the tagged plane cannot address — R2 doctrine) and the
   next tagged save marked the plane sticky-dead. snapShadowSync now
   guards untagged saves against a sid that holds active plane rows with a
   PREFIX RULE: the save is allowed only when its ledger-id view
   prefix-identically extends the stage-0 l1 envelope (that save is the
   one that materializes the rows stage-0 declared — ids agree,
   idempotent); a stale view (re-derived kernel ids, pre-turn transcript,
   compaction) is skipped for the plane, the disk record still lands, and
   the plane converges at the next tagged save or stage-0 resync. The
   guard runs BEFORE the transaction (after it the destructive diff is
   already committed).

2. Residual-2 e2e: the mint-id scan (the record that lets the next chained
   round seed the delta frontier at the stored-world boundary) existed
   only in the compress-loop (injected) branch; plugin and non-injected
   responses turns never entered it, so the first chained round re-seeded
   the whole world. The scan is factored into MintIdTap and rides a tee'd
   twin of the plugin-lane stream and of the plain (non-rewrite) proxy
   stream — both branches where the client-facing bytes ARE the upstream
   bytes. The tap fix also repairs a latent infinite-spin in the old
   inline scan: the event-boundary recompute ran only at the loop bottom,
   which the non-marker  skipped — a non-marker event preceding
   a marker event pinned the boundary on a shrinking tail and spun the
   loop forever (event loop blocked; no timers fired). The recompute is
   now unconditional on every pass.

p3 42/42 (incl. B4 frontier-commit-on-accept + Residual 1a+2); p0 10/10,
p1 4/4, fastpath 2/2, p2 17/17, responses-previous-response-id 7/7,
snapshot-render 13/13, render-chain 6/6, shadow 5/5, storage-engine-proofs
12/12 (kernel rebuild for #5901's compress.ts), opencode-rolling 10/10.
ranxianglei added a commit that referenced this pull request Nov 14, 2026
… rule) + mint tap spin

Two regressions surfaced when the branch was re-based onto the #5901 head
(f000197, which carries #6326 durability-before-render):

1. B4 e2e: #6326's ingestRowsNow persists the ledger rows BEFORE render as
   an UNTAGGED save; its final-state diff then deleted/re-homed the pinned
   plane rows (ids the tagged plane cannot address — R2 doctrine) and the
   next tagged save marked the plane sticky-dead. snapShadowSync now
   guards untagged saves against a sid that holds active plane rows with a
   PREFIX RULE: the save is allowed only when its ledger-id view
   prefix-identically extends the stage-0 l1 envelope (that save is the
   one that materializes the rows stage-0 declared — ids agree,
   idempotent); a stale view (re-derived kernel ids, pre-turn transcript,
   compaction) is skipped for the plane, the disk record still lands, and
   the plane converges at the next tagged save or stage-0 resync. The
   guard runs BEFORE the transaction (after it the destructive diff is
   already committed).

2. Residual-2 e2e: the mint-id scan (the record that lets the next chained
   round seed the delta frontier at the stored-world boundary) existed
   only in the compress-loop (injected) branch; plugin and non-injected
   responses turns never entered it, so the first chained round re-seeded
   the whole world. The scan is factored into MintIdTap and rides a tee'd
   twin of the plugin-lane stream and of the plain (non-rewrite) proxy
   stream — both branches where the client-facing bytes ARE the upstream
   bytes. The tap fix also repairs a latent infinite-spin in the old
   inline scan: the event-boundary recompute ran only at the loop bottom,
   which the non-marker  skipped — a non-marker event preceding
   a marker event pinned the boundary on a shrinking tail and spun the
   loop forever (event loop blocked; no timers fired). The recompute is
   now unconditional on every pass.

p3 42/42 (incl. B4 frontier-commit-on-accept + Residual 1a+2); p0 10/10,
p1 4/4, fastpath 2/2, p2 17/17, responses-previous-response-id 7/7,
snapshot-render 13/13, render-chain 6/6, shadow 5/5, storage-engine-proofs
12/12 (kernel rebuild for #5901's compress.ts), opencode-rolling 10/10.
Louie0806 pushed a commit to Louie0806/billion-context that referenced this pull request Nov 15, 2026
…ng changed since the sync durability save (ranxianglei#6780)

0.1.214 runs two full unified-index ingests before the model call on every
turn: the synchronous durability ingest (8849b66/ranxianglei#6326) and the writer-off
coalesced build armed by prepare's own markDirty (5a80e2b removed the
debounce on this lane). The second one is a no-op delta that still pays the
three-table full reads, the whole-record re-serialization and a second
commit round-trip (~1.3s of ranxianglei#6780's pre-forward cost on a 23k-message
session; local=2315ms->7708ms).

ingestRowsNow now arms a per-session proof (revision epoch + lastMessages
array ref + metadata fingerprint) when it has made the current state
durable; the writer-off coalesced build validates it against live state at
run time and, on a full match, skips its heavy rebuild + full delta-ingest
while still running the settle protocol unchanged. Any mismatch (extra
markDirty, re-snapshot, decomposition payload, CCR re-key, pending ledger
align, metadata mutation) falls through to the full build — the arm is a
hint, never a commitment.

Also document in the sessions upsert why a same-value skip does not help
the big JSON columns: an unchanged OVERFLOW blob is copied onto fresh pages
whenever ANY column of the row changes, so moving the blobs out of the row
is the real byte-cost fix (write-side issue ranxianglei#6787).

Tests: new tests/issue6780-pre-forward-dedup.test.ts pins the skip and its
counterfactual (a changed state still takes the full build);
issue3397-durability-before-render and storage-unified suites stay green.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant