Skip to content

[PR] feat: one-time ledger restamp of legacy refs + fork row cloning (#4316 layer 3 PR3/3) - #4575

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-24_seq-ledger-pr3
Oct 24, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-24_seq-ledger-pr3

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #4316 (layer 3, final PR of the series).

What this does

Closes the loop opened by #4346 (ledgerSeq stamping) and #4552 (interval coverage): legacy sessions still carry counter-minted refs that drift from store row seqs (BLOCKED messages take no number, edits/churn widen the gap). This PR re-homes the citation space onto the ledger exactly once, and makes fork preserve numbering by cloning rows.

Kernel

  • restampRefsToLedger(state, seqByRawId): atomic one-shot restamp (draft maps, swap at end).
    • BLOCKED stays unnumbered; dead refs untouched; out-of-range seqs skipped.
    • Displaced no-row ids re-home to fresh high numbers (highestUsedIndex+1 walk — no-reuse contract intact), recorded in superseded.
    • CompressionState.refLedgerAligned marks done (rides the state-hydration single-source-of-truth merge → auto-persisted).
  • Tests kernel/tests/seq-restamp.test.ts 4/4: BLOCKED offset, swap atomicity, no-row keep + displaced re-home + dead-ref survival, aligned-idempotent.

Host

  • Restamp moment = cold load only (loadSync afterLoad). Mid-conversation alignment was tried (getSession hook) and rejected: it breaks the D-suite byte-stability contract when rows materialize after serving started (snapshot-render flip m00024→m00027 in testing). A restart boundary has no in-flight compress call, so nothing can observe a half-renumbered map. Post-load arrivals keep counter minting — fresh numbers, no collision.
  • Superseded-ref guard: old refs carried by in-flight resends are refused by the compress funnel with explicit guidance (never silently fold the wrong span); the map rides the Session object for this process only. Refusal is safe (no wrong fold), the cost is one re-read of acp_status.
  • fork row cloning: cloneSessionRows(parent, child, throughSeq) (sqlite + mysql) — rows copied preserving (session, seq), parentSessionId/forkPointSeq lineage get real values, content/media CAS refcounts bump, replay idempotent; child appends continue from parent maxSeq+1 naturally. Storage unavailable → kernel-only clone (previous behavior).
  • seqByRaw only maps single-row msg_ids — multi-generation (edited) ids are excluded, their legacy refs stay put (honest gap).

Parity contract

kernel-change

kernel-change: refs.ts restampRefsToLedger + types.ts refLedgerAligned + index.ts exports; version 0.0.162 → 0.0.163 (lock regen).

Verification

  • kernel seq-restamp 4/4; host seq-ledger-pr3 4/4 (clone full/partial + superseded refusal + cold-load restamp once/idempotent).
  • Family: storage-unified 14, storage-corpus 9, plugin-fork 49, storage-full-flip 10, storage-review-p2s 6, incremental-lane 24, msgid-lane-flow 5, seq-interval-lane 1, storage-state-fields 2, compress family (refold 15 / recovery 10 / reject 7 / span 8 / success 10), plugin-render-endpoint 7, msgid-pin 9, msgid-algebra 14, acp-status 8, compress-observability 19, issue1839 3, snapshot-render 12 — all green. tsc 0, unused-exports clean, lifecycle OK, npm run build OK.

…ng (#4316 layer 3 PR3/3)

Kernel: restampRefsToLedger re-homes the citation space onto the unified
ledger (ref = store row seq + 1) exactly once per state — BLOCKED stays
unnumbered, dead refs survive, displaced no-row ids re-home to fresh high
numbers (no-reuse contract), multi-generation ids excluded by seqByRaw.
CompressionState.refLedgerAligned rides the single-source-of-truth merge.

Host: the restamp fires at cold load only (loadSync afterLoad) — the
restart boundary is the one moment no in-flight compress call can observe
a half-renumbered map; mid-conversation alignment would break the D-suite
byte-stability contract. Post-load arrivals keep counter minting (fresh
numbers, no collision). Superseded old refs are refused by the compress
funnel with guidance (never silently fold the wrong span); the map rides
the Session for this process only. fork endpoints clone parent rows to
the child session preserving seqs (store+mysql), lineage columns get real
values, content/media refcounts bump, replay is idempotent.

Parity: storage-unified now pins 'full serve = legacy serve + the same
deterministic alignment' via an aligned-legacy oracle digest; the corpus
manifest keeps pinning the pure view. kernel 0.0.163.
@ranxianglei ranxianglei changed the title feat: one-time ledger restamp of legacy refs + fork row cloning (#4316 layer 3 PR3/3) [PR] feat: one-time ledger restamp of legacy refs + fork row cloning (#4316 layer 3 PR3/3) Oct 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-24_seq-ledger-pr3 (380ef4d)

Latest build — install this exact version

npm install -g billion-context@0.0.5521-pr.4575.28b89d5

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
380ef4d billion-context@0.0.5521-pr.4575.28b89d5 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4575.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

发现并修复一个 master 回归(与本 PR 叠层):snapshotRenderTailStart 记的是客户端域下标,判官拿它索引渲染视图 → 视图压缩后指向末尾之外 → 判官空转(T1 合车后 #4241 牙齿测试被打红,bisect+probe 定位)。修复+D5-5 渲染分窗在 #4584(base=本分支)。

@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.824-master.8ad3aed (rolling master tag).

npm install -g billion-context@0.0.824-master.8ad3aed  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

@Zhishulo

Copy link
Copy Markdown
Contributor

#4316 PR3/3 makes the restart wire-visible — the 6 storage-engine proofs are the detector, not born-red

Master 52567f19b (push run 40167778049) is red on 6 shards. Splitting the 8 failing tests by cause:

family tests where cause
A HTTP fork revision and originals survive a new proxy process (tests/plugin-fork.test.ts:745) ubuntu 22,3 / 24,3 / windows 22,4 that test's store runs legacy-only (log: cold-loaded parent from legacy file), so refLedgerAligned is legitimately absent — #4652 is on the right axis
B 6 × tests/storage-engine-proofs.test.ts (4 × full-mode-restart-invisible-on-${chat,anthropic,responses,google}-wire, rollback contract, corrupt index.db) ubuntu 22,3 / 24,3 / windows 22,4 this commit: the one-time ledger restamp rewrites refs that are already rendered into the wire
C cold load recovers the pre-fix orphan freeze … (tests/storage-orphan-recovery.test.ts:53) ubuntu 22,2 / 24,2 / windows 22,3 same mechanism, different assertion

Evidence (job 121293… ubuntu-latest 22,3, same on 24,3)

[storage] unified FULL mode ENABLED … [engine: node:sqlite/builtin]
[persist] ledger-aligned refs for golden-chat: kept=30 restamped=5 rehomed=0 blocked=0 superseded=5 (#4316 PR3/3)
[persist] cold-loaded golden-chat from unified store: 36 ref(s), 1 block(s) …
[storage] unified shadow DIGEST DRIFT for golden-chat: unified replay != legacy load view [diverged: state] — dropped the ingest watermark, the next save re-ingests the full tail
[persist] cold-loaded golden-chat from legacy file (drift fallback): 42 ref(s) …
not ok 2259 - storage engine: full-mode restart is invisible on the chat wire
  chat/keepFile: outbound body #13 digest drift vs legacy restart

Refs are text: kernel/src/render-refs.ts injects <acp> tags into message text (RenderStrategy = "all" | "text-only"), and summaries cite mNNNNN/bN (kernel/src/parse-compress-input.ts:456). So restamped=5 genuinely rewrites 5 messages' bytes; the proof compares outbound body digests against a legacy restart (tests/storage-engine-proofs.test.ts:160-166), and body #13 — the first turn after the restart — differs by construction. The proof is doing exactly its job.

Two side effects that are also by construction:

  1. The Storage Subsystem Wrap-up Checklist (#3130 Review): Dropping the last tick upon self-disable; BILI_PERSIST=0 falsely triggering a "Storage Degraded" banner; storageMode stamp mismatched with the actual enabled mode; and 3 more (6 items total) | 存储子系统收尾清单… #3135 sampler now disagrees with its own store. After a restamp the unified view carries the aligned map while the legacy envelope still carries the pre-alignment one → [diverged: state] → watermark dropped, full tail re-ingested, every cold load. Alignment and the drift detector can't both be right one restart later.
  2. One-time wire change per pre-After model switch / client-side history rewrite, all message identities (content hashes) are replaced: ACP numbers re-m | 切模型/客户端重写历史后,消息身份(内容哈希)全部更换:acp 编号重铸、折叠覆盖大规模重挂、死 ref、churn 区临时膨胀重折 #4316 session. First cold load after the upgrade renumbers citations → the model's prelude text changes once → provider prompt-cache miss for that prefix on that turn. Client-cached refs dangle too: superseded is process-lifetime only (kernel/src/refs.ts:196-200), so the honest-reject path does not survive the restart that produced the shift.

Options (owner call)

Note the family-B/C failures are deterministic, not born-red-flaky: I reproduced C locally (Node 24 + node:sqlite, master tree) —

[persist] ledger-aligned refs for frozen-session: kept=0 restamped=1 rehomed=0 blocked=0 superseded=1 (#4316 PR3/3)
AssertionError: Expected values to be strictly equal:
  actual: undefined, expected: 'raw-new'   (tests/storage-orphan-recovery.test.ts:72)

Fix for C is small and does not need mode gating (it fails on every host with a working engine): assert the round trip instead of the pre-alignment literal — const ref = recovered.state.messageRefs.byRaw["raw-new"]; assert.equal(recovered.state.messageRefs.byRef[ref], "raw-new") — or make it assert the aligned value when refLedgerAligned is set.

#4652 will clear A only; B and C stay red behind it.


中文摘要

本条提交把「冷加载时一次性把旧 counter ref 重编号到 ledger seq+1」落了地,而 ref 是写进 wire 文本的(<acp> 标签 + 摘要里的 mNNNNN 引用)。所以它一旦执行,重启后的出站字节就必然与 legacy 重启不同:6 个 storage-engine proofs(body #13 digest 漂移)与 orphan-recovery 单测(m00003 被改成 m00001)都是在抓这件事,不是 born-red。并且重编号后 unified 视图与 legacy 信封天然不一致,会立刻触发 #3135 的 DIGEST DRIFT (diverged: state) 并重灌整条尾巴——对齐与漂移检测器在重启后互斥。

可选出路:(a) 让 proofs 只比较已对齐状态、并让采样器豁免这一次对齐(最省的 CI 解);(b) 冷加载时同时重写信封,让下一次 legacy 重启看到同一组编号;(c) 只对「ref 从未被渲染过」的会话做重编号,新消息交给 assignRefs 的 ledgerSeq 路径(唯一不产生一次性 wire 变更的方案,我认为是 #4316 layer 3 应有的形状)。plugin-fork 那条属另一族(该测试的 store 本来就走 legacy-only),#4652 可清;但它清不掉 B/C。

@Zhishulo

Copy link
Copy Markdown
Contributor

Follow-up: byte-level proof that the restamp is what changes the wire

I drove the same golden script twice locally (master tree, Node 24 + node:sqlite, emitted build), once with BILI_STORAGE_UNIFIED unset and once full, reloadAfterEdit: "keepFile", and diffed outbound body #13:

turn 13  len 83007 vs 83009  sha-equal false
first diff at char 17358
legacy: …"content":"<acp tokens=\"1.4K\" type=\"text\">m00032</acp>\nTurn 2 [GOLDEN-EDIT-SENTINEL]…
full  : …"content":"<acp tokens=\"1.3K\" type=\"text\">m00036</acp>\nTurn 2 [GOLDEN-EDIT-SENTINEL]…

Same message, same scripted turn, different ref — the one-time restamp renumbered it m00032 → m00036. Because kernel/src/render-refs.ts renders the ref into the message text, the outbound body changes and the wire-parity assertion fires:

AssertionError: chat/keepFile: outbound body #13 digest drift vs legacy restart
AssertionError: anthropic/keepFile: …   responses/keepFile: …   google/keepFile: …

All four wires, deterministically, on a host where the engine works — same signature as CI's ubuntu 22,3 / 24,3 shards. So the proofs are not born-red and not flaky: they are pinning the contract the restamp breaks. (I also reproduced tests/storage-orphan-recovery.test.ts:72 the same way — restamped=1 superseded=1, expected m00003, actual undefined.)

Nothing here changes the options in my first comment; it just moves the question from "is the CI flaky?" to "does #4316 layer 3 accept one wire-visible renumber per pre-alignment session?" — and if the answer is yes, the goldens and the #3135 sampler have to be taught about it in the same PR, otherwise the detector keeps reverting the aligned view ([diverged: state] → watermark dropped → full tail re-ingested).

ranxianglei pushed a commit that referenced this pull request Oct 25, 2026
… proof re-pin (#4599)

Inherited master red (storage-engine-proofs digest drift, #4575) — per the
CI-coordination policy the fix converges on ci-fixes, not on PR branches.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants