Skip to content

[PR] fix: initialize fork ledgers atomically before publication - #5044

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
kuviki:2026-10-28_fix-fork-ledger-publication
Oct 28, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
kuviki:2026-10-28_fix-fork-ledger-publication

Conversation

@kuviki

@kuviki kuviki commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #5043

Follow-up to the persistence findings in #4982 (#4982 (comment)), targeting #4982's branch 2026-10-27_release-v0.1.209 directly. The membership/billing follow-up #5042 remains separate.

Summary

  • Reserve child identity before asynchronous persistence; publish the child in memory and conversation bindings only after successful persistence.
  • Clone parent rows and ingest the initial child and replay receipt in one transaction, preserving sparse ledger positions, reference mappings, retired history and inherited block spans.
  • Keep established parent/fork lineage immutable. Exact replay remains idempotent, while conflicting child creation is rejected rather than overwritten.
  • Recover committed receipts after envelope-write interruption; preserve JSON-writer-off behavior and the fork_base checkpoint/resident-folding lifecycle.
  • Correct MySQL snapshot/tail remapping on the actual persisted session object and use the same transaction connection for initial ingest.

Reproduction

On baseline master b0b0e66 (package 0.1.208), the actual HTTP /__bili/plugin/fork endpoint returned 201 but converted parent live seq [0,3,4] into child [0,1,2]. The regression includes inherited compression blocks, stable refs, and ordinary save/reload. A separate SQL-contract regression exposed stale dense snapshot/tail pointers after remapping.

Validation

  • Nine targeted suites: 107 tests, 105 passed, 2 real-MySQL-gated tests skipped, 0 failed.
  • After the final test-fixture-only adjustment, the changed HTTP regression was rerun and passed (1/1), with typecheck rerun successfully.
  • Typecheck, build (including dist import/external checks), unused exports, test import graph and git diff --check: passed.
  • Real SQLite coverage includes rollback/refcounts, immutable lineage/reopen, legacy/unmirrored parents, sparse seq/ref/block reload, interrupted envelope recovery, subsequent child turns, transaction retry, writer-off behavior and existing concurrent-fork cases.
  • MySQL coverage is query/connection mocks only: transaction sharing, rollback, conflicts and absolute snapshot pointers. No real MySQL integration/concurrency claim.

Limits and review boundary

Linux Node 24.22.1 only; Windows and Node 22 await remote CI. Full-suite verdict must come from remote CI on this pushed head, not local targeted results. No kernel/version, package/version, configuration-schema, release-workflow or deployment changes. This touches storage and identity logic: draft for human review, not an auto-merge request.

Target correction and revalidation

The original master target was incorrect and is now corrected to #4982's release branch. Head 58d0d0c8e9e153a31fa12af90c008c1a852dcded preserves existing history with an additive merge commit; no force-push. Relative to release head 2f1828a, the diff is exactly the original 10 fix files (+594/-69), with an identical patch-id. Release/version metadata and the unrelated draining-twin test are unchanged from the release branch. Inherited master cleanup commits remain visible in ancestry but contribute no unrelated file changes to this PR.

Revalidated on the release-based tree: 107 targeted tests (105 passed, 2 real-MySQL-gated skipped), typecheck and build passed. Remote CI for the updated head/target is separate evidence and not yet claimed green. Human merge into #4982 remains required; this PR does not merge or publish the release.

Remote CI limitation

On updated head 58d0d0c, CI run 40690041335 ended with a version-guard failure before the base correction. That guard compares origin/master...HEAD -- package.json, so it sees the release branch's inherited version 0.1.209 on this non-release-named contribution branch. Relative to the actual target release branch, package/version metadata is unchanged. The test/static matrices were skipped; this is not evidence of a code-test pass or regression. The other 13 workflows completed successfully.

The current CI pull_request base allowlist does not include release branches, and edited/base-change events do not launch this workflow. Changing from draft to ready does not resolve the base/guard mismatch. Maintainer guidance is needed for the supported validation route. No CI rules, release metadata or branch protections were changed to bypass this gate.

@kuviki
kuviki changed the base branch from master to 2026-10-27_release-v0.1.209 October 28, 2026 12:44
@kuviki
kuviki marked this pull request as ready for review October 28, 2026 12:48
@ranxianglei

Copy link
Copy Markdown
Owner

(ai: anthropic/claude-opus-4-8)

Independent verification of this patch, run from a clean checkout of baseline master (b0b0e66f) and of this head (58d0d0c8):

Baseline reproduces the MAJOR finding. With this PR's HTTP fork preserves sparse parent ledger before the child's first save test checked out onto untouched master, it fails exactly as reported in #4982 (comment) : parent live ledger [0, 3, 4] becomes dense [0, 1, 2] in the forked child (the child's own snapshot rows land first, so the row-clone guard treats the child as "already exists" and skips the clone). Several cases in issue4982-fork-lineage-conflict.test.ts also fail on baseline because the strict conflicting-lineage behavior does not exist there yet.

On this head, all fork/persistence lanes pass locally (Linux, Node 25):

  • tests/plugin-fork.test.ts: 54/54
  • tests/issue4982-fork-lineage-conflict.test.ts: 18/18
  • tests/issue4910-fork-lineage-survives-save.test.ts: 1/1
  • tests/seq-ledger-pr3.test.ts: 4/4
  • tests/issue4730-fork-live-seq.test.ts: 2/2
  • storage-backend-contract: 7/7 + 1 skipped (real-MySQL gate), bounded-storage 11/11, storage-json-writer-off 3/3, fork-clone-media-refcount 3/3, proxy-persist 23/23
  • typecheck clean; build green including dist-import-annotations and dist-external checks.

One local failure is pre-existing environment noise, not caused by this patch: tests/issue3157-persist-durability.test.ts (quarantine tests) fails identically on untouched origin/master in this environment (Node 25 vs CI's 22/24); track separately, don't read it against this PR.

Crash-recovery path read-through: recoverCompletedFork repairs the JSON envelope from the committed receipt, and resolveForkConversation reaches the repaired record via its loadSync(persistedId) fallback — so an exact same-key retry after a mid-fork restart returns the stored receipt instead of dead-ending in 503 (pinned by the new restart-style recovery test).

Remaining gate for merge is operational: the remote test/static matrices are skipped on this head because the base is the #4982 release branch, which is not in the workflow's pull-request base allowlist, and version-guard compares package.json against master (0.1.209 vs 0.1.208). Remote CI has therefore not independently validated this head yet — the evidence above is local-only. Also note that master itself carries the bug until #4982 merges; if the release PR were redone rather than merged, this patch would need re-landing on master (same patch, normal master-based PR) — worth keeping in mind when deciding the validation route.

@ranxianglei

Copy link
Copy Markdown
Owner

(ai: anthropic/claude-opus-4-8)

@kuviki — one operational ask from the maintainer side:

We've finished an independent review of this patch (see the verification comment above: baseline reproduces the MAJOR finding, this head is green across all fork/persistence lanes plus typecheck/build). No code defects were found — the fix itself is complete as-is.

If you could toggle "Allow edits and suggestions from maintainers" on for this PR (branch update settings), that lets us push any follow-up fixes directly onto your branch if anything later surfaces during release validation. Not required to merge — just makes the remaining CI-validation route smoother (the skipped test-matrix lanes + the structural version-guard red are a byproduct of targeting the #4982 release branch; the authoritative matrix will run when this lands in #4982).

Thanks again for picking up the release-review finding so quickly!

@kuviki

kuviki commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

勾上了

@ranxianglei

Copy link
Copy Markdown
Owner

(ai: anthropic/claude-opus-4-8)

@kuviki — 进展同步:你的修复已通过 maintainer 的独立验证(见上方验证评论)。为了让这份补丁立即拿到完整的远端 CI 矩阵裁决(当前这个面向 #4982 release 分支的形态下,test-matrix 三个车道因 base 分支触发规则被跳过,version-guard 也因非 release 命名分支携带 release 版本号而结构性红),maintainer 决定在 master 上镜像一份内容完全相同的 PR 承载 CI 裁决。你的署名保留在 commit 作者与 Co-authored-by 中,你的 #5043/#5044 是问题的正式记录源。**两个 PR 谁先合入即可,另一个随后关闭;如果 #4982 这条线先行,#5044 并入 release 分支时在 release 侧跑的矩阵同样算数。**谢谢!

Fixes ranxianglei#5043

Regression review: ranxianglei#4982 (comment 6447703837, MAJOR).

Co-Authored-By: Cury <13548063+kuviki@users.noreply.github.com>
@ranxianglei
ranxianglei force-pushed the 2026-10-28_fix-fork-ledger-publication branch from 58d0d0c to 4bf931c Compare October 28, 2026 13:23
@ranxianglei
ranxianglei changed the base branch from 2026-10-27_release-v0.1.209 to master October 28, 2026 13:25
@ranxianglei

Copy link
Copy Markdown
Owner

(ai: anthropic/claude-opus-4-8)

Owner-decided consolidation: this PR is now the single vehicle for the fix.

Changes applied under your maintainer-edit permission:

  • Your fix commit bea2845 (author preserved) was rebased onto current master b0b0e66: new head 4bf931c, with a Co-Authored-By line added so both authors appear on GitHub. Patch content is byte-identical (verified via clean cherry-pick, zero diff).
  • Base retargeted from 2026-10-27_release-v0.1.209 ([PR] release v0.1.209 #4982) to master. That makes the full remote test matrix actually run here (it is skipped on release-branch bases) and clears the structural version-guard red: a release-based head carries the 0.1.209 version line, which that guard rejects on non-release branch names, and no push to a fork could change that without downgrading the release version.

Consequence at merge time: master gets the fix now (ships in v0.1.210+). The open v0.1.209 release candidate (#4982) would NOT carry this fix unless someone additionally cherry-picks it into the release branch - an owner call at merge time.

Mirror PR #5045 (identical patch on master) was closed as superseded by this PR. The verification evidence posted earlier applies byte-for-byte to this head: baseline RED on old master (parent sparse ledger seqs [0,3,4] reproduced dense child rows [0,1,2]); green at head across plugin-fork 54 / issue4982-fork-lineage-conflict 18 / issue4910 1 / seq-ledger-pr3 4 / issue4730-fork-live-seq 2 / storage-backend-contract 7 (+1 real-MySQL skip) / bounded-storage 11 / storage-json-writer-off 3 / fork-clone-media-refcount 3 / proxy-persist 23, plus clean typecheck and build.

@ranxianglei
ranxianglei merged commit 226c338 into ranxianglei:master Oct 28, 2026
17 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.897-master.226c338 (rolling master tag).

npm install -g billion-context@0.0.897-master.226c338  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重结论:不重复。#5044 是 bug #5043 的修复载体,已 supersede 同补丁镜像 PR #5045(已关);membership/billing 的 #5042 明确分开。本 PR 是该修复线的单一正式记录源。

状态核实(git 实测,非转述):PR 已合入 master —— merge 226c33893,fix commit 4bf931cc3(parent b0b0e66f),merge-base --is-ancestor 4bf931cc3 origin/master = YES;rolling master 构建 0.0.897-master.226c338 已发布。目标 release #4982(v0.1.209)也已合入,且 fix 同为 tag v0.1.209 的祖先 → v0.1.209 npm 包已携带此修复。#5043/#5044 均已关闭。

以下是对实际 diff(单 commit,10 文件 +594/-69,其中 7 个生产文件)的独立审读——未照抄上方 floor 1 的结论,逐文件重读后给出自己的判断:

结构化八问

  1. 是什么问题:plugin fork(HTTP /__bili/plugin/fork)时,子会话的持久化信封在克隆父会话 ledger 行之前落库,克隆守卫据此误判"子会话已存在"而跳过克隆——父会话稀疏 live seq [0,3,4](中间槽位已被压缩退役)在子会话坍缩成稠密 [0,1,2],继承的压缩块/稳定 refs/retired history 全部丢失。
  2. 什么原因导致的:根因是发布顺序竞态——旧 publishForkSession 先 flushSync(child) 再 sessions.set,把子会话落库提前到了父行克隆之前;这不是表面现象,"先落库后克隆"的顺序错误本身就是病因。
  3. 影响范围与严重程度:任何走 unified-storage 的 plugin fork(dsh/opencode/pi 等插件车道的 fork/子代理继承)只要 fork 一个已含压缩块或稀疏 ledger 的父会话即触发;属 fork 继承被永久破坏类,按 owner 2026-10-19 修订定级 p1(数据丢失级)。
  4. 怎么修复的:把顺序倒过来并在源头原子化——先预留子 identity(publishingForks 集 + getSession 拒绝半发布态读取),在同一 backend 事务里克隆父行并 ingest 初始子快照+回执(SQLite db.transaction / MySQL beginTransaction 单事务),成功后才 sessions.set 公开内存态;血缘改为不可变(首个 parent/fork-point 对锁定,后续 metadata 不得 reparent);精确重放幂等、冲突创建拒绝而非覆盖;envelope 写中断由 recoverCompletedFork 从已提交回执修复;MySQL 侧修正 remap 对 tailJson 的就地改写落到真正被持久化的 session 对象上,并复用同一事务连接做初始 ingest。
  5. 是否有回归问题:逐文件独立读过,未发现缺陷。重点核对的既有行为:(a) throughSeq 仍取 lastLiveSeq(parentId)(Fork clones history using live-row COUNT as the cut, so any retired row makes the child silently lose its newest messages #4730"含末位 live seq"语义保留,未漂移);(b) legacy-only 无镜像父会话仍降级 kernel-only fork,不硬失败;(c) JSON-writer-off 路径保留;(d) fork_base checkpoint / resident folding 生命周期保留;(e) SQLite 事务包装器确认 auto-execute 且返回体值(对照既有 store.ts:2057 用法),非 better-sqlite3 双调用陷阱;(f) MySQL clone 的 catch 有 rollback() + finally release(),不会泄漏开放事务污染连接池。唯一行为变更(须披露):fork 持久化从"存储不可用时静默降级为 kernel-only(旧代码 catch+log)"改为"fail-closed 抛错"(新 createForkAsync 在 storage disabled 时 throw)。这是有意收紧且已在 PR 正文披露("Strict initial creation"),方向正确——宁可报错重试也不产出半初始化的子会话。
  6. 是否破坏缓存:不涉及前缀缓存命中;影响的是持久化 ledger 一致性,反而消除了子 ledger 错位埋下的后续一致性隐患。
  7. 是否改了设计方向:无架构转向;在既有 fork 持久化机制内补正"发布顺序+原子性",符合 KDD 身份不可变/单一投影的既有教义。
  8. 是否引入了新的配置:无新 env/config/默认值。

测试有效性:新增 tests/issue4982-fork-lineage-conflict.test.ts(+251)断言有牙——覆盖 queued-fork 遇 mirror disable 时 fail-closed、不可变 pair 跨 retry/conflict/reopen(populated/empty 参数化)、MySQL clone 事务契约、initial snapshot 原子性(ingest rollback/success 双向)、MySQL initial 复用 clone 连接、绝对 snapshot+tail 指针落盘;tests/plugin-fork.test.ts(+143)钉住稀疏 [0,3,4] HTTP 回归。门控双向断言齐全,非恒真测试。

判定:独立代码审读未发现缺陷,同意该修复正确且打到根因(源头原子化,而非在传输层掩症状)。无需进一步动作——已合入 master 与 v0.1.209,全量回归裁决以远端 CI 在已合并 head 上的结果为准(随合并已跑过)。分类记档:bug · scope/storage · p1(数据丢失类);因条目已合入/关闭且 owner 已拍板,未再对已关闭项追加标签操作。

@kuviki 修复已落地 master 与 v0.1.209,谢谢快速跟进 release-review 的发现。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] (ai: vllm-qwen/qwen3.8-27b)

状态汇报(进程此前因长时间无输出被系统多次强制重启,现已恢复):

工作已完成,无待办。 本 PR(#5044,修复 #5043)的独立审核已在上方那条 (ai: vllm-qwen/qwen3.8-27b) 评论里完整给出——结构化八问 + 独立逐文件审读,结论:修复打到根因(plugin fork 持久化的「先落库后克隆」发布顺序竞态 → 改为源头原子化:先预留子 identity、同一 backend 事务内克隆父行并 ingest 初始快照+回执、成功后才公开内存态),未发现代码缺陷。

核实的落地状态(git 实测,非转述):

分类记档:bug · scope/storage · p1(数据丢失类)。

说明:floor 13 之后的连续重启都是系统层面的进程挂起重连,不是本 issue 出现了新输入或新工作项——本线程自审核完成后没有新增人类评论、没有新的 CI 状态需要处置。无需进一步动作,此条仅作收尾确认。@kuviki 修复已落地 master 与 v0.1.209,谢谢快速跟进 release-review 的发现。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] plugin fork 发布顺序导致 ledger 克隆被跳过,稀疏 seq 与持久化血缘不一致

2 participants