Skip to content

[PR] Recover legacy cross-session fold blocks stuck in parent sessions (#5312) - #5343

Merged
ranxianglei merged 5 commits into
masterfrom
2026-10-31_foreign-block-sweep
Oct 31, 2026
Merged

ranxianglei merged 5 commits into
masterfrom
2026-10-31_foreign-block-sweep

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Fixes #5312

What

Adds a host-side, reversible sweep that recovers legacy foreign fold
blocks stranded in a parent session by the old shared-parent-session-id era
(<=v0.1.209): bili sessions foreign-blocks [--apply] [--json] [--restore <sid>:<bid>].

Why (root cause, verified at current master head)

Fold blocks created while a sub-agent shared the parent's session id keep
living in the parent session after upgrade because two mechanisms reinforce
each other:

  1. the zombie-reap e-gate (src/fold-reconcile.ts) never accrues a death
    streak for a block whose covered ids never appear on the current wire —
    a foreign block's covered messages were never part of this session, so its
    presence is permanently 0 and it is never reaped;
  2. syncBlocks' stillPresent check (kernel/src/sync.ts) keeps any block
    active whenever one of its covered ids hashes-collides with a message in
    the session — parent and child reading the same files produce identical
    content-hash message ids, so the collision is structural, not accidental.

Neither the e-gate nor orphan-gc (which only reaps full wire-orphans over
3 consecutive turns) can catch a block with even a single recurring collision.

How

The sweep judges each persisted block against its session's full persisted
transcript
(live AND retired rows — the unified store retires rows, never
deletes them, #4390). This is the decisive separation from the e-gate:

  • a healthy fold's covered ids stay in the persistent transcript (they are
    this session's own history) → overlap >= 1 → spared;
  • a true foreign block's covered ids never belonged to this session →
    overlap == 0 → provably foreign.

Zero-overlap blocks are archived through a new block-level reversible
archive primitive
(UnifiedStore.archiveBlock / restoreBlock, byte-exact
round-trip of row + content, logged to block-archive.jsonl). Any-overlap
blocks are reported as collision and left alone — they need the kernel
lineage field (option B below), not a host-side guess.

Safety properties, each pinned by a test:

  • dry-run by default — prints the plan, changes nothing; --apply acts;
  • empty-transcript sessions are skipped (never mass-archived);
  • empty-coverage blocks are undecidable → never archived;
  • base-id matching across model switches (an id with a #suffix variant
    still counts as present), so we under-report rather than mis-archive;
  • healthy folds structurally spared (regression guard).

Config surface

None. Reuses the existing bili sessions CLI; no new config key, env var, or
CLI flag beyond the additive --apply / --restore on this subcommand.

Behavior changes

Purely additive. No existing path is modified. New: a sessions foreign-blocks
subcommand (dry-run default). Archiving a block takes effect on the next
session load/restart (a running proxy keeps its in-memory copy until reload).
--restore <sid>:<bid> brings a previously archived block back.

Option B (tracked separately, NOT in this PR)

Collision-bearing foreign blocks cannot be distinguished from healthy folds
host-side (the blocks table carries no conversation-origin field). The root
fix is a kernel change: give each block a creation-time conversation-key
lineage field and have syncBlocks activate only lineage-matching blocks.
That is a kernel-change requiring owner approval + human-only review, filed
as a separate p1 decision request.

Verification (on rebased head = current master)

  • npm run typecheck clean;
  • new tests/foreign-block-sweep.test.ts (6 tests: engine round-trip, policy
    classification, empty-transcript/no-covered guards, failed-archive surfacing,
    full facade e2e) green;
  • adjacent suites green: storage-archive-gc, storage-maintenance-cli,
    doctor (39 total);
  • live CLI smoke: sessions foreign-blocks renders the plan, --apply
    parses, unknown-action guard intact.

@ranxianglei ranxianglei changed the title Recover legacy cross-session fold blocks stuck in parent sessions (#5312) [PR] Recover legacy cross-session fold blocks stuck in parent sessions (#5312) Oct 31, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:重复(置信度 100%)——本 PR(#5343)与在途 PR #5334 是同一问题的两份并行交付:

标签:bug / claude-code / scope/storage / p1 / duplicate(已落库并复核;#5312 同步打上同样一组,#5308 打前四个不带 duplicate)。理由:缺陷类;受害车道 Claude Code;修复面是存储侧持久化块清理;命中你 2026-10-19 p1 条款②「上下文/存储只增不减失控增长」。

审核进度(进行中):我已独立通读 #5343 全部 601 行 diff 与 #5334 全部 432 行真实 diff(未照抄任何一侧结论)。两者判据核心相同且都成立(零交集 = 可证明外来;retire-don't-delete 保证健康折叠的结构性豁免);差异在归档机制与风险面,见后续对比表。当前状态:两分支都与最新 master 在 src/cli.ts 冲突(master 前进了 15 个提交),CI 尚未在任一 head sha 上注册 check-run。下一步本地 typecheck + 定向测试验证后,给结构化八问结论 + 合并建议(保留哪个、另一个怎么收)。

@ranxianglei ranxianglei added bug Something isn't working claude-code duplicate This issue or pull request already exists p1 scope/storage labels Oct 31, 2026
@ranxianglei
ranxianglei force-pushed the 2026-10-31_foreign-block-sweep branch from 054510c to cdcf6be Compare October 31, 2026 11:15
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-31_foreign-block-sweep (ac484ff)

Latest build — install this exact version

npm install -g billion-context@0.0.7530-pr.5343.1cc3fa3

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
ac484ff billion-context@0.0.7530-pr.5343.1cc3fa3 build
93d3a88 billion-context@0.0.7523-pr.5343.e04ca2a build
7b78137 billion-context@0.0.7521-pr.5343.8f03033 build
9727b4a billion-context@0.0.7515-pr.5343.a544e5b build
cdcf6be billion-context@0.0.7507-pr.5343.67b8826 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5343.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(bot) 🏷 (ai: vllm-qwen/qwen3.8-27b)

Review 结论

#5343 实现方向正确、工程质量过关,但与 #5334 构成 100% 重复(同机制、同 issue 家族)。D2 被并行立了两票(#5308 @04:54Z 先,#5312 @06:15Z 后),两个 agent 各自交付 PR(#5334 @08:52Z,本票 #5343 @09:41Z)。先来后到:建议保留 #5308/#5334,关闭 #5312/#5343(duplicate 标签已打好,关闭由 owner 执行)。以下是对本 PR 代码本身的独立审核(未照抄 issue 侧结论,diff 已逐文件通读并在 rebased head 054510c2c 本地跑过定向验证)。

结构化八问(本 PR #5343)

  1. 是什么问题:≤v0.1.208 时代 sub-agent 与父会话共用 session id,其创建的 fold block 在升级到 v0.1.209 后永久滞留父会话,每轮都被重新渲染进出站 wire——父上下文里无关摘要只增不减。本 PR 加一条手动可逆清理命令 bili sessions foreign-blocks(默认 dry-run)。
  2. 什么原因导致:两个机制互相加固——① zombie-reap e-gate 豁免(src/fold-reconcile.ts:1975,death streak 仅在 e===1 时累积,而从未出现在 wire 上的外部块 e≡0,永不收割);② syncBlocks 的 stillPresent(kernel/src/sync.ts:159-166)只要任一 covered id hash 碰撞就保持活跃——碰撞是结构性的(content-hash id:父子读同一批文件产生相同 msg_id)。orphan-gc 只收连续 3 轮全量 wire-orphan,单碰撞块永远重置 streak。根因(blocks 表无会话来源字段)属 option B(kernel change,等 owner 拍板);本 PR 处理的是不依赖决策的 provably-zero-overlap 子集,边界划得对。
  3. 影响范围与严重程度:仅影响 ≤v0.1.208 有过父共享 id sub-agent 且已升级的会话(其父会话上下文失控增长 = p1,owner 条款②)。PR 本身纯 additive:用户显式运行前零影响,dry-run 不改任何东西。
  4. 怎么修复的:每个持久化块对该会话**完整持久 transcript(live+retired 行)**判重叠:overlap≥1 → 健康折叠,结构性豁免;overlap==0 → 可证明外部,经新原语 UnifiedStore.archiveBlock/restoreBlock 归档(byte-exact round-trip + block-archive.jsonl 日志 + 内置 --restore)。安全前提成立:unified store retire-don't-delete(Four storage-layer defects: identity hash contaminated with positional metadata (truncation triggers identity swap) / sw | 存储层四缺陷:身份哈希掺位置元数据(截断即换血)/清扫删除回退历史/渲染接缝图片只比长度/fork 谱系死列 #4390)保证健康块的 covered ids 恒在 transcript;base-id 匹配(#suffix 变体算存在)保证只少报、不错删。
  5. 是否有回归问题:无。触碰路径既有行为逐项过:parseArgs(additive 子命令+flag,unknown-action guard 保持);store.ts/persist.ts 全为新增方法,既有方法零改动;storage-maintenance.ts 仅加可选参数与新分支;kernel/ 未触碰(kernel-guard 安全)。无语义漂移/阈值漂移/输出格式变化。新代码里有 4 处自身缺陷(见下方清单),均不构成回归。
  6. 是否破坏缓存:不涉及。不碰 prefix-cache/wire 渲染路径;归档的块下次 load/restart 生效(这正是修复的预期效果)。
  7. 是否改了设计方向:没有。纯 additive host-side tooling;根治(kernel lineage field)留给 option B 独立审批流,未混入本 PR。
  8. 是否引入新配置:无。仅既有 sessions 子命令上新增 --apply/--restore flag;config surface 一节正确声明 "None"。

回归审查(既有行为清单)

  • CLI 解析面:保持(additive;相邻 CLI 套件绿);
  • content refcount CAS:releaseContent(delete-on-zero,src/storage/store.ts:790)/putContent(upsert+bump,:735)语义核对无误,新路径不改旧行为;保持;
  • session load/render 路径:A 完全不进 buildSession/state 路径(这也是它与 B 的本质差异):保持;
  • 相邻套件 storage-archive-gc / storage-maintenance-cli / doctor 33/33 本地绿 + 本票 6/6 绿,typecheck clean(均在 rebased head 054510c 实跑)。
  • 结论:审了上述四条既有行为面,全部保持,无回归。

测试有效性审查

6 个测试都是真行为测试(byte-exact round-trip 断言、策略双向 stub、真 index.db e2e),非假测试。缺口三处:① 核心安全前提「retired 行也算存在」只在测试名/注释里写了,没有任何测试真的 retire 一行再断言——未来谁给那条 SQL 加 retired_at 过滤,整个安全前提崩塌而测试全绿(B 同样缺这条 pin);② restore-against-already-live-twin 路径无测试(缺陷①正藏在那);③ apply 的 exit-code 语义未钉住。

缺陷清单(无论哪个 PR 存活,合并前应修)

若本 PR 存活,必修五项:

  1. restoreBlock refcount 泄漏:对已 live 的同名 twin 恢复时先 putContent bump refcount、再 INSERT DO NOTHING 落空 → 每 hash 永久多挂 1 引用且 archive log 行被消费(src/storage/store.ts restoreBlock;修法 = putContent 前先查块是否存在);
  2. apply 部分失败仍 exit 0(result.failed>0 只进文本)——自动化无法感知(B 的 orphanSweepExitCode 做对了);
  3. 本地 baseId() 复制 kernel/src/prune.ts 逻辑——应 import acp-kernel 已导出的 baseIdOf(已验证导出);
  4. 补 retired-row pinning 测试(真 retire 一行断言仍计入 overlap);
  5. cli.ts help 文本空白 churn(两行无关帮助从 2→3 空格缩进)回退。

A/B 对比(独立判断)

#5343 foreign-blocks(本票) #5334 orphan-sweep
判定核心 zero-overlap vs live+retired transcript 同
块来源 blocks 表直读(流式 SQL) loadAll() 水合全会话(内存成本,#5084 OOM 类)
归档机制 物理删行+CAS release+byte-exact 导出,真实释放字节 metadata 排除清单+buildSession 过滤,数据留库(不缩容)
可逆性 内置 --restore <sid>:<bid> 手改 metadata(无 CLI 出口)
模式门控 shadow+full 均可执行(shadow=为 flip 预清,需披露) 仅 full 可执行,shadow/legacy 只读(更保守)
风险面 触碰 content refcount CAS(新原语) metadata-only,不碰 CAS
运维 部分失败 exit 0(缺陷);无单会话 scope 错误/部分失败 exit 1 ✓;支持 [id] 单会话
已知缺陷 上述 5 项 review 未发现问题(loadAll 内存是设计取舍)
CI 状态 与 master 冲突 src/cli.ts(merge-tree 实证),check-runs 未注册 已 merge ci-fixes 并自修 unused-exports 红(head 2d8f680),但同样与当前 master 冲突 src/cli.ts(merge-tree 实证,behind 14)

两者都是同一 option-A 设计的合格实现,差别在形状:A 更彻底(物理清理+内置恢复),B 风险面更小。

决策请求(owner 拍板)

我未向本分支推任何 nit 修复——查重判定下本票大概率关闭,推修复是浪费;以上清单可直接移植给存活 PR。

判定:review 完成——#5343 方向正确、无回归、质量过关,但与 #5334 平行重复;建议按选项 1 收口(关本票、保 #5334),待 owner 拍板。

ci-check and others added 3 commits October 31, 2026 20:10
… residue (#5312)

D2 leg of #5095: fold blocks created under the <=v0.1.208 shared parent
session-id era linger forever in the parent session after upgrade — the
zombie-reap e-gate never accrues for a block whose covered ids never appear
on the current wire, and syncBlocks' stillPresent collision keeps any
shared-file-read block active every turn.

Host-side option A (this PR): `bili sessions foreign-blocks` judges each
persisted block against its session's FULL transcript (live + retired rows).
Zero intersection = provably foreign -> reversibly archived (new block-level
archive primitive, byte-exact round-trip); any intersection = collision that
needs the kernel lineage field (option B, tracked separately) -> left alone.
Dry-run by default; empty-transcript sessions are skipped (never mass-archived).
Healthy folds are structurally spared because their covered ids stay in the
persistent transcript. No config surface added (reuses the sessions CLI).
… test

Review-driven, test-only (no production change). The sweep's entire safety
argument rests on a healthy fold's covered messages staying queryable in the
session transcript even after they are consumed/rewound. The first test was
named "live + retired" but only ever created live rows, so a future
regression that turned the historian sweep's RETIRE into a DELETE would slip
through. Rewind id A out of the authoritative view (completeLog re-ingest of
[B,C]) and assert allMessageIds still returns it alongside live B/C.
Review-driven one-line hardening in runForeignBlocks. Splitting on the FIRST
colon truncated a session id that itself contains ':' (sid cut short, bid
wrong). Block ids are plain (bNN) and never carry a colon, so lastIndexOf is
strictly safer and byte-identical for every normal <sid>:<bid> input.
typecheck clean + foreign-block sweep suite 6/6.
@ranxianglei
ranxianglei force-pushed the 2026-10-31_foreign-block-sweep branch from 9727b4a to 7b78137 Compare October 31, 2026 12:10
…flation when a live twin exists (review finding 1)
@ranxianglei
ranxianglei force-pushed the 2026-10-31_foreign-block-sweep branch from 7b78137 to 93d3a88 Compare October 31, 2026 12:12
…-export sweep-only types

- config-surface-guard: --apply/--restore now referenced under 'bili sessions' (README)
- unused-exports: ForeignBlockFinding/ForeignBlockApplyResult are only used inside
  foreign-block-sweep.ts — drop the leading export keywords
@ranxianglei
ranxianglei merged commit a13132b into master Oct 31, 2026
48 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.960-master.a13132b (rolling master tag).

npm install -g billion-context@0.0.960-master.a13132b  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei pushed a commit that referenced this pull request Nov 1, 2026
…e environment (corrects 9eb1aa3)

9eb1aa3 regenerated outside the suite's preload chain, so it captured the
display-gate ON bytes (default since #5368) while every CI leg pins
BILI_MESSAGE_TIMESTAMPS=0 via tests/preload-test-env.mjs — the committed
digests could never match the pinned runners.

This run uses the exact CI invocation
(node --import ./tests/preload-storage-baseline.mjs --import ./tests/preload-test-env.mjs --import tsx),
so wire bodies are back on the pre-flip gate-off values and the
stateDigest fields pick up the persisted-state contract moves that landed
on master after the goldens' last refresh (#5338 cost ledger, #5412 async
latch parity, #5343 block archive). Turns=15/foldAt=9 unchanged; verified
green locally under the same preloads (storage-engine-proofs +
golden-storage-render).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working claude-code duplicate This issue or pull request already exists p1 scope/storage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant