Skip to content

fix: sparse-ref unknown semantics + actionable compress failure receipts (#3302) - #3533

Merged
ranxianglei merged 3 commits into
masterfrom
2026-10-17_issue3302-sparse-ref-feedback
Oct 18, 2026
Merged

ranxianglei merged 3 commits into
masterfrom
2026-10-17_issue3302-sparse-ref-feedback

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Closes #3302

kernel-change: host-side pruning (session.ts pruneRefsToLiveIds on the compaction-archive and zombie-reap paths) makes the session ref space SPARSE, but kernel error semantics treated every byRef miss as cross-session/typo ("(typo or wrong session)"). Models then probed endIds one number at a time — each probe a full round (up to 102s in the reporter's log). The error wording and gate message live in the kernel, so the host alone cannot fix them. #3302 is the declaration thread (human-opened, owner accepted with 接单).

What changed

Kernel (acp-kernel 0.0.130 → 0.0.131)

  1. Unknown-ref errors split by position (boundaries.ts): a ref ABOVE the highest allocated ref is reported as a stale artifact from an earlier generation (names the frontier); a ref at/below it is reported as an unallocated-or-pruned gap in a sparse ref space — explicitly "this is not a cross-session ref". The false "(typo or wrong session)" assertion is gone from message-ref boundaries.
  2. Tombstoned dead refs snap instead of failing (boundaries.ts + compress.ts): when state.deadRefs marks a number as pruned, boundary resolution snaps to the nearest visible message (the range folds slightly less, note emitted). No neighbor ⇒ honest consumed/DEAD classification ("nothing left to fold here"), never "unknown". danglingMessageRefs now counts tombstoned endpoints as dangling so the existing compress failure is self-amplifying: failed calls are hidden as orphan refs, making the same range permanently un-compressible #2362 DEAD receipt path covers host-pruned refs.
  3. All-unknown gate names the failing endpoints (compress.ts): e.g. failing endpoints: endId=m10750 per range; drops the two false assertions ("cannot come from an earlier compress in this session…"); adds "Do not probe for valid numbers." The marker phrase "every ref is unknown to this session" is kept verbatim — host detection (stream.ts, plugin.ts) and tests depend on it.
  4. Diagnostics reworded: bare session highest ref= (which invited upward probing) becomes session highest allocated ref=mNNNNN (the ref space is sparse — numbers at or below it are NOT necessarily allocated, so probing numbers will not find a valid one; use only refs acp_status lists).

Host

  1. Producer side (session.ts): pruneRefsToLiveIds records every pruned ref into state.deadRefs (deduped, numeric-sorted), feeding the kernel tombstone path from both producers (compaction archive omp 原生压缩后 bili 注册表跨代失配:syncBlocks 静默失活全部块、decompress 永久不可达(无视 omp session_compact 钩子;#19 搁置项的完整机制) #395, zombie reap [Bug][proxy] unframed host-side history deletion leaves zombie fold blocks rendering carriers forever — one carrier flaps in/out every turn (persistent prefix-cache churn) #2695).
  2. Failure receipt fallback (stream.ts): when no live nudge decision exists, the receipt falls back to the LAST nudge's compressible-range table ("use these refs exactly as listed; recheck acp_status if one is rejected") instead of the generic "Call acp_status before retrying".
  3. Probe-family observability (loop/core.ts): one-shot [warn] log when ≥2 distinct values fail at the same endpoint across rounds (Ask 6 conservative option — the fix: break the small-model same-parameter compress failure loop early (#156) #370 breaker stays identical-args-only, no early break added).
  4. stale-ref cause text and FOLD BASE GENERATION MISMATCH body no longer assert cross-session origin (stable prefixes/markers preserved).

Behavior changes (old → new)

  • Boundary error text: endId="m10750" does not exist in this session (typo or wrong session) — run acp_status for current refs. → three-class wording (above-frontier / in-band sparse gap / dead-snapped). Same tool-call shape, same errors[] positions.
  • All-unknown gate message: rewritten (endpoint names added, two false assertions removed, numeric probing discouraged). Marker phrase preserved.
  • Diagnostics line format changed (sparse-space framing).
  • Failure receipts may now contain "Current compressible ranges from the last compression check…" where they previously said "Call acp_status before retrying".
  • New [warn] [acp-loop] compress probe family at round N: … log line (observability only, no control-flow change).
  • state.deadRefs can now be populated by host pruning (previously only kernel-side drift minted entries); the acp_status DEAD REFS section may show more entries — all truthful (those messages were actually pruned from the view).

Tests

  • New: kernel/tests/issue3302-sparse-ref-unknown.test.ts (6 tests: in-band vs above-frontier wording, end/start snap, no-neighbor DEAD classification, gate endpoint naming) and tests/issue3302-sparse-ref-tombstones.test.ts (6 tests: tombstone recording/sort/merge via reapDestroyedSubstrate, host-lane refused/applied through applyRanges, lastNudge fallback receipt, recordProbeFailures unit).
  • Updated: kernel/tests/compress.test.ts old-wording assertions (7 sites) — they pin the NEW three-class semantics.
  • Targeted local runs green: kernel compress/dead-refs/issue3302 files 44/44; host issue3302/issue2360/issue2592-rebase-floor/issue2432/issue2146/compress-span-hint/compress-recovery-ranges/pins-storage-refactor 52/52. Full-suite verdict = CI on the pushed head.

Rebased onto current master (#3526/#3524/#3519/#3516 merges): only lockfile version-echo conflicts; kernel bump renumbered 0.0.128 → 0.0.131 to sit above master's 0.0.130, and the kernel lockfile drift left on master (0.0.129) is synced in the kernel commit. Re-verified on rebased head: typecheck clean, kernel targeted 49/49 (incl. dead-ref-split), host targeted 52/52.

awork added 2 commits October 17, 2026 17:16
…erations, name failing endpoints (#3302)

- resolveAnchorIndex: a byRef miss at/below the highest allocated ref is reported as an unallocated-or-pruned sparse-space hole, not '(typo or wrong session)'; above the highest it names the frontier

- tombstoned dead refs (state.deadRefs) resolve by snapping to the nearest visible message instead of failing outright; no neighbor => honest consumed/DEAD classification

- all-unknown gate names each failing endpoint (startId=/endId= per range) and stops inviting numeric probing; diagnostics state the ref space is sparse

- danglingMessageRefs counts tombstoned endpoints as dangling so the #2362 DEAD path covers host-pruned refs

kernel-change: #3302 — host-side pruning (session.ts pruneRefsToLiveIds) makes the ref space sparse, but kernel error semantics assumed absence means cross-session/typo; models then probed endIds one number at a time, each probe a full round. Host alone cannot fix the error semantics or the gate wording.

Bumps acp-kernel to 0.0.128.
…tionable (#3302)

- pruneRefsToLiveIds records pruned refs into state.deadRefs (deduped, sorted) so both producers (compaction archive, zombie reap) feed the kernel tombstone path

- failure receipt falls back to the last nudge's compressible-range table when no live decision exists, instead of 'Call acp_status before retrying'

- loop logs a one-shot warning when a fixed-endpoint/varying-endpoint compress probe family is detected (#370 breaker stays identical-args only)

- stale-ref cause text and FOLD BASE GENERATION MISMATCH body no longer assert cross-session origin
@ranxianglei
ranxianglei force-pushed the 2026-10-17_issue3302-sparse-ref-feedback branch 2 times, most recently from 74455b0 to 4745333 Compare October 17, 2026 09:20
@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-17_issue3302-sparse-ref-feedback (4870c29)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4205-pr.3533.6e5aa5c

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3533.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei
ranxianglei force-pushed the 2026-10-17_issue3302-sparse-ref-feedback branch from 4745333 to 5a6f07d Compare October 17, 2026 09:26
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

PR 分支已与最新 master 对齐:rebase 完成并已推送,冲突解除,PR 恢复可合并状态(等待新 head 上的 CI)。

冲突来源:master 在分支切出后合入了 132 个提交,其中多次 kernel 版本递进(0.0.127 → 0.0.128 → … → 0.0.130)与本 PR 自带的 kernel 版本 bump(0.0.127 → 0.0.128)在三个版本字段上正面相撞:kernel/package.json、kernel/package-lock.json、根 package-lock.json。源码文件(kernel/src/boundaries.ts、kernel/src/compress.ts、src/session.ts、src/stream.ts、src/loop/core.ts、src/acp-status.ts)无文本冲突——master 对同文件的并发改动(#3385 acp_status 降级原因、#3400 图片降采样、#3430 config epoch)落在不相交区域,自动合并干净。

解决方式:rebase 两笔提交到当前 master(3f513c2),版本字段统一取 acp-kernel 0.0.131(master 现值 0.0.130 的下一个;Kernel Boundary 规则要求 kernel/src 变更必须 bump 版本)。顺带修掉一个 master 存量小漂移:kernel/package-lock.json 停在 0.0.129 而 kernel/package.json 已是 0.0.130——本 PR 合并后三处统一为 0.0.131,漂移随之消除。

验证(最终树 5a6f07d,本地定向跑过):

  • typecheck ✅ / build ✅(dist-import-annotations OK,dist-external-zod OK)
  • 定向测试全绿:host 侧 19/19(tests/issue3302-sparse-ref-tombstones.test.ts + 3 个 compress-receipt/stream 套件)、kernel 侧 55/55(kernel/tests/issue3302-sparse-ref-unknown.test.ts + compress.test.ts + dead-ref/orphan-snap 相邻套件)、wire-contract gate 21/21
  • 全量回归按纪律交给远程 CI(新 head 已触发)

结构化八问:

  1. 是什么问题:PR 与 master 冲突(mergeable_state=dirty),卡在不可合并状态。
  2. 什么原因导致的:纯版本字段碰撞——master 侧其他 PR 连续 bump kernel 版本到 0.0.130,与本 PR 从旧基线(0.0.127)出发的 bump 撞在同一行;不是代码逻辑冲突。
  3. 影响范围与严重程度:仅调度层面(PR 无法合并),无用户可见行为影响;触发条件 = 长生命周期分支跨越多次 kernel 版本递进。
  4. 怎么修复的:rebase 到当前 master,版本统一取 0.0.131,force-push 本 PR 分支(自有分支,未触碰 master)。
  5. 是否有回归问题:无。diff 相对 master 仍是原来那 12 个文件、同样的 514+/28-,rebase 前后改动内容逐字节一致;master 并发改动落在不相交区域且全部被定向测试覆盖验证。唯一数值变化:kernel 版本号 0.0.128(原 PR 写法)→ 0.0.131,是 rebase 越过 master 递进的必然结果,无语义变化。
  6. 是否破坏缓存:不涉及——rebase 本身不触碰缓存/重试/游标/去重键逻辑。
  7. 是否改了设计方向:不涉及——纯 rebase,无架构或数据流转向。
  8. 是否引入了新的配置:无。

收工判定:冲突已解除,PR 回到正常 review/CI 流程;全量结论以远程 CI 在新 head(5a6f07d)上的结果为准。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 本 PR 是 #3302(human-opened,owner 已接单)的唯一实现 PR;检索 open+closed 的 issue/PR(按失败签名:sparse-ref 错误语义、compress 失败回执、tombstone/probe-family 关键词),无其他在途工作项针对同一机制。#3302 的六个 Ask 与本 PR 的八项改动一一对应,无第二份平行修复。

状态:这是早前会话已完成的工作([PR] 工单),我不重做——按 [PR] 规则拉分支核对代码 + 确认 CI + 给出审核结论。当前进度:

正在本地定向跑新增回归测试做最后验证,随后给出结构化八问结论。kernel 变更属 human-only review 层级,最终合并需要你拍板。

…3533 review nit)

The rebase hand-edit of the version field left packages."".version at 2-space
indent instead of the npm lockfile's 6-space level. Whitespace-only.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

PR #3533 审核结论(独立审 diff,未复用 issue 侧结论)

已做的核验:逐段读完全部 12 个文件的 diff(kernel src/tests ×5、host src ×4、host tests ×1、版本字段 ×2);对每个改动点回读改动前的完整函数与消费方(grep 全仓旧文案、golden/wire-contract pin、startsWith 匹配方);本地定向复跑新增回归 + 相邻标记消费方套件 62/62 绿(kernel:issue3302-sparse-ref-unknown + compress.test.ts = 38;host:issue3302-sparse-ref-tombstones + issue2360-compress-receipt + issue2432-breaker-awareness + issue2592-rebase-floor = 24);CI 在 5a6f07da4 上 48 个 check run 全绿。

顺手修掉一处小瑕疵(独立 commit 4870c296a,已推 PR 分支):kernel/package-lock.json 的 packages."".version 缩进错位(rebase 手改留下,2 空格应为 npm lockfile 的 6 空格层级)。纯空白,新 head 上 CI 会重跑确认。

结构化八问:

  1. 是什么问题:客户端 native compaction(或 zombie reap)把消息从会话视图里抹掉后,模型再引用被抹掉编号的 ref 时,bili 报的是「typo or wrong session」——一个错误归因。模型于是逐个编号探测有效 endId([compress] Still present after #156: the LLM also gets stuck on an invalid end ref; #370's same-parameter short-circuit fails to prevent it. | [compress] #156 之后仍存在:大模型同样卡在无效 end ref,#370 的同参数短路挡不住 #3302 日志:m10750→m10820→m10770→m10930✓→m11026✗),每次探测烧掉一整轮(最长 102s),而失败回执只给一句「Call acp_status」不给可用 ref 表,acp_status 报告每轮又被剥离,模型全程 0 次调用它。
  2. 什么原因导致的:三层机制缺陷叠加——生产层:pruneRefsToLiveIds(compaction archive omp 原生压缩后 bili 注册表跨代失配:syncBlocks 静默失活全部块、decompress 永久不可达(无视 omp session_compact 钩子;#19 搁置项的完整机制) #395 与 zombie reap [Bug][proxy] unframed host-side history deletion leaves zombie fold blocks rendering carriers forever — one carrier flaps in/out every turn (persistent prefix-cache churn) #2695 共用)删 byRef 条目时不留任何痕迹,kernel 的 compress failure is self-amplifying: failed calls are hidden as orphan refs, making the same range permanently un-compressible #2362 tombstone 机制只对 kernel 自己观察到的消失生效;语义层:kernel 把所有 byRef miss 一律断言为跨会话/笔误,且诊断行裸给 session highest ref=,字面意思就是「有效编号在下面」,直接邀请数字探测;反馈层:失败回执在无 live nudge 决策时退化为通用指引,不含任何可直接使用的 range。
  3. 影响范围与严重程度:所有经历过 native compaction/zombie reap 且后续引用了被 prune ref 的 proxy-mode 会话;压缩请求失败 + 探测循环烧整轮(单轮可达百秒级)+ 回执不可行动导致循环不收敛。特定条件下的严重效率退化,无数据丢失、无窗口违规。
  4. 怎么修复的:三层各打一层——host 两个生产者把被 prune 的 ref 记入 state.deadRefs tombstone(Set 去重、数值升序、空则 undefined,与 kernel 自身 sort 比较器一致);kernel 遇 tombstone 不再报错而是 snap 到最近可见消息(range 少折一点,note 明说),无邻居则诚实归为 consumed/DEAD 走既有 compress failure is self-amplifying: failed calls are hidden as orphan refs, making the same range permanently un-compressible #2362 回执路径;unknown 按位置三分法拆成「高于最高分配编号 = 旧世代残留(点名 frontier)」与「带内空洞 = 稀疏空间,明确不是跨会话」;all-unknown gate 点名失败 endpoint(failing endpoints: endId=m…)并加「Do not probe for valid numbers」;诊断行改为稀疏空间表述;回执在无 live 决策时回退到上一次 nudge 的 compressible-range 表(套用与 acp_status 表相同的 viable/dead-exclusion/submit-gate 管线);probe-family 一次性 [warn] 观测(Ask 6 保守选项,fix: break the small-model same-parameter compress failure loop early (#156) #370 breaker 保持 identical-args-only 未动)。
  5. 是否有回归问题:逐项清点触碰路径的既有行为——byRef 命中后的四条分支(visible/block-snap/hidden-orphan/consumed)字节级未动;bN block-ref 路径未动(其「typo or wrong session」措辞保留是对的:block id 不经 host prune、不复用,该断言在那里成立);标记短语 every ref is unknown to this session 逐字保留(stream.ts:362/398 的 includes 检测、plugin.ts 依赖它);stale-ref — 前缀(acp-status.ts:231 startsWith)与 FOLD BASE GENERATION MISMATCH — marker(status-body.ts/acp-panel.ts)保留;golden/wire-contract 无任何旧文案 pin;无 host 代码解析 boundary 错误文本(absorb.ts 那条是它自己的消息);tool-call 形状与 errors[] 位置不变;持久化格式不变(deadRefs 是既有可选字段);无 fail-fast→吞错、无阈值/时序漂移。CI 全绿 + 本地定向 62/62。未发现回归。
  6. 是否破坏缓存:不涉及缓存/重试/游标/去重键逻辑。诚实披露一点:state.deadRefs 现在可由 host 写入(此前仅 kernel 铸造),acp_status 的 DEAD REFS 段可能显示更多条目——全部为真(那些消息确实被移出视图);ref 在会话内永不复用(Kernel Contract),这些 tombstone 不会被误清,随 compaction 次数单调增长但受会话长度约束,量级可忽略。净效果是减少浪费的探测轮次,前缀改写更少。
  7. 是否改了设计方向:无架构转向。修复严格落在 [compress] Still present after #156: the LLM also gets stuck on an invalid end ref; #370's same-parameter short-circuit fails to prevent it. | [compress] #156 之后仍存在:大模型同样卡在无效 end ref,#370 的同参数短路挡不住 #3302 点名的三层机制上;两个可选项按保守形态采纳(tombstone 做了;breaker 不加 early-break,只加观测)。kernel 变更声明齐全(PR body 有 kernel-change: 段,[compress] Still present after #156: the LLM also gets stuck on an invalid end ref; #370's same-parameter short-circuit fails to prevent it. | [compress] #156 之后仍存在:大模型同样卡在无效 end ref,#370 的同参数短路挡不住 #3302 为 owner 接单的声明线程),版本 bump 0.0.130→0.0.131 符合 Kernel Boundary CI 规则,顺带消除了 master 上 kernel/package-lock.json 停在 0.0.129 的存量漂移。
  8. 是否引入了新的配置:无新 env/config/默认值。

测试有效性:两份新测试都是真测试——kernel 侧驱动真实 createCore().applyCompression,钉住三类措辞、snap 后 block 的 exact effectiveMessageIds(deepEqual)、DEAD 分类与 deadRefs 状态;host 侧走真实 reapDestroyedSubstrate/applyRanges 链路,钉住 tombstone 记录/合并/排序、refused/applied outcome、fallback 回执文本、recordProbeFailures 的 ≥2-distinct/same-endpoint/compress-only 语义;compress.test.ts 7 处旧断言全部换成钉新语义的断言(含 endpoint 点名),不是恒真替换。

分支状态:新 head 4870c296a(含空白修复),mergeable,落后 master 10 个提交但 git merge-tree 验证干净合并(BEHIND 非阻塞,GitHub 合并时自动处理)。新 head 的 CI 已触发,纯空白变更,预期与上一 head 相同。

收工判定:代码、测试、CI、回归审计四项全过,我这边可合。注意 kernel 变更属 human-only review 层级(Kernel Boundary 规则 3,§7.4 不可 auto-merge)——请 owner 过目后自行合并:#3533

@ranxianglei
ranxianglei merged commit eda8fb7 into master Oct 18, 2026
48 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.559-master.eda8fb7 (rolling master tag).

npm install -g billion-context@0.0.559-master.eda8fb7  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

@github-actions github-actions Bot mentioned this pull request Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant