Skip to content

fix: dsh native-compaction detector requires user/text carrier + novelty (#2621) - #2628

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-10_dsh-native-compaction-fp-2621
Oct 10, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-10_dsh-native-compaction-fp-2621

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What / why

Fixes a false positive in the dsh native-compaction detector introduced with #2432. A long session whose history merely quoted the <compacted-summary> checkpoint string — without any host compaction actually happening — was misjudged as a landed native compaction: the ACP fold state was rebased/cleared and a misattributed native-compaction conflict was recorded (reported on v0.1.190, commit e1cd9e1).

In the reported session the transcript had zero compaction/ events and zero command/run compact, yet 36 lines carried the marker — all as quotes (a tool result that read another session's transcript, model reasoning citing it, bili's own compress tool-call args quoting it), never as its production. Combined with an ordinary fold-coverage gap from bili's own folding/drift, both detector signals were satisfied with no host compaction.

Root cause

carriesDshLocalCompactionSummary() matched the framing bytes on any resent message's text, ignoring who produced it. The detector ANDs two signals (a fold-coverage gap + "checkpoint framing present"); both are host-compaction-independent in this scenario, so the more folded / lower-retained a session is — i.e. the one most needing protection — the more likely the false positive. It is also self-reinforcing: investigating the false positive (reading the transcript) creates more carriers that keep it firing.

Fix

Tighten the framing match to "a checkpoint just landed" with two gates (both facets of one mechanism — detector precision):

  1. Carrier — the message must be role="user" AND plain text (contentType="text" when typed). Tool results, reasoning blocks, tool-call arguments, and assistant prose are structurally incapable of being a checkpoint and are excluded. On the Anthropic lane a tool_result rides inside a wire role:"user" message but flattens to role:"tool"/contentType:"tool-result", so the role gate alone already drops it; the contentType gate is belt-and-suspenders.
  2. Novelty — the framing message's raw id must not already be in the pre-turn ref map (Object.keys(session.state.messageRefs.byRaw) taken before the turn). A freshly-landed checkpoint is a NEW id; a historical quote/replay hits an OLD id. Closes the residual of gate 1 (a user who literally pasted the framing in an earlier turn).

Both call sites pass the set: prepare-openai.ts reuses its existing knownRefsBefore; prepare-responses.ts adds an equivalent pre-turn snapshot. When omitted (direct callers) the novelty gate is skipped and only the carrier gate applies.

Behavior change (intended)

  • Old: a resend carrying the marker bytes anywhere + a fold-coverage gap ⇒ ACP state rebased + native-compaction conflict recorded.
  • New: the same additionally requires the marker to sit on a fresh user-role plain-text message. A healthy session merely quoting the string no longer rebases; a genuine host compaction landing (a fresh user/message checkpoint) still does.
  • Net effect: strictly fewer, more-precise rebases. No change to the true-positive path (verified below).

No config surface added or changed. No kernel/ changes (host-side src/server only).

Tests

  • Unit truth table (tests/issue2432-dsh-local-compaction.test.ts): positive only on a user/text carrier; negative for assistant reasoning, tool result, assistant tool-call args, assistant prose, untyped messages, and the bare word "compacted"; plus a novelty test (old id ⇒ no match, fresh id ⇒ match, omitted set ⇒ carrier-only).
  • New e2e: seed+fold, then replay a decimated head where the marker survives only inside an assistant message ⇒ asserts no rebase and no misattributed conflict. Fails on the pre-fix code, passes here.
  • Existing positive case (real fresh user/text checkpoint ⇒ rebase) still green.
  • Local: typecheck clean; targeted suite 9/9 pass on the rebased head (current master 98fb91cc2). Full suite left to CI.

Closes #2621

Related: #2432/#2436 (the detector this targets); #2592/#2599 (anthropic/google lane wiring gap — unrelated).

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-10_dsh-native-compaction-fp-2621 (29fb444)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.2852-pr.2628.3915499

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2628.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到,按 [PR] 流程做了核对(拉分支读代码、查 CI、查原 issue),不重做。先给结论:这个 PR 本身逻辑是对的,但它和已经合并的 #2629 重叠,且基于过期 master + CI 有一处红,现状不可合并。 下面是结构化八问 + 关键发现。

结构化八问

  1. 是什么问题:dsh native-compaction 检测器误报(FP)。一个历史里只是「引用」了 <compacted-summary> 框架串的会话(工具结果读到别的 session transcript、模型 reasoning 引用它、bili 自己 compress 的 tool-call args 引用它),被误判为发生了 host native compaction → ACP fold 状态被 rebase/clear,并记一条误归因的 native-compaction conflict。调查过程会制造更多 carrier,自我强化。

  2. 什么原因导致的:carriesDshLocalCompactionSummary() 在任意 resend message 文本上匹配框架字节,不看生产者是谁。检测器 AND 的两个信号(fold-coverage gap ≥ 8 + 框架存在)都与 host compaction 无关,所以引用串也能凑齐两条件 → FP。

  3. 影响范围与严重程度:特定条件才触发——历史里出现该框架串引用、且同时满足 fold-gap 阈值。后果是 fold 状态被错误重置 + 一条误导性 conflict 记录(非数据损坏,但污染会话状态并自强化)。严重度:中。

  4. 怎么修复的:两道门(同一机制的两面):① CARRIER —— 只认 role=user && contentType=text(anthropic tool_result 走 kernel 展平成 role=tool/contentType=tool-result,天然排除);② NOVELTY —— raw id 必须不在 pre-turn ref map(Object.keys(session.state.messageRefs.byRaw))里。两个 call site(prepare-openai.ts 复用已有 knownRefsBefore;prepare-responses.ts 新增 knownIdsBeforeLocalCompact snapshot)都传入。我已核验时序不变量在两 lane 都成立(snapshot 取于 detector 之前、processTurn 分配新 ref 之前):openai line 109→149→169,responses line 225→227→254。所以 fresh checkpoint 是 novel(TP 保留)、旧引用命中 knownIds(FP 消除);codex lane 走独立函数 carriesCodexLocalCompactionSummary,未受影响。

  5. 是否有回归问题:

  6. 是否破坏缓存:不涉及。改动只影响 native-compaction 判定,不改 wire 结构,prefix cache / 增量游标不受影响。

  7. 是否改了设计方向:不涉及。纯判定精度收紧,无架构转向。

  8. 是否引入了新的配置:不涉及。无新 env/config/默认值。

关键发现:与已合并的 #2629 重复(同源 issue #2621)

建议(owner 拍板)

收工判定

不可合并(stale base + CI red + 与已合并 #2629 重叠)。请 owner 二选一:① rebase-keep(我可在你确认后执行 rebase + 定向验证,不擅自 push);② close-redundant。无论哪种,都需要先定位 test (windows-latest, node24) 的失败原因(建议 CI 重跑一次看是否复现,若是 flake 则与本 PR 无关)。

—— 以上仅审查结论,未做任何 push/merge。

…lty (#2621)

carriesDshLocalCompactionSummary matched the checkpoint framing bytes on ANY resent message, so a session whose history merely quoted the marker (a tool result that read a transcript, model reasoning citing it, the bili compress tool-call args quoting it) plus an ordinary fold-coverage gap was misjudged as a landed native compaction: ACP fold state rebased/cleared and a misattributed native-compaction conflict recorded (v0.1.190, #2621).

Two gates tighten the match to the case where a checkpoint just landed: (1) CARRIER - role=user AND plain text (contentType text when typed); tool results, reasoning, tool-call args, and assistant prose are excluded. (2) NOVELTY - the framing message raw id must not already be in the pre-turn ref map; a freshly-landed checkpoint is a NEW id while a quote/replay is old. Both call sites pass the set (prepare-openai reuses knownRefsBefore; prepare-responses adds a pre-turn byRaw snapshot). The positive landing path is unchanged and still green.

Regression tests: carrier+novelty unit truth table plus an e2e proving a marker quoted by a non-user carrier together with a real fold gap does NOT rebase.
@ranxianglei
ranxianglei force-pushed the 2026-10-10_dsh-native-compaction-fp-2621 branch from a2043ce to 29fb444 Compare October 10, 2026 09:03
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到上游冲突唤醒。先独立核实了前提(没盲从那段被标记为不可信的转发文本):PR #2628 mergeable=false / dirty,head a2043ce6、base 98fb91cc2 确实过期。已把该 PR 分支 rebase 到当前最新 master → 解冲突 → 本地定向验证 → force-push 全部完成。

做了什么

结构化八问

  1. 是什么问题:dsh native-compaction 检测器误报 —— 历史里只是「引用」了 <compacted-summary> 框架串(工具结果 / reasoning / tool-call args)被误判为发生了 native compaction,fold 状态被错误重置并记一条误导性 conflict。(同上次 review)
  2. 什么原因导致的:检测器在任意 resend message 上匹配框架字节、不看生产者;两道门(carrier 只认 user/text + novelty id 不在 pre-turn ref map)收紧判定。
  3. 影响范围与严重程度:特定条件才触发(历史含框架串引用且满足 fold-gap 阈值);后果是 fold 状态被错重置 + 误导性 conflict(非数据损坏,但自强化)。中等。
  4. 怎么修复的:同一机制两道门;openai/responses 两 lane 都传 pre-turn ref map,codex lane 走独立函数不受影响。本次 rebase 未改任何逻辑,只按「取超集 / 取并集」原则消解与 fix: gate dsh compaction framing detector on user text messages (#2621) #2629 的重叠。
  5. 是否有回归问题:逻辑无回归(判定更严 → rebase 更少);rebase 本身是机械操作,已核验 —— 恰好 2 个 production caller 都传了第二参、全仓无残留冲突标记、类型成立。本地 typecheck 通过 + 定向 dsh-compaction 套件 11/11 绿。⚠️ 上一 head 的 test (windows-latest, node24) 红因在本环境无法定位(日志接口 404),新 head 会触发全新 CI,以 CI 为准。
  6. 是否破坏缓存:不涉及(只改判定,不动 wire / prefix cache / 增量游标)。
  7. 是否改了设计方向:不涉及(纯精度收紧)。
  8. 是否引入新配置:不涉及(无新 env/config/默认值)。

待 owner 拍板(结论不变)

相对当前 master,这个 PR 的净增量已收敛为「novelty gate + 接线 + 测试」。novelty gate 守护的是一个未在报告 repro 中出现过的窄残留(user 在更早的 user/text turn 逐字粘贴框架串),而 #2629 的 carrier gate 已经修好报告场景。所以仍是二选一:

收工判定

冲突已解决,分支已 rebase 到最新 master 并推送,预期 mergeable 转干净(基于最新 master 且无文件重叠)。本地 typecheck + 定向测试绿,但全量 CI 尚未在新 head 上跑完(尤其 windows-node24 要看这次是否复现)。请 owner 定 keep / close;若 keep,等本轮 CI 全绿即可合并。

—— 仅完成 rebase / resolve / push + 定向验证,未做任何 merge。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 推送后核实了实际状态(补上一条里「预期转干净」的实测):

  • PR fix: dsh native-compaction detector requires user/text carrier + novelty (#2621) #2628 现在 mergeable: true、mergeable_state: behind —— 冲突已解决(不再是 dirty)。
  • head = 29fb444c2(我推的新提交)。之所以是 behind 而非 clean:rebase 期间 master 又前进了(现 7c993f9ed,超过我 rebase 所基于的 2d31e4212)——这仓库此刻很活跃。behind ≠ 冲突,且 mergeable: true 说明即便对着最新 tip 也能干净合并,不需要再 rebase 来消冲突。
  • 若想合并前钉到绝对最新的 tip,我可以再做一次 touch-up rebase(但 master 可能还会动);默认我不动,等你拍板。
  • 全量 CI 正在新 head 29fb444c2 上跑,重点看 windows-node24 这次是否复现。
  • keep / close 的决定仍等你;若 keep,等本轮 CI 全绿即可。

@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.234-master.d101fb5 (rolling master tag).

npm install -g billion-context@0.0.234-master.d101fb5  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] dsh 原生压缩检测误报:历史里引用过 <compacted-summary> 即判为宿主压缩落地并 rebase ACP 状态(0.1.190)

1 participant