Skip to content

[PR] fix: auto-mode classifier gate ignores marker strings quoted in history (#3908) - #3920

Closed
ranxianglei wants to merge 2 commits into
masterfrom
2026-10-20_auto-mode-classifier-gate-narrow
Closed

ranxianglei wants to merge 2 commits into
masterfrom
2026-10-20_auto-mode-classifier-gate-narrow

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

Fixes the #3908 silent kill: once a session's resent history contained BOTH the <transcript> marker and an auto-mode policy phrase (the reporter's case: a tool result carrying src/auto-mode-forge.ts itself), isAutoModeClassifier() matched on every subsequent request — 54/58 requests became byte-identical passthrough, processTurn/nudge/tag-rendering silently off, ACP state frozen at 0 blocks while billed input grew toward the window.

Root cause

The detector scanned UNBOUNDED history — system + all message text INCLUDING tool_result payloads + tools JSON — for the conjunction "transcript marker AND policy wording". The module's design assumption ("the conjunction cannot reasonably be faked by a coding session") was falsified by bili's own source file, which contains both families in one place (<transcript> literal + paired comment span + "Err on the side of blocking"). Once quoted into history via any tool result (or pasted into chat text), the match became permanent for that session.

Fix (decision point, not symptom)

Narrowed the gate to where the classifier actually places its signals. Single predicate (src/auto-mode-forge.ts); both call sites (src/server/handle.ts forge path, src/server/prepare-anthropic.ts passthrough path) share it.

  1. Structure — require a well-formed <transcript>…</transcript> pair in USER TEXT enclosing >= 512 chars (MIN_TRANSCRIPT_SPAN). A genuine wrapper carries the ENTIRE flattened session (KB..MB scale from the first classified-tool exchange); quoted artifacts (source literals, doc examples, comment spans) enclose at most ~100 chars.
  2. Placement — policy wording is searched in SYSTEM + user text blocks ONLY; tool_result payloads / assistant text / tools JSON no longer count (the one-shot classifier request carries no tool results). Magic stop_sequences stay a top-level field check.
  3. Bias — deliberately toward false negatives: a missed classifier forwards raw upstream (pre-auto mode 状态丢失,会话累计输入了593.44M tokens #2740 behavior: billed, safe); a false positive forges an allow verdict on a real turn or kills a session's compression.

Behavior change (old -> new)

Traffic Old New
Genuine CC classifier shapes (all stages/flavors, documented placements) detected detected (unchanged)
Session whose history quotes both marker families in a tool_result or chat text permanently misdetected -> silent no-compression passthrough (or forged verdict if a flavor marker also present) not detected -> normal compression resumes
Sub-512-char paired span + magic stops / policy phrase detected not detected (forwards raw upstream)

Residual risk accepted: a genuine classifier whose flattened transcript were < 512 chars degrades to raw forwarding — assessed implausible (CC's jsonl envelope makes even the first-exchange transcript KB-scale), and the degradation direction is safe.

No config surface change. No kernel/ change. No wire-shape change.

Tests

tests/auto-mode-forge.test.ts (22 pass; full-suite verdict via CI):

Closes #3908

@ranxianglei ranxianglei changed the title fix: auto-mode classifier gate ignores marker strings quoted in history (#3908) [PR] fix: auto-mode classifier gate ignores marker strings quoted in history (#3908) Oct 20, 2026
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-20_auto-mode-classifier-gate-narrow (579329b)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4685-pr.3920.591de01

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3920.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复——本 PR 是 #3908 唯一在途的修复项(正文 Closes #3908);扫描全部 open PR,没有其他触碰 auto-mode/classifier 检测的工作在途(#3746 threads-lane reforge、#3610 transient-4xx classifier 均为不同机制)。

标签: bug / claude-code / dsh / scope/wiring / p1(缺陷类;CC 与 DSH 两侧均受影响;压缩主路径被静默关停、计费照涨,按 owner 修订规则定 p1)。

审核进行中,先报第一轮结论:

方向关 ✅:根因分析独立复核成立。旧 isAutoModeClassifier() 对无界历史(system + 全部消息文本含 tool_result 载荷 + tools JSON)扫「transcript 标记 AND policy 措辞」的合取——任何 tool result 把两族标记引进历史(本次事故就是 bili 自己的源文件),该会话从此永久误判。修复把门收窄到分类器真正放置信号的位置:user 文本里成对 <transcript>…</transcript> 且跨度 ≥512 字符 + policy 措辞只查 system/user 文本 + magic stops 保持顶层字段检查。修的是判定决策点,不是沿路过滤载荷,方向正确。

回归清点:两个调用点(src/server/handle.ts:1917 forge 路径、src/server/prepare-anthropic.ts:78 passthrough 路径)共用同一谓词,diff 未触碰这两个文件;#3359 负例(magic stops 单独不触发)与 #353 不可判定→passthrough 语义均保留。行为变更表已在 PR 正文披露 old→new。

⚠️ CI 红(本轮主要发现):head a6e175c 上 3 个 test shard 失败(ubuntu-22 shard1 / windows-22 shard1 / ubuntu-24 shard1),三个 shard 失败签名完全一致——tests/e2e-anthropic.test.ts 里 4 个 #2740 e2e 回归测试挂掉:

  • auto-mode classifier forged locally by default — zero upstream egress
  • block flavor forged as bare <block>no</block>
  • no-stop shape (stage-2/fast) forged via policy+transcript detection
  • forged classifier call leaves ACP state byte-equivalent (acceptance item 3)

根因已定位:这 4 个 e2e fixture 用的是几十字符的占位 transcript(如 <transcript>\n[user] Is this command safe? rm -rf /\n</transcript>),低于新的 512 字符 span floor,被新门正确地不检测、转发到上游,「zero upstream egress」断言因此失败。真实 CC one-shot 请求的 wrapper 里装的是 KB 级压平会话——单测套件被升级成了 KB 级 fixture,e2e 套件漏了(典型首遍未覆盖全路径)。这是 fixture 真实性问题,不是门本身的问题。

处置:直接在 PR 分支推 additive commit,把这 4 处 e2e fixture 升级为 KB 级 transcript(与单测同等待遇),本地定向验证后 push 看 CI。完成后补完整八问结论。

…ans (#3908)

The #3908 span floor correctly rejects sub-512-char placeholder transcripts,
but four e2e fixtures in tests/e2e-anthropic.test.ts still used them, so the
new gate forwarded them upstream and the zero-egress assertions failed on all
CI shards. They now share one KB-scale CLASSIFIER_TRANSCRIPT constant — the
same treatment the unit suite's BIG_TRANSCRIPT already got. A genuine CC
one-shot carries the entire flattened session in the wrapper; sub-KB fixtures
were never realistic shapes of this lane.

Targeted local runs green: tests/auto-mode-forge.test.ts +
tests/e2e-anthropic.test.ts (33/33); full-suite verdict via CI.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 已转绿:在 PR 分支推了一个 additive commit(test: upgrade #2740 e2e classifier fixtures to KB-scale transcript spans,579329b),新 head 全量 CI 40/40 全绿(28 success + 12 skipped,含之前红的 ubuntu-22 / windows-22 / ubuntu-24 三个 test shard)。PR 状态:open、mergeable(behind master,非阻塞)。

审核发现的处置(上一轮报的 CI 红):根因不在门本身,在 e2e fixture——4 个 #2740 e2e fixture 用的是几十字符的占位 transcript,低于新的 512 字符 span floor,被新门正确地拒绝、转发上游,「zero upstream egress」断言失败。真实 CC one-shot 请求的 wrapper 里装的是整个压平会话(KB 级);单测套件升级成了 KB 级 fixture,e2e 套件漏了。修法 = 4 处共用一个 KB 级 CLASSIFIER_TRANSCRIPT 常量(与单测 BIG_TRANSCRIPT 同等待遇),本地定向验证 33/33 绿、全量 CI 绿。纯测试 fixture 真实性修复,无 src/ 变更。

结构化八问:

  1. 是什么问题:会话历史一旦同时引用了两族标记(<transcript> 标记 AND auto-mode policy 措辞——本次事故是 bili 自己的源文件被 tool result 引用),该会话后续每个请求都被误判为「CC 安全分类器 one-shot」而静默直通(或伪造裁决),压缩全停、计费照涨。
  2. 什么原因导致的:旧检测对无界历史(system + 全部消息文本含 tool_result 载荷 + tools JSON)扫「transcript 标记 AND policy 措辞」合取;「这个合取在编码会话里不可能被伪造」的设计假设被 bili 自己的源文件(两族标记同处一文件)证伪——引用进历史即永久生效。
  3. 影响范围与严重程度:任何把含两族标记的源文件/文档引用进会话的 CC/DSH 用户都会中招(isAutoModeClassifier() false positive: once <transcript> appears in the history, compression for the entire session is | isAutoModeClassifier() 误判:历史中出现 <transcript> 后,整个会话的压缩被静默跳过 #3908 实例 54/58 请求受影响);p1——压缩静默失效、计费输入逼近窗口,长会话实际不可用。正常流量行为不受影响。
  4. 怎么修复的:把门收窄到分类器真正放置信号的位置——user 文本里要求成对 <transcript>…</transcript> 且跨度 ≥512 字符 + policy 措辞只查 system/user 文本 + magic stops 保持顶层字段检查;forge 与 passthrough 两个调用点共用同一谓词;偏向假阴性——漏检退回 raw 转发(pre-auto mode 状态丢失,会话累计输入了593.44M tokens #2740 行为,计费但安全)。
  5. 是否有回归问题:逐项清点既有行为——① magic stops 单独不触发([bug] auto-mode forge detection intercepts on the stop-word arm alone — non-classifier requests answered by a locally forged allow verdict instead of the real model | [bug] auto-mode forge 检测停止词臂单独命中即拦截——非分类器请求被本地伪造 allow 判定替代真实模型回答 #3359 负例保留,绿);② flavor 不可判定 → verbatim passthrough(Auto mode classifier requires confirmation for this command. 3 consecutive actions were blocked. #353 纪律不变);③ forge 响应形状/事件顺序/stop_reason 语义不变;④ kill switch BILI_AUTO_MODE_FORGE=0 语义不变;⑤ 正常流量 wire shape 不变(diff 未触碰 handle.ts / prepare-anthropic.ts,只收窄谓词本体);⑥ CI 钉住的 4 个 auto mode 状态丢失,会话累计输入了593.44M tokens #2740 e2e 回归测试恢复绿。结论:无意回归为零;唯一行为变化是有意的收窄本身(old→new 已在 PR 正文披露)。
  6. 是否破坏缓存:不涉及。forge/passthrough 车道本就不进 kernel 状态;被毒化会话修复后恢复正常压缩,缓存行为随之复原;缓存/重试/游标/去重策略均无改动。
  7. 是否改了设计方向:没有。同一车道、同一拦截点、更窄的谓词。
  8. 是否引入了新的配置:没有(BILI_AUTO_MODE_FORGE kill switch 为既有条款)。

收工判定:可合并——方向正确(修决策点而非过滤载荷)、回归清点干净、全量 CI 绿。合并是 human-only:#3920

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

已并入合并列车 #4525(master 6d7c587,合并方式 merge commit,提交保留原作者署名)。本 PR 关单收尾;若列车中对应提交出问题,回滚点为 #4525 单提交 revert。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

1 participant