Skip to content

fix: strip refs-only residue adjacent to render tags in the echo filter (#2023) - #2025

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-04_tag-echo-refs-residue
Oct 6, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-04_tag-echo-refs-residue

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #2023

What & Why

The echo filter only treated a body of EXACTLY ONE bare ref as tag content (#1720 REF_LIKE), so refs-only text adjacent to render-tag structure was released as prose — which is how mNNNNN remnants leaked into the client reasoning stream after compress confirmation markers (which enumerate ref ranges).

Leak shapes confirmed on master (reproduced in-process before fixing):

  1. orphan ref + close: m01233</acp> — close stripped alone, ref leaked (whole-text + streaming)
  2. refs-only pair bodies: multi-ref lists, en/em-dash ranges — the whole list leaked
  3. EOF tails after an opening (bare or attrs) with no close — released as prose
  4. chunk-boundary splits: ref ends chunk N / close starts N+1 (incl. mid-ref splits) — ref already emitted

Fix (one mechanism, every decision point)

Behavior changes (old -> new)

shape old new
refs-only pair body (multi-ref/range) body released as prose stripped as tag content
orphan refs run + unmatched close close stripped, ref leaked unit stripped together
EOF refs-only tail after an opening released as prose discarded (imitated marker list, not a citation)
over-budget (>80B) refs-only swallow tail released losslessly discarded
chunk-boundary split around an orphan unit ref leaked held toward the unit, stripped whole

Unchanged (pinned by tests): well-formed single-ref pairs; prose-wearing pairs keep their content incl. cited refs (#1720); word-like xm01233; kebab elements; #644 lossless release of MIXED over-cap tails; lone trailing ref with NO tag context at EOF still released (genuine citation).

Both compression modes reasoned: the filter acts on model PROSE fields only (content/reasoning/thinking/summary) — tool-call arguments stay byte-exact (#1039); plugin mode and proxy mode consume the same createTagEchoFilter/stripAcpTags paths.

Documented residuals (out of scope, pre-existing)

Tests

New tests/issue2023-refs-residue.test.ts: every shape pinned whole-text AND streaming (char-by-char equivalence over all shapes + explicit chunk splits), plus #1720 protection pins, #644 lossless pins, residual pins. Targeted family green locally (200 tests across the 9 suites touching this module); full suite via CI.

Config surface: none.

…er (#2023)

The filter only treated a body of EXACTLY ONE bare ref as tag content
(#1720 REF_LIKE), so refs-only text adjacent to render-tag structure was
released as prose: orphan ref+close units, multi-ref/range pair bodies,
EOF tails after an opening, and chunk-boundary splits all leaked mNNNNN
remnants to the client reasoning stream.

- Generalize single-ref to refs-run (REFS_RUN/REFS_ONLY: lists, en/em-dash
  ranges, comma-separated range lists) at every decision point: PAIRED
  body, swallow inner decision, over-budget release, flush EOF branch,
  PARTIAL_TAIL hold (incl. run+tag-fragment composite and bare-m/partial
  ref prefixes so any split position holds toward the unit).
- New orphan unit: a refs run directly against an UNMATCHED close is dead
  markup and is stripped with the close. Whole-text applies it only to
  depth-zero closes (orphanCloseSpans greedy pairing); streaming only when
  no opening is live — so a ref cited inside a prose-wearing pair survives
  (#1720) in both modes.
- The [\w>] lookbehind extends across chunk boundaries via lastEmitted:
  word-like tokens (xm01233) and tag-terminator residue (#644 over-cap
  drop) are never orphan-stripped or held.

Tests: tests/issue2023-refs-residue.test.ts pins every shape whole-text AND
streaming (char-by-char equivalence + explicit chunk splits), plus #1720
protection pins, #644 lossless pins, and documented residual pins.
@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-04_tag-echo-refs-residue (27ba88a)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.2059-pr.2025.22a1b90

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2025.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到,这是本 daemon 针对 #2023 提交的 PR。开始核验:拉取分支核对代码、跑定向测试、确认 CI 状态,完成后按结构化七问给结论。

…ounting (#2023)

Review finding on the #2025 head: the three new PARTIAL_TAIL refs-run
alternatives flow into mayStartRenderTag, which isOrphanMarkupText used as
its render-tag probe. A released tail shaped like a refs run (a genuine
citation held to EOF) was therefore counted as dead markup residue, so a
tool-call-free turn whose only visible output was a bare ref citation read
as degenerate and burned the one-shot retry (#732/#821) — and could end in
the #870 in-band error instead of delivering the citation.

isOrphanMarkupText now probes with TAG_PARTIAL_TAIL (PARTIAL_TAIL's
\x3c-prefixed alternatives only): byte-identical classification to pre-#2023
for every tag-shaped tail, refs-run tails flip to prose. Tagged echoes need
no help from this path — their drop already sets sawStrippedEcho upstream.
mayStartRenderTag stays broad (fast-path gating must keep engaging on
refs-run tails so split units are still stripped).

Pinned: unit classification + a full openai-lane pipe test proving a
bare-citation answer reaches the client with zero refetches; both fail on
the previous head. #870 unclosed-tag retry pin stays green.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 ## PR review — 结论:发现并修复了一处未披露的跨模块语义副作用,已补提交到本分支,CI 绿后可合

做了什么:拉取分支核对代码(未重做)。逐条验证了 diff 触碰路径的全部既有行为、流式状态机各决策点、whole-text/streaming 等价性,审了测试有效性。审查中发现一处未列在 PR 行为变更表里的回归,按「小问题直接修」政策已在 PR 分支补一个 additive commit:27ba88af(2 files, +35/-1),定向验证过(117 tests 全绿 + typecheck)。

发现的问题(审查新增)

PARTIAL_TAIL 新增的三个 refs-run 备选会流入 mayStartRenderTag,而 isOrphanMarkupText(src/loop/tag-echo-filter.ts:504)正是拿它当渲染标签探针——它的作用是把「EOF 时释放的 held tail」分类为 dead markup,喂给 chat 管道的 degenerate-turn 记账(releasedMarkupChars,src/plugin.ts:1474/1660/1851 → retryEmptyTurn 门)。后果:一个不带工具调用、可见输出整体就是一个裸引用(如回答恰好是 m01233)的正常回合,会被判为退化回合,烧掉一次性续写重试(#732/#821);若重试也退化,用户看到的是 #870 in-band error 而不是那个合法引用。旧代码里这种 tail 既不会被 hold 也不会被分类,不触发重试。Responses 管道不受影响(其重试门要求 visibleTextChars===0,flushTail 无条件计数)。带标签的真 echo 不受此修复影响——它们的 drop 早已置位 sawStrippedEcho,重试保护本来就在。

修法:把两个探针解耦——mayStartRenderTag 保持宽(快速路径门控必须继续对 refs-run tail 挂接,分块单元才能整段剥离);isOrphanMarkupText 改用新的 TAG_PARTIAL_TAIL(仅 < 前缀备选),对所有标签形 tail 与修复前逐字节同判,只有纯 refs-run tail 从 residue 翻转为 prose。回归钉死:单测分类断言 + openai 全管道测试(裸引用回合零 refetch、引用完整到达客户端)——两条钉在修复前的 head 上均复现失败(已实际验证),#870「未闭合标签仍重试一次」的原钉保持绿。

结构化七问

  1. 是什么问题:PR 本身修的是 tag-echo filter occasionally lets m01233</acp> remnant through to the client reasoning stream (model echoes refs from compress confirmation markers) #2023(渲染标签旁的 refs-only 文本泄漏成 prose);本轮 review 的问题是——该修复顺带改变了 degenerate-turn 记账对「EOF 释放的 refs-run tail」的分类,且未在行为变更表中披露。
  2. 什么原因导致的:broad-hold/strict-decide 纪律在 decide 侧被破坏:为 hold 而放宽的 PARTIAL_TAIL(hold 探针,放宽正确)被 isOrphanMarkupText 复用为严格分类器,导致「合法的可见引用」被误判为「对宿主无用的死标记」。
  3. 怎么修复的:探针解耦——hold 侧保持宽,分类侧收窄回修复前的 < 形集合(TAG_PARTIAL_TAIL),对非 refs tail 判定与修复前完全一致;依赖 sawStrippedEcho 已有的覆盖,不损失任何真 echo 的重试保护。
  4. 是否回归问题:逐项清点完毕——缺陷:PAIRED 正则吞掉最多 64 个字符的助手正文 #1720 prose-wearing pair(含引用保留)、xm01233 词形、kebab 元素、[bug] #361 residual, form 3: turn whose entire text is a render-tag echo strips to an EMPTY message — agent records a marker-only turn, stalls until manual nudge (v0.1.92, post-#365) #644 over-cap mixed lossless、tag-echo filter leaks bare <ac / <ac> render-tag fragments to the client (bare truncated open tag survives both process() and flush()) #1755 bare-fragment(owner-held,未触碰)、Plugin passthrough leaves the client an empty turn when the model echoes a render tag #870 未闭合标签重试契约、[bug] 单行中文答案被误判为退化轮:marker 行首前缀按设计被扣住,终点释放却被计成 markup(chat-completions 线整轮报错) #1760 CJK/prose tail 分类、快速路径字节一致透传(更宽的挂接只增加 no-op 过滤,输出字节不变)、工具参数([bug] 插件透传会剥掉工具调用参数里的 render tag(arguments 流挂了 tag-echo 过滤器) #1039,过滤只作用 prose 字段)——全部保持。PR 披露的 5 项行为变更经代码逐条核实与实现一致。修复后未发现无意回归。
  5. 是否破坏缓存:不涉及。未触碰会话持久化、缓存、重试游标、去重键;过滤只在流内作用于模型 prose 字节。
  6. 是否改设计方向:不涉及架构转向;是在既有 broad-hold/strict-decide 框架内恢复 [bug] 单行中文答案被误判为退化轮:marker 行首前缀按设计被扣住,终点释放却被计成 markup(chat-completions 线整轮报错) #1760 确立的纪律(hold 宽、decide 严)。
  7. 是否引入新配置:不涉及。config surface 零变化。

测试与 diff 质量

新套件(tests/issue2023-refs-residue.test.ts)是真测试:字节精确断言、全形状 char-by-char streaming≡whole-text 等价性 harness、显式分块切分、#1720/#644 保护钉、文档化残留在册钉,无恒真断言。diff 干净(模块 + 其测试文件,我补的 commit 同样收敛到 2 文件)。

状态与判定

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 CI 全绿确认:head 27ba88af 上 15/15 check run 全部 success(含全平台 test、e2e、registry e2e、version-guard)。

更正上一条中的一处状态:等待期间 master 前进了 8 个提交(合入的 #2019 provider protocol field 与 #2020 opencode-v2 routed-warn)。已核对:两者触碰路径(src/server.ts、src/config.ts、src/web/api.ts、src/agent/opencode-native.ts、文档)与本 PR(src/loop/tag-echo-filter.ts + 两个测试文件)零交集,无共享模块语义交互;GitHub 对新 master 重算后 mergeable=true。按「BEHIND 不是 blocker」与避免空耗一轮 CI 的考量,不做无谓 rebase。

最终判定:可合并(人工操作):#2025 —— 修复完整、披露充分(含审查中补修的 degenerate-turn 记账项)、CI 绿、diff 干净。方向二(#2023 floor 3,压缩确认标记减少 refs 枚举)仍待你拍板,不阻塞本 PR。

ranxianglei pushed a commit that referenced this pull request Oct 6, 2026
…tack compat)

One-line compat for the agreed #2025 -> #2211 -> #2067 merge order: #2025 folds the
master REF_LIKE constant into REF_TOKEN/REFS_RUN, which would leave this call dangling
(silent tsc break once stacked, each PR green alone). Standalone const carries the
identical regex /^\s*m\d{4,}\s*$/; behavior unchanged.
@ranxianglei
ranxianglei merged commit 58b064e into master Oct 6, 2026
15 checks passed
@ranxianglei
ranxianglei deleted the 2026-10-04_tag-echo-refs-residue branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tag-echo filter occasionally lets m01233</acp> remnant through to the client reasoning stream (model echoes refs from compress confirmation markers)

1 participant