Skip to content

[PR] fix: keep ordering-repair separators out of compression range analysis - #5143

Merged
ranxianglei merged 6 commits into
masterfrom
2026-10-29_turn-separator-range-endpoints
Oct 29, 2026
Merged

ranxianglei merged 6 commits into
masterfrom
2026-10-29_turn-separator-range-endpoints

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Closes #5139

Root cause

repairResponsesAssistantOrdering() (src/server.ts) inserts acp_turn_sep_* user markers into the Responses send view only to keep assistant ordering legal upstream; they never exist in the original history (the apply view).

The status/recovery entry points analyze the send view while screening candidates against the apply view:

  • liveCompressionRanges() (src/acp-status.ts): processTurn({ messages: <send view>, applyView: <original> })
  • uncoveredHistoryRuns() (src/uncovered.ts) — same wiring via the acp_status audit section
  • the post-compress recovery re-run (src/stream.ts postCompressTail)

During that analysis run the markers get transient refs, and the segmenter can start a candidate unit at a marker (a marker is a user message; user messages start new groups). Screening then runs projectUnit() against the apply view, where the marker id does not exist → BoundaryNotFoundError → screenRecommendedUnits() demotes the whole unit to a gap silently. Every range behind such an endpoint disappears from the offer while the uncovered-history audit still shows it going upstream raw — exactly the reported split (~182K uncovered vs ~12K offered, "little left to fold"), while prepare-time logs correctly saw pendingT1≈180K because prepare runs on the original history without markers.

Fix

Strip outbound-only markers from the analyzed array before processTurn — new withoutTurnSeparators() in src/turn-separator.ts (zero-copy when absent), applied at the three analysis entry points above (+ re-export from session.ts). Real-message refs are id-stable across views, so every offered endpoint resolves in the apply view again. Matches the reporter's offline experiment: removing the separators from the analysis view restored 10 ranges / 199,434-token candidates.

Untouched: the ordering repair itself (send wire unchanged), tool pairing, turn-integrity protection, kernel/, config surface.

Behavior change (intended)

  • Ranges previously dropped behind separator endpoints reappear in acp_status / compress-receipt offers (old: silently demoted → new: offered when resolvable in the apply view).
  • Consequence: the "little left to fold" pressure note stops firing while a large pending mass is actually offerrable; maxPending reflects the true pendingT1.

Tests

New tests/issue5139-turn-separator-ranges.test.ts (4 pins, wired exactly like the relay: send view as messages, original as compressMessages):

  • steady-state acp_status: full recoverable mass offered, every endpoint resolves in the apply view, offer and uncovered audit agree (no spurious UNCOVERED section);
  • post-compress internal re-request (round-1 send view vs post-fold state): mass around the new block stays offered;
  • unit pin for withoutTurnSeparators (no-op zero-copy + strip);
  • plain control: no separators → behavior unchanged.

Pre-fix verification by temporarily reverting the call sites: the offer collapses to the pre-separator head (["m00001-m00003"]) — the reported symptom reproduced. Related suites green locally (issue564 ordering, issue2627 sep coverage, issue4331 uncovered, acp-status*, compress-recovery-ranges, ccr-v2, marker-render-contract, search-plan-aware, …); full-suite verdict on CI.

ework-agent and others added 4 commits October 29, 2026 14:46
…ession persistence (#4599)

The four native-lane e2e suites SIGTERM the hermetic proxy and then wait a
fixed 500ms before reading persisted sessions from disk. That races the
product's own shutdown order: SIGTERM -> server.close() drains keep-alive
connections first (SHUTDOWN_DRAIN_GRACE_MS = 10s backstop) and only then
flushAllSessions() runs, so on a slow socket teardown the 500ms read lands
pre-flush and the suite flakes ("proxy must persist the session"). Seen in
the e2e-native-opencode scheduled lane at 5594c13 (run 40751086427 red,
same-sha rerun green).

Process exit implies flushAllSessions completed, so waiting for the recorded
instance pids to actually exit makes the disk assertions deterministic:

- e2e-native-opencode / e2e-subagent-sessions: reuse the file's existing
  awaitProxyExit() bounded poll (20s budget > 10s drain grace).
- e2e-native-pi / e2e-native-dsh: extract liveInstancePids() and poll it
  until empty (same 20s budget).

e2e-pi-codex-ws's fire-and-forget variant is untouched: it only runs in
teardown hooks and gates no disk assertion.

Verified locally: tsc --noEmit clean; the previously flaking subtest passes
4/4 consecutive hermetic runs (zero tokens). Full-suite verdict: remote CI.
#4599)

Master red at 1fad21d (run 40852517737, 6 shards): two mechanisms, both in
the #5107/#5115/#5120 hermetic data-root cluster.

1. preload-test-env.mjs pinned BILI_CONFIG_FILE to an absent file.
   configFile() (src/paths.ts) consults that env BEFORE the XDG default, so
   the pin shadowed every fixture that re-points XDG_CONFIG_HOME at its own
   dir and writes the default config path: the claude-native attach-gate /
   lane-priority / policy / url file-layer assertions failed on all shards,
   and the #3436 compat checks resolved mitm.domains=[] -> spawn path -> 60s
   fake-child timeout. The XDG_CONFIG_HOME pin already keeps the default
   config path inside the throwaway root; DELETE the env instead of pinning
   so fixtures are un-shadowed AND a developer's ambient pin is dropped. The
   isolation probe now asserts configFile() resolves UNDER the sandboxed
   XDG_CONFIG_HOME (same property, sibling-check style) instead of equality
   with the removed pin.

2. test-suite-lock.mjs snapshotSessionsTree keyed entries by raw
   path.relative(): Windows yields 'anthropic\\s.json' while the diff, the
   banner and the tests all compare '/'-separated paths (4 failures on the
   windows s2 shards). Normalize keys to '/' at the source.

Verified locally with the CI preload chain (targeted): claude-native +
#3436 + test-suite-lock suites 96/96 pass, 0 fail. Windows separator
behavior holds by construction (split(path.sep).join('/')); full verdict =
remote CI.
#5141)

repairResponsesAssistantOrdering inserts acp_turn_sep_* user markers into the
Responses SEND view only; they never exist in the original history (the apply
view). liveCompressionRanges / uncoveredHistoryRuns / the post-compress
recovery path re-run processTurn on the send view while screening candidates
against the apply view, so any candidate unit that endpoints on a marker ref
throws BoundaryNotFoundError in projectUnit and screenRecommendedUnits
demotes it to a gap silently — large amounts of offerrable history vanish
from the acp_status offer ("little left to fold" while ~180K was recoverable).

Analysis entry points now strip outbound-only markers before processTurn
(withoutTurnSeparators, zero-copy when absent). Real-message refs are
id-stable across views, so every offered endpoint resolves in the apply view
again. Ordering repair, tool pairing and turn-integrity protection are
untouched — only the analyzed array class changes.

Behavior change: ranges previously dropped behind separator endpoints
reappear in acp_status / compress-receipt offers; the "little left to fold"
pressure note stops firing while a large pending mass is actually offerrable.
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-29_turn-separator-range-endpoints (44cd192)

Latest build — install this exact version

npm install -g billion-context@0.0.6778-pr.5143.bcf580e

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
44cd192 billion-context@0.0.6778-pr.5143.bcf580e build
9e10fcf billion-context@0.0.6776-pr.5143.0f69575 build
7bf8396 billion-context@0.0.6774-pr.5143.0002c26 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5143.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: keep ordering-repair separators out of compression range analysis [PR] fix: keep ordering-repair separators out of compression range analysis Oct 29, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — open/closed issue 与 PR 中均无修复此缺陷(排序修复分隔符污染压缩范围分析)的在途工作;最近邻 #5141(cold fork snapshot 恢复,不同机制)、#4266(fork 家族)均不重叠。

标签: bug + codex + scope/kernel + p2(Responses 车道 ordering-repair 分隔符静默吞掉 acp_status/compress offer — 功能性缺陷;prepare 自动压缩路径不受影响、无数据丢失/缓存泄露 → p2)。

编号澄清(避免找错 PR):本镜像 issue 对应 GitHub PR #5143(ranxianglei:2026-10-29_turn-separator-range-endpoints,head 7bf8396);GitHub #5144 是另一个 PR(cold fork snapshot,Closes #5140)。另发现一处需要确认:PR 正文 Closes #5141 指向的是 @abworks-dev 在途的 fork-snapshot PR(GitHub 侧),合并本 PR 会自动关掉那个无关的外部 PR——疑似引用了错误编号,请确认真正的 bug report issue 号(或确认是否有意)。

CI 初查(单次拉取):静态门禁全绿(kernel-guard / config-surface-guard / lane-guard / lock-sync / docs-sync / build-artifact…);test 分片:windows-22 的 shard 1、2 已红,其余仍在跑。完整结论随结构化 review 跟进。

@ranxianglei ranxianglei added bug Something isn't working codex p2 scope/kernel labels Oct 29, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核结论(结构化八问)

已独立拉取分支通读全部 6 个文件的 shipped bytes(非照抄 PR 描述),并清点所有 processTurn() 调用点与全部 applyView 使用点。

  1. 是什么问题:Responses 车道的出站顺序修复会把 acp_turn_sep_* user 标记只插进「发送视图」;而 acp_status / 压缩回执的范围分析拿发送视图切候选、却按「原始历史(apply 视图)」做投影筛查——以标记为端点的整段候选被静默降级成 gap。结果:~182K 可折叠质量从 offer 里消失,但仍在原样上行(面板显示 little left to fold,实际大量可折)。
  2. 什么原因导致的:KDD refactor: reapOrphanBlocks should live in acp-kernel, not the proxy #12([Bug] recommend/apply 对同一区间的 turn-integrity 判定永久不一致:acp_status 反复推荐的区间被 apply 侧整段拒绝(0.1.196,Range would split N turn(s) at the protected-zone boundary) #3236)共享投影契约下的视图类错配——分段跑在发送视图、projectUnit() 筛查跑在 apply 视图,而标记 id 只存在于前者 → BoundaryNotFoundError → screenRecommendedUnits() 把整个 unit 静默降为 gap(无日志、无回执)。是分析入口的输入类错误,不是压缩引擎或账本的问题。
  3. 影响范围与严重程度:仅 Responses 协议且顺序修复真实插入了标记的会话(折叠后出现相邻 assistant 连发时)。属功能性降级而非不可用——prepare 侧仍按原始历史看到全量 pending(报告者日志 pendingT1≈180K 为证),核心折叠能力未停摆;受损的是交互式 offer 面与 UNCOVERED 审计的一致性。p2 定级合理(已打标)。
  4. 怎么修复的:在三个跨视图分析入口处,processTurn 之前先把出站专用标记从被分析数组中剥掉——新增 withoutTurnSeparators()(src/turn-separator.ts,无标记时零拷贝原样返回),应用于 liveCompressionRanges(acp-status.ts)、postCompressTail(stream.ts)、uncoveredHistoryRuns 入口(uncovered.ts,同时覆盖 acp-status:310 与 stream:649 两个内部调用方)。发送线本身不动,顺序修复照常生效。
  5. 是否有回归问题:无。逐项审过:①无标记会话(绝大多数)函数返回同一数组引用,行为逐字节不变——控制测试钉住;②有标记会话唯一变化就是原先被丢的区间重新出现在 offer(PR 正文已披露旧→新);③UNCOVERED 审计的条目扫描不受影响(标记从无持久化 ref,refForRaw 本来就跳过它们);④压力提示/阈值/时序/输出格式零漂移(processTurn 此处是只读重算,返回 state 不被采纳);⑤fail-fast 方向是反的——本修正是移除静默吞错,不是引入吞错;⑥插件车道(plugin.ts:1746)核实不受影响:标记只在状态采纳之后才被合成进出站 payload,进不了插件记忆;其余 processTurn 调用点(preflight×4、prepare×N、relay/server 各点)逐一核对,均不存在「带标记发送视图 + applyView 筛查」的组合。高危模式逐项过完,无意回归。
  6. 是否破坏缓存:不涉及——只改分析输入,前缀缓存所依赖的上行字节一个没动(这正是修复刻意保持的部分)。
  7. 是否改了设计方向:不涉及——是把既有的 [Bug] recommend/apply 对同一区间的 turn-integrity 判定永久不一致:acp_status 反复推荐的区间被 apply 侧整段拒绝(0.1.196,Range would split N turn(s) at the protected-zone boundary) #3236 共享投影纪律补到最后一个漏掉的视图类错配实例上,架构/边界/数据流不变。
  8. 是否引入了新的配置:不涉及——零新 env/config/默认值。

测试有效性(④):4 个 pin 全部有牙——复现前置条件双向断言(有标记会话断言标记确实进了发送视图;控制组断言确实没有,#3617 门的双向纪律)、验收契约为「每个 offer 端点必须在 apply 视图可解析」、质量覆盖断言(mid1..o3 / f1..o3 span)、双表面一致性断言(offer 与 UNCOVERED 审计不得互相矛盾)。作者另做了修复前验证:临时回退调用点后 offer 坍缩到分隔符前的头部(["m00001-m00003"]),即报告症状本身。接线方式与 relay 一致(send 视图作 messages、原始历史作 compressMessages)。

代码整洁(⑤/⑤b):命名达意、单一 helper 三处复用无重复、复用既有前缀常量无魔法串、注释讲原因且带 issue 出处、缩进风格随各文件既有惯例、无新依赖、无死代码。全项过。

CI 状态:首 head 7bf8396 的 win22 s1/s2 红 = 继承性失败(master@1fad21de8 同签名 fixture-shadowing 家族,见 #4599 floor 472 已按政策处置:merge ci-fixes@cf99d935f,新 head 9e10fcfee;该 merge 仅带入 7 个 test-infra 文件,与本 PR 6 文件零交集,我已在本地核对该 merge 内容)。新 head 当前:26 success / 12 in_progress / 0 failed / 14 skipped,此前红的分片无一复发。全矩阵终判以远端 CI 为准,绿了我在本线程补一行确认。

遗留提醒(首条回复已提,再强调一次):PR 正文 Closes #5141 指向的是 @abworks-dev 那个在途的 fork-snapshot PR(GitHub #5141,与本缺陷无关)——合并本 PR 会把它自动关掉。请 owner 确认编号是否笔误(真正的 bug 报告 issue 号),合并前处理一下。

判定:代码审查通过,§7.4 自动合入门禁各项满足(单机制 fix、回归测试钉住原症状、不触 kernel/config/wire/persistence/ref-id 逻辑),待全量 CI 矩阵转绿即可合并(合并动作仍归人)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 终判:head 9e10fcfee 全矩阵绿——53 个 check run 全部完成,38 success / 15 skipped(门控 e2e 车道)/ 0 failed,此前继承红的 win22 s1/s2 在并入 ci-fixes 后未复发。镜像缓存已同步确认同 sha。

收工判定:可合并(合并动作归 owner)。合并前唯一待办仍是上一条提醒的编号问题——正文 Closes #5141 会误关 @abworks-dev 的在途 PR #5141,请确认后处理。

@kuviki

kuviki commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

issue 对不上,是 #5139 才对,而不是 #5141

…ments cite #5139, not #5141

Owner correction on #5144: the Codex separator-endpoint bug report is GitHub issue #5139 (kuviki), not #5141 (an unrelated in-flight external PR). Renames tests/issue5141-turn-separator-ranges.test.ts -> tests/issue5139-turn-separator-ranges.test.ts and re-points every in-code citation. No behavior change: comments, test names/session-id prefix only.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

已核实并修正:GitHub #5139 = @kuviki 的 Codex bug 报告「临时分隔符被选作压缩端点,acp_status 静默漏报 ~180K」(open),确为本 PR 的真正目标 issue;#5141 是无关的在途外部 PR。已做两处处理:

  1. PR 正文:首行 Closes #5141 → Closes #5139,Tests 节的测试文件名引用同步更新(合并后将正确自动关闭 [Bug][Codex] Temporary separator selected as compression endpoint, causing acp_status to silently fail to report ~180K t | [Bug][Codex] 临时分隔符被选作压缩端点,导致 acp_status 静默漏报约 18 万 token 可压缩历史 #5139,不再误关 [PR] fix(fork): restore cold parents from verified unified raw tails (#5140) #5141)。
  2. 代码引用(独立标注 commit 44cd192fa,已推送到 PR 分支):tests/issue5141-turn-separator-ranges.test.ts → tests/issue5139-turn-separator-ranges.test.ts(git mv,95% 相似重命名),src 四处注释与测试文件内全部 14 处 #5141 引用一并改指 [Bug][Codex] Temporary separator selected as compression endpoint, causing acp_status to silently fail to report ~180K t | [Bug][Codex] 临时分隔符被选作压缩端点,导致 acp_status 静默漏报约 18 万 token 可压缩历史 #5139。纯引用修正,零行为变化(仅注释、测试名、会话 id 前缀)。

说明:原始 fix commit 的 subject 里仍带 (#5141)——改写历史需对该共享分支 force-push/rebase,风险大于收益,未动;影响 auto-close 与可追溯性的正文和代码引用已全部纠正。

新 head 44cd192fa 上 CI 正在重跑(改动仅为重命名+注释,预期无新红),全矩阵绿了我再补一行确认。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 终判(引用修正后的最终 head 44cd192fa):全矩阵绿——53 个 check run 全部完成,38 success / 15 skipped / 0 failed。

收工判定不变:可合并,合并动作归 owner。合并后将正确自动关闭 #5139(kuviki 的 bug 报告),#5141 不再被误关。本 PR 审核完毕。

@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.912-master.58e61a3 (rolling master tag).

npm install -g billion-context@0.0.912-master.58e61a3  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working codex p2 scope/kernel

Projects

None yet

2 participants