Skip to content

[PR] feat: audit uncovered history behind the "no ranges remain" verdict (#4331) - #4339

Closed
sorasoras wants to merge 2 commits into
ranxianglei:masterfrom
sorasoras:2026-10-23_uncovered-history-audit
Closed

sorasoras wants to merge 2 commits into
ranxianglei:masterfrom
sorasoras:2026-10-23_uncovered-history-audit

Conversation

@sorasoras

Copy link
Copy Markdown
Contributor

Problem

compress() answers "No compressible ranges remain — the context is already at its minimum" from the fold scanner's current offer, not from a coverage check. A span can be both uncompressed and unofferable: the fold committed at turn N covers m00001..mN (the compress call itself is turn N), and the next fold starts after N, so the traffic around the compression turn is orphaned forever. It stays raw on every request while the receipt claims completeness.

Measured on two live sessions (SQL over index.db, active blocks, coverage expanded through directBlockIds):

session messages active blocks uncovered mid-history share of live context
current (0.1.204, 864 msgs) 864 13 138 refs in 23 runs, 262 KB (~65k tok) 16% of 404k
older (0.1.197, 12k refs) 11,248 53 161 refs, 489 KB (~122k tok) 20% of 596k

Largest single run: m00312–m00411 — 100 consecutive messages (50 tool-call/result pairs, 201 KB). Roughly 30 of the 138 are the compression-machinery messages themselves.

This is the mechanism behind #4331. This PR implements option 1 from the issue: make the gap visible. It does not absorb spans (option 2) — that stays open.

Change

New src/uncovered-history.ts:

  • coveredRefNumbers(state) — coverage of every active block, walking effectiveMessageIds through state.messageRefs.byRaw and recursing directBlockIds (a tier-2 parent can carry an empty effective list; its children still cover).
  • uncoveredHistory(state, messages, options) — contiguous runs of live messages that are in no active block, skip state.deadRefs, merge split cores that share one ref, exclude the same preserveRecentMessages tail the scanner protects, and estimate tokens with countMessageTokens.
  • formatUncoveredHistory(report, { compact? }) — one head line with total + largest runs, plus the caveat that the Compressible-ranges table does not include these refs.

Two call sites:

  • acp_status — always renders the audit (top 3 runs) after the live-ranges section, before DEAD REFS. This is the full inventory.

    UNCOVERED HISTORY — 138 message(s) in 23 run(s) are in no active block and still ride the wire raw (~65K tok). Largest: m00312–m00411 (100 msgs, ~50K tok) · m00416–m00429 (14 msgs, ~3.4K tok) · m00120 (1 msg, ~97 tok).
      The Compressible-ranges table above comes from the fold scanner and does NOT include these refs. Fold any span directly with compress({startId,endId}); the protected recent tail is already excluded here.
    
  • compress receipt — NO_RANGES_REMAIN_TEXT stays verbatim; a note is appended only when the gap is material (>= 10k tokens, top 2 runs). The compress 成功输出缺剩余范围快照与停止提示(pi 侧 #420/#521 对齐) #1387 / pi feat: dsh 进 PLUGIN_AGENTS——native 插件部署 + 拦截层(native 模式 D) #521 rule ("declined leftovers must not re-nag") is preserved: the sub-threshold residue fixture still emits the bare stop signal.

Tests

  • tests/uncovered-history.test.ts (new, 6 cases): DirectBlockIds recursion; gap detection with the protected tail; split-core duplicate refs; deadRefs + minTokens filtering; acp_status section; receipt note fires for a material gap and stays silent for sub-threshold residue.
  • tests/compress-success-tail.test.ts — 8/8 unchanged.
  • tests/acp-status.test.ts — 9/9 unchanged.
  • tsc --noEmit -p tsconfig.json — clean.

Scope

Option 1 only (visibility). Closing the gap still needs fold-boundary absorption or a maintenance fold; with this PR at least the verdict no longer overstates coverage, and the refs are actionable.

Refs #4331.

…anxianglei#4331)

compress()'s stop signal is derived from the fold scanner's current offer,
not from an actual coverage check. Messages that fall between two fold
boundaries (the compress call's own turn, then the next fold starting after
it) are never offered again and stay raw on every request — measured at 16%
of live context in the current session and 20% in a 12k-message session.

Add src/uncovered-history.ts:
- coveredRefNumbers(state): active-block coverage, recursing directBlockIds
  so a tier-2 parent with an empty effectiveMessageIds list still covers.
- uncoveredHistory(state, messages, options): contiguous runs outside every
  active block and older than the protected tail, token-estimated.
- formatUncoveredHistory(): one head line plus the scoped-table caveat.

acp_status always reports the audit (top 3 runs). The compress receipt keeps
NO_RANGES_REMAIN_TEXT verbatim and only appends a note when the blind spot is
material (>=10k tokens), so the ranxianglei#1387 "declined leftovers never re-nag" rule
still holds.

Tests: 6 new cases; compress-success-tail and acp-status suites stay green.
tsc --noEmit clean.

Refs ranxianglei#4331 (visibility only; fold-boundary absorption remains open).
@ranxianglei ranxianglei changed the title feat: audit uncovered history behind the "no ranges remain" verdict (#4331) [PR] feat: audit uncovered history behind the "no ranges remain" verdict (#4331) Oct 22, 2026
@sorasoras

Copy link
Copy Markdown
Contributor Author

Local verification (Windows, node 22, base 94f75cc + this branch):

  • node --import tsx --test tests/uncovered-history.test.ts → 6/6 pass
  • node --import tsx --test tests/compress-success-tail.test.ts → 8/8 pass (verbatim stop text untouched; the note appears only when uncovered >= 10k tok)
  • node --import tsx --test tests/acp-status.test.ts → 9/9 pass
  • npx tsc --noEmit -p tsconfig.json → exit 0
  • node scripts/check-unused-exports.mjs → clean (report types kept module-internal)

The red checks on run 39823517252 are not from this diff. Both are master-side and already have open fixes:

  1. Every test (*) shard aborts before running a test with
    process-spawning test files missing from LIVE_FILES in scripts/run-ci-tests.mjs: tests/storage-durability.test.ts.
    master aa198fe tracks that file, but scripts/live-files.mjs has no entry for it. Covered by [PR] ci: register tests/storage-durability.test.ts in LIVE_FILES (#4333) #4334.

  2. mysql-contract fails with
    ERR_MODULE_NOT_FOUND: .../node_modules/acp-kernel/dist/index.js imported from src/config.ts.
    kernel/dist is gitignored and that lane never builds it; locally node_modules/acp-kernel is a junction to kernel/, so it only works because a build exists. Covered by [PR] fix(ci): build acp-kernel in the mysql-contract lane (sub-defect B of #4326) #4330.

I do not have admin rights to re-run workflows, so a re-run (or merging #4334/#4330 and then re-pushing this branch) is needed to get a real signal here.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 本 PR 即 #4331 option 1(可见性)的实现本身。检索了 open/closed issue 与全部在途 PR:没有其他工作在做「活跃块覆盖 ∪ vs 线上 ref 全集」的覆盖审计。最接近的 #4045(codex forge 交接携带未覆盖尾巴)机制不同;#4334/#4330 是 CI 车道修复,与本改动无关。

标签: pr scope/wiring p2(改动骑在请求管道的 compress receipt / acp_status 诊断面上,不动内核/折叠判定/持久化/UI 面板;p2:这是可见性缓解,gap 本体要等 option 2/3 才关闭)。

结构化八问

  1. 是什么问题:compress 的「No compressible ranges remain」判决来自折叠扫描器的当前 offer,不是覆盖审计——两个折叠边界之间的消息既没被压、也永远不被 offer,却仍逐请求裸跑在线上,receipt 却声称「已到最小」。
  2. 什么原因导致的:覆盖只按块跟踪,没有任何一处拿「活跃块覆盖并集」对照会话的存活 ref 全集;判决措辞默认了「扫描器 offer 耗尽 == 覆盖完整」,这个等式在跨边界流量上不成立。
  3. 影响范围与严重程度:所有发生过多次折叠的 proxy 模式会话。实测两个活会话:16% / 20% 的 live context(~65K / ~122K tok)裸跑且不可 offer。程度:每请求持续浪费 token + 完整性误报;可用性不受损。本 PR 只做可见性,不关 gap。
  4. 怎么修复的(option 1):独立覆盖审计 src/uncovered-history.ts——活跃块覆盖并集(tier 父块经 directBlockIds 递归,空 effective 列表也能从子块取回覆盖)、扣除 deadRefs、排除与扫描器同源的 preserveRecentMessages 保护尾、按连续 ref 段聚合并估 token。两个出口:acp_status 恒显完整清单(top 3 段);compress 停止信号 receipt 仅在缺口 ≥10K tok 时附一行注记(top 2 段)。NO_RANGES_REMAIN_TEXT 保持逐字节不变(注记追加在其后,契约字符串本身未动);低于阈值的残余保持静默——compress 成功输出缺剩余范围快照与停止提示(pi 侧 #420/#521 对齐) #1387/pi feat: dsh 进 PLUGIN_AGENTS——native 插件部署 + 拦截层(native 模式 D) #521「declined leftovers 不得再唠叨」规则保留,且有负向测试钉住。
  5. 是否有回归问题:对触碰路径的既有行为逐项清点——
    • 停止信号 receipt 字节:无实质缺口时与旧版逐字节一致。既有断言全部是 includes 型;唯一严格字节钉(tests/compress-fingerprint-refold.test.ts:102 的 deepEqual)其 fixture 全量折叠、零未覆盖 ref,不受影响。本地在 tip 上定向跑了四个受影响文件共 38/38 绿,tsc --noEmit 干净。
    • acp_status 输出:仅当存在未覆盖 ref 时在 DEAD REFS 段之前插入新段;tests/acp-status.test.ts 9/9 不变。
    • 配置解析链路已端到端核实:两个调用点的 ctx.config 都是 resolveRequestConfig(handle.ts:746 → forward → RewriteCtx)产出的按请求解析值,provider/model 级 preserveRecentMessages 覆写会正确传导到审计的保护尾,不会把实际受保护的尾部误报成可回收。
    • 无 wire/协议变更、无持久化变更、无 kernel 变更(不需要 kernel-change 声明)、不触碰缓存语义(注记在 tool result 内,该位置本就随会话变化)。
    • 行为变更披露(有意):(a) compress 停止信号在缺口 ≥10K tok 时多一行注记;(b) acp_status 多一个 UNCOVERED HISTORY 段。两者均为模型可见文本增量,PR 描述已披露。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及——纯 host 侧诊断面,内核未动;option 2/3(吸收/维护性折叠)留在 [Bug] compress reports 'no compressible ranges remain' while mid-history spans sit outside every block #4331。
  8. 是否引入了新的配置:无。minTokens=10_000 / maxRuns 为代码常量,刻意不进配置面。

代码审查要点(独立复核)

CI 状态:红,但 100% 继承自 master,与本 diff 无关

head 776b636295 的失败全部来自 master 侧,已逐一取证:

结论:#4334 与 #4330 合入 master 后,本 PR 重跑 CI 即应转绿,无需为本失败做任何改动(分支落后 master 13 个提交,rebase 是好习惯但非本失败的前置条件)。

判定

代码与行为通过审查:方向对(先让盲区可见,再谈吸收)、契约保真(停止信号逐字节不变 + 静默规则保留)、回归清点干净、测试有牙。待 #4334/#4330 落地且 CI 重跑绿后即可合并;我无法代合(系统规则:合并仅限人工),请 owner 操作:#4339

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复——本单即 GitHub PR #4339 的跟踪记录(同号);评论提到的两处 CI 红对应 master 既有问题 #4333/#4336,修复 PR 已在途(#4334/#4330),无需新立项。

标签: bug, pr, p2, scope/kernel(压缩判定面「no ranges remain」结论的可见性缺陷,客户端无关;原 scope/wiring 改判为 scope/kernel——改动落在 src 侧压缩决策面而非协议接线)

验证结论:评论的说法成立,已独立复现。 两处红确属 master 既有缺陷,与本 PR 的 diff 无关。另有一层机制评论没点透:fork PR 的 CI 检出的是 merge preview(base+head)——本分支自身树里其实没有 tests/storage-durability.test.ts(它的 base 早于 #4177),但 merge preview 里有,所以 tripwire 照样触发。推论:当前对 master 的任何 PR 都会撞上这两处红。

逐项核实:

  1. 全部 test shard 中止:master aa198fe([PR] storage: durability trio — open-time quick_check + online VACUUM INTO backup + second-process lockfile (#4175) #4177)新增 tests/storage-durability.test.ts,scripts/live-files.mjs 无登记;scripts/run-ci-tests.mjs:125-131 对未登记的 process-spawning 测试硬退出,且每个 shard 都扫全量文件集 → 全体同死。✅ 逐行核实。
  2. mysql-contract ERR_MODULE_NOT_FOUND:该 lane(ci.yml)只做 npm install --ignore-scripts + 直接 tsx 跑测试,无 kernel 构建步骤;src/config.ts 导入 acp-kernel,kernel/dist 被 gitignore → fresh checkout 上确定性失败。✅ 核实。
  3. 本地复现(PR head + master 合成 merge-preview 树,合并干净):uncovered-history 6/6、compress-success-tail 10/10、acp-status 7/7、tsc --noEmit 干净、check-unused-exports 干净。
  4. run 39823517252 的 head 确认为 776b636(即当前 head):当前 head 已被 CI 测过,只有 master 侧两 lane 红;第一次 forward 里的 unused-exports 红来自旧 commit faca7b9,已被第二个 commit(报告类型模块内化)修复。

代码审查(实际 diff 4 文件 +341/-1,已通读):只读审计面,不碰 kernel/、无配置面变更、无 wire 格式变化;NO_RANGES_REMAIN_TEXT 保持逐字不变,仅当 uncovered ≥ 10k tok 时在其后追加一行说明(测试双向断言齐备:大缺口点名 / 小残留静默,符合 #3617 纪律);审计调用包在 try/catch 中,失败降级回原回执。质量达标。

八问:

  1. 是什么问题:本 PR 的 CI 红不是 PR 代码造成的——是 master 两个既有缺陷(LIVE_FILES 漏登记 + mysql-contract lane 缺 kernel 构建),正挡住所有 PR 的绿色信号。
  2. 什么原因导致的:[PR] storage: durability trio — open-time quick_check + online VACUUM INTO backup + second-process lockfile (#4175) #4177 新增 process-spawning 测试文件后未在 scripts/live-files.mjs 登记,触发 run-ci-tests 的硬 tripwire;mysql-contract lane 的导入图到达 acp-kernel,但 lane 从不构建 kernel/dist(gitignored),fresh checkout 必炸。两者都与本分支的 4 文件 diff 无交集。
  3. 影响范围与严重程度:当前对 master 的所有在途 PR(含本 PR)全红 → 仓库级合并信号被阻塞;终端用户零影响(纯 CI 面)。CI 层面 P1,用户面无。
  4. 怎么修复的:修复已存在——[PR] ci: register tests/storage-durability.test.ts in LIVE_FILES (#4333) #4334 一个 PR 覆盖两处缺陷(LIVE_FILES 登记 + mysql-contract 补 kernel 构建步骤),其 CI 全绿;[PR] fix(ci): build acp-kernel in the mysql-contract lane (sub-defect B of #4326) #4330 只含后半部分(被 [PR] ci: register tests/storage-durability.test.ts in LIVE_FILES (#4333) #4334 覆盖,建议 owner 二选一)。本 PR 自身无需任何代码改动。
  5. 是否有回归问题:审过既有行为——compress 停止文案保持逐字(仅 uncovered≥10k tok 时追加说明,issue 正文已披露的有意变更);acp_status 新增 UNCOVERED HISTORY 段(additive 展示,不删行);错误处理 fail-fast 未变、阈值/时序/输出格式无漂移。未发现回归。
  6. 是否破坏缓存:不涉及(只读审计,不触碰 fold state/cache/去重键)。
  7. 是否改了设计方向:没有——纯可见性补充,即 [Bug] compress reports 'no compressible ranges remain' while mid-history spans sit outside every block #4331 规划的 option 1,机制零改动。
  8. 是否引入新配置:没有。

下一步(需 owner 动作):合并权限在人这边——请合并 #4334(一处覆盖两处红);之后我可以直接重跑本 PR 的失败 job(我有 admin 权限),或等任意 push 触发新 CI。现在重跑没有意义(master 没变,会原样再红)。

判定:本 PR 代码本身 ready,阻塞纯粹来自 master 侧两个待 #4334 合入的 CI 缺陷。

@sorasoras

Copy link
Copy Markdown
Contributor Author

Superseded by #4340, which implements option 1 with a kernel-backed coverage audit (empirical zone diff, kernel coveredMessageIds, verdict rewrite) - wider and better integrated than this host-side version. The independent measurements from two live sessions are kept in #4331.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants