Skip to content

[PR] fix(#3635): steer external summarizers to the enforced summary cap, name rejections, make the caps settable - #4445

Closed
ranxianglei wants to merge 3 commits into
masterfrom
2026-10-24_3635-external-summary-length-cap
Closed

ranxianglei wants to merge 3 commits into
masterfrom
2026-10-24_3635-external-summary-length-cap

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

What this fixes

External summaries longer than compress.maxSummaryLength were rejected whole after paying full latency — no truncation, no retry, and the only trace left was a bare invalid_summary outcome nobody could act on. On top of that, the cap itself was accepted nowhere: the panel PUT silently dropped it (COMPRESS_SETTING_FIELDS had no entry), file edits never reached the resolved kernel config (mergeCompress/applyCompressSettings never mapped it), and the model-tool compress path never told its summarizer the number its own validator enforces — while the preflight path has done exactly that since #1775. So overshoots on the compress path were routine and unexplained; the floor-1 report measured 2/11 folds discarded this way on v0.1.204.

Root cause, not symptom: the producer of the bad bytes is a prompt/validator asymmetry — the summarizer is asked for output it can never accept, and the operator has no dial to widen what IS acceptable. Both halves are fixed here.

Duplicate check: none open. Twin issue #3634 was already closed; closest prior art is #1775/#1779 (length-budget clause + rescue-truncate on the preflight path only) — this PR brings the same discipline to the compress path and makes the caps themselves settable.

Changes

  1. State the exact enforced cap up front (src/external-summary-compress.ts, new shared helper summaryLengthBudgetClause in src/external-summary.ts). The compress-path work items now carry the same LENGTH BUDGET clause preflight has had since preflight 摘要超长重试循环:summarizeRange 提示词无长度预算,超限摘要整体丢弃而非截断救援 #1775, rendered from the same number the validator enforces (min(range.summaryMaxChars ?? Infinity, config.compress.maxSummaryLength)). The two inline preflight copies were replaced by the shared helper — byte-identical prompts (existing pinned suites pass unchanged).
  2. Name the rejection reason (src/external-summary.ts). invalid_summary attempts now carry a detail mirroring the validator's check order (summary N chars exceeds cap M / below floor M / response N bytes exceeds byte cap M / empty or non-string response), rendered by describeExternalSummaryFailure — moved next to its data type, re-exported from src/preflight.ts for historical importers. The compress-path log line uses it (parity with preflight; was bare batch=<status>), and the all-fail refusal text now names length as the cause with the two real levers.
  3. Make the caps settable (src/config.ts, src/compress-settings.ts). compress.maxSummaryLength / compress.minSummaryLength flow through parse → merge → apply into the resolved kernel config (panel PUT and file edits alike). 0 stays meaningful (unbounded cap / disabled floor); negative/non-finite rejected at parse. Docs: CONFIGURATION.md + zh-CN + website seeds (config-surface-guard green).

Deliberately NOT changed: over-cap replies are still rejected whole on the compress path (no truncation added) — whether to add a bounded rescue-truncate there (mirroring preflight's #1775 pattern) is an open product question, tracked in the issue thread.

Behavior changes (old → new)

# Surface Old New
① Compress-path external-summarizer prompt No length instruction at all Same clause as preflight, stating the exact enforced cap (model compliance varies; validator unchanged)
② All-candidates-fail refusal text "…do not retry unchanged configuration." Same sentence + names length as the cause and both levers (compress.maxSummaryLength, per-range summaryMaxChars)
③ [external-summary] completed … log (compress path) batch=<status> Full failure description incl. per-attempt detail; preflight rendering without detail is byte-identical (pinned by tests/issue2484-overflow-and-summary.test.ts)
④ Config surface Cap/floor accepted nowhere (silently inert) Settable via file + panel PUT; defaults unchanged (kernel 20000 / 50)
⑤ [external-summary] completed (preflight) … log + unusable diagnostics attempts=[t0:invalid_summary] Same, plus the parenthesized rejection reason on invalid_summary attempts (the shared executor records detail on every path); log/diagnostic-only — no wire or model-visible change

Preflight prompts: byte-identical (pure refactor into the shared helper).

Config surface

  • Added: compress.maxSummaryLength (number, default kernel 20000, 0 = unbounded) and compress.minSummaryLength (number, default kernel 50, 0 = disabled) under the existing three-level compress hierarchy.
  • Why the existing surface couldn't express them: both map onto EXISTING kernel knobs (defaultConfig().compress.{max,min}SummaryLength — no new semantics invented), but were missing from COMPRESS_SETTING_FIELDS, parseCompressSettings, mergeCompress and applyCompressSettings, so every acceptance path dropped them. Considered alternatives: nothing — the per-range tool arg summaryMaxChars remains the finer-grained lever and already worked.
  • Migration: none needed — defaults unchanged; previously-set values simply start taking effect (that is the point of the fix).

Both compression modes (§6)

Touches only the side-call prompt sent to configured external summarizers plus proxy-side config plumbing. No client wire shape change in plugin mode or proxy mode; acp_summary carrier handling untouched; client-visible history bytes unchanged, so prefix cache is unaffected.

Tests

New tests/issue3635-summary-length-cap.test.ts (7): executor rejection reasons (over-cap / below-floor / byte-cap / empty), describe-format preservation + detail rendering, clause gating, parse/merge/apply plumbing, and a wire integration asserting the clause reaches the HTTP body at both the default cap and a configured cap. Targeted neighboring suites local-green (157 tests across external-summary*, issue2484, compress-settings, web-routing, removed-compress-keys-inert, preflight-summary-*); full suite = remote CI. config-surface-guard and unused-export gates run green locally.

Closes #3635

…ame rejections, make the caps settable

External summaries longer than compress.maxSummaryLength were rejected whole after paying full latency, with the only trace a bare invalid_summary outcome - and the cap was accepted nowhere: the panel PUT dropped it, file edits never reached the resolved config, and the model-tool compress path never told its summarizer the number its validator enforces (the preflight path has since #1775).

- state the exact enforced cap in the compress-path summarizer prompt (shared clause helper, byte-identical to preflight's)
- name the rejection reason on each invalid_summary attempt and log it (describeExternalSummaryFailure moved next to its data type; re-exported from preflight for historical importers)
- accept maxSummaryLength/minSummaryLength through parse/merge/apply so panel PUT and file edits reach the resolved kernel config
- document both fields (CONFIGURATION en+zh, website seeds)

Closes #3635
@ranxianglei ranxianglei changed the title fix(#3635): steer external summarizers to the enforced summary cap, name rejections, make the caps settable [PR] fix(#3635): steer external summarizers to the enforced summary cap, name rejections, make the caps settable Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-24_3635-external-summary-length-cap (d475598)

Latest build — install this exact version

npm install -g billion-context@0.0.5345-pr.4445.c3ca967

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
d475598 billion-context@0.0.5345-pr.4445.c3ca967 build
aa84e55 billion-context@0.0.5337-pr.4445.34ae98f build
006b683 billion-context@0.0.5333-pr.4445.5e30fe3 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4445.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 本条是 PR #4445 的镜像,无独立查重对象;原 issue #3635 侧查重已做(不重复,孪生 #3634 已关闭,最近先例 #1775/#1779 只修 preflight 路径)。

标签: bug scope/kernel p2(缺陷类;摘要长度上限属 src 侧压缩决策面,未触碰 kernel/ 目录;p2=有绕行的功能性缺陷,与 #3635 定级一致)。

本 PR 是 agent 已完成的工作(单 commit,awork),我不重做,按 [PR] 流程核验:拉分支读全量 diff、审方向/回归/代码/测试、本地定向验证。目前进展:

  • 方向关:与 External summaries over compress.maxSummaryLength are silently discarded after paying full latency — and the cap is not configurable #3635 及 1L 的两层根因(提示词/校验器不对称 + 配置管道缺失)逐条对上,修法为疏(告知确切上限 + 接通配置)不堵,截断抢救按产品决策留待拍板——方向成立。
  • 本地定向验证:issue3635 新测试 + external-summary*/issue2484/compress-settings/preflight-summary-*/removed-compress-keys-inert/web-routing 共 243 个测试全绿;typecheck 干净;config-surface-guard 与 unused-exports 门禁绿。preflight 提示词字节级不变的声明经真实 wire body pinned 测试确认。
  • 发现一处小缺陷:summaryLengthBudgetClause 注释声称 cap <= 0(不限)时省略子句,但实现只 gate 在 budget < minChars——当 maxSummaryLength: 0(不限)且 minSummaryLength: 0(禁用下限)同时成立时,会向摘要模型渲染出 "AT MOST 0 characters"。触发条件苛刻(两个配置都设 0,或模型对某 range 传 summaryMaxChars: 0 且下限禁用),后果是提示词自相矛盾而非数据损坏。按「小问题直接修」处理:一行修正 + 补一条测试钉住,作为独立标注 commit 推到 PR 分支,随后继续 review。
  • CI:pushed head 的 check-run 目前 17 queued / 5 success / 6 skipped,无红;我推完修正 commit 后 CI 会在新 head 重跑,最终判定以远程 CI 为准。

修正推完后我会给结构化八问结论。

@ranxianglei ranxianglei added bug Something isn't working p2 scope/kernel labels Oct 23, 2026
awork added 2 commits October 24, 2026 01:54
…e floor disabled

Review follow-up on this branch: summaryLengthBudgetClause documented
"<= 0 / Infinity" as omitted but only gated on budget < minChars, so cap 0 +
floor 0 rendered "AT MOST 0 characters" to the summarizer. Zero means
unbounded; omit it. Pinned by a new assertion in
tests/issue3635-summary-length-cap.test.ts.
…c passes

Review follow-up: the summary-length rows were hand-written into CONFIGURATION.md / CONFIGURATION.zh-CN.md with wording that drifted from website/config-reference/compress.yaml, and website/docs/index.html + website/main.js were never regenerated, so the docs-sync artifact-drift gate went red. Regenerated all four derived files from the seed (node tools/gen-config-docs.mjs generate); the seed wording is the canonical one.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

核验完成。这个 PR 是 agent 已完成的工作(无 ework-agent-pr 标记,按 [PR] 流程做了独立审核而非重做)。结论:方向对、实现干净、测试有牙,审核中发现并直接修掉了两个小问题(已作为独立标注 commit 推到 PR 分支),CI 最终头全绿。建议合并(合并权在你,agent 不合并)。

结构化八问

1. 是什么问题:外部摘要链有两个缺陷——①模型工具压缩路径的摘要器从未被告知被强制执行的字符上限(preflight 路径自 #1775 起就告知),超长回复烧完全程延迟后被整段拒绝,且日志只有裸 invalid_summary;②compress.maxSummaryLength/minSummaryLength 在配置面上无处可设(面板 PUT 静默丢弃、文件编辑永远到不了内核配置),用户连绕行都做不到。Floor-1 报告实测 v0.1.204 上 11 次折叠丢 2 次。

2. 什么原因导致的:两个独立根因——提示词/校验器不对称(校验器强制执行上限,提示词却不声明)+ 配置管线缺失(parse/merge/apply 三处都没映射这两个字段)。修复针对根因(疏),不是拦截症状(堵)。

3. 影响范围与严重程度:p2。能用但有明显问题:启用外部摘要链且回复超上限时整段丢弃(浪费延迟与成本),且上限/下限不可调。不涉及客户端 wire 形态与会话数据,不影响默认配置下的既有行为。

4. 怎么修复的:三层——①压缩路径复用 preflight 的共享 helper summaryLengthBudgetClause,把确切强制上限(min(per-range summaryMaxChars, config maxSummaryLength))写进摘要器指令,preflight 两处内联副本换成同一 helper(字节不变);②invalid_summary 尝试携带 detail(镜像校验器检查顺序),日志与全部失败的拒绝文案点名具体原因和两个调节杠杆;③两字段接入 parse→merge→apply,文件+面板可设,0=不限/禁用,负数/非有限值在 parse 期拒绝。超长回复仍整段拒绝、不做救援截断——这是刻意留给 owner 拍板的产品决策(#3635 floor 5 待决,推荐选项 A)。

5. 是否有回归问题:逐项审过既有行为——preflight 提示词字节不变(纯重构,真实 wire body 测试钉住,含禁上限的负例);默认值不变(kernel 20000/50);wire 形态、acp_summary 载体、kernel/、持久化格式均不动。审核中发现并已修复两处:

  • corner-case 缺陷(commit aa84e5531):cap=0 且 floor=0 时 doc 注释声称省略该子句,代码却渲染出 "AT MOST 0 characters"——补了 budget <= 0 门控 + 测试钉;
  • PR 正文披露不全:共享执行器给所有路径(含 preflight)的 invalid_summary 尝试都记 detail,即 preflight 日志也会多一个括号原因——已在 PR 正文行为表补第 ⑤ 行(仅日志/诊断面)。
    另修了 CI 红:docs-sync 失败(hand-written 的 CONFIGURATION.md/zh-CN 两行措辞与 seed yaml 漂移,website/docs/index.html + website/main.js 漏重生成)→ 从 seed 重生成全部四个派生文件(commit d47559803)。

6. 是否破坏缓存:不涉及——客户端可见的出站前缀字节不变;长度预算子句只进外部摘要器的 side-call 提示词,unusable 字符串只流向日志/诊断(grep 过消费点,无一模型可见)。

7. 是否改了设计方向:没有。「整段拒绝、不截断」的方向保持;救援截断仍作为开放产品问题在 #3635 跟踪。

8. 是否引入了新的配置:有——compress.maxSummaryLength(number,默认 kernel 20000,0=不限)与 compress.minSummaryLength(number,默认 kernel 50,0=禁用),映射到既有内核 knob,无新语义、无迁移。文档 CONFIGURATION.md / zh-CN / website 三端同步(config-surface-guard 绿:167 leaf fields)。

验证证据

  • 本地定向验证:243 个相关测试全绿(issue3635 新套件 7 个 + compress-settings / external-summary* / issue2484 / preflight-summary-* / removed-compress-keys-inert / web-routing 等 14 个文件);typecheck 干净;config-surface-guard、unused-exports 本地绿。
  • 远程 CI(最终头 d47559803):全绿——docs-sync ✅、11 个 test shard ✅、static-gates ×2 ✅、plan/version/kernel/pi-subagents/lane/lifecycle/lock-sync/docs-coverage/config-surface/test-import-graph/unused-exports/mysql-contract/build-artifact 全 ✅;各 e2e workflow(dsh/pi/opencode/codex-fake/persona/ws/local-registry/image/windows-overlay)单独跑也全 ✅。

收工判定

可合。PR 分支现有 3 个 commit(原提交 + 两个标注清晰的 review follow-up),diff 收敛到 13 个文件且每行都与 #3635 目的相关。等你合并;「超长摘要是否救援截断」留 #3635 由你拍板。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

已在 #4558 落地(预算子句共享化 + 零上限语义,与 #4487 targetKey 并集)。本 PR 关单收尾;回滚点为 #4558 单提交 revert。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

External summaries over compress.maxSummaryLength are silently discarded after paying full latency — and the cap is not configurable

1 participant