Skip to content

[PR] fix: classify proxy tool results by structured outcome, not just the FAILED keyword - #7124

Open
ranxianglei wants to merge 3 commits into
masterfrom
2026-11-18_outcome-authoritative-result-classification
Open

ranxianglei wants to merge 3 commits into
masterfrom
2026-11-18_outcome-authoritative-result-classification

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

The first-line "FAILED" classifier (from #5384) misclassifies proxy tool results in both directions:

The dispatcher already computes a structured outcome (applied/partial/refused for compress, success/failure for every other tool) but dropped it when pushing to proxyResults, so the five control-flow predicates only ever saw the text.

Fix

Carry outcome through proxyResults and make it authoritative at the five control-flow sites (src/loop/core.ts):

  • a compress is failed iff refused (applied/partial both folded something);
  • every other tool (decompress/absorb/…) is failed iff failure;
  • the first-line FAILED check remains only as the legacy fallback when no outcome is present.

⚠️ Behavior change (flagged)

Tool failures whose first line lacks FAILED (e.g. decompress not-found) now classify as failed instead of success. Consequences:

  • a failed-only round now ends the turn when visibilityMarkers is off (instead of re-requesting until the loop limit), and splices a recovery receipt + re-requests when markers are on — exactly how a failed compress is already handled;
  • compress applied/partial and decompress/absorb success/failure map identically to the first-line behavior, so no other path changes.

Regression test

tests/plugin-protocol.test.ts: a decompress not-found call (no FAILED on the first line) with visibilityMarkers:false. Pre-fix it re-requests to the loop limit (10 upstream calls); post-fix the turn ends after 1. The existing #5384 test (summary containing FAILED continues) still passes — no regression.

Scope

Single module (src/loop/core.ts + test). No config schema, persistence, wire-shape, or kernel change. Supersedes the first-line approach originally shipped for #5383 (#5384) by using the structured outcome the dispatcher already computes.

Fixes #5383

…FAILED keyword

The first-line "FAILED" classifier misclassifies both directions: a successful
compression summary that quotes an earlier FAILED result is read as a failure
(false positive), and a genuine tool failure whose first line lacks the keyword
— e.g. decompress not-found returning "[Block X not found]" with
outcome:"failure" — is read as success (false negative). The latter makes a
failed-only round re-request until the loop limit (10 upstream calls).

The dispatcher already computes a structured outcome (applied/partial/refused
for compress, success/failure for every other tool) but dropped it when pushing
to proxyResults. Carry the outcome through and make it authoritative at the five
control-flow sites: a compress is failed iff refused, every other tool iff
failure. The first-line FAILED check remains only as the legacy fallback when no
outcome is present.

Behavior change: tool failures without "FAILED" on the first line (e.g.
decompress not-found) now classify as failed. A failed-only round ends the turn
when visibility markers are off (instead of re-requesting) and splices a recovery
receipt when they are on — consistent with how a failed compress is already
handled. compress applied/partial + decompress/absorb success/failure map
identically to the first-line behavior; no other change.

Fixes #5383
@ranxianglei ranxianglei changed the title fix: classify proxy tool results by structured outcome, not just the FAILED keyword [PR] fix: classify proxy tool results by structured outcome, not just the FAILED keyword Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-18_outcome-authoritative-result-classification (6f125b6)

Latest build — install this exact version

npm install -g billion-context@0.0.10832-pr.7124.95ad1af

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
6f125b6 billion-context@0.0.10832-pr.7124.95ad1af build
9477a0d billion-context@0.0.10752-pr.7124.db81bb5 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr7124.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Verdict: the fix is correct and well-scoped — I approve the code. CI is red on 3 shards, but I traced it and the failures are not in this PR's change.

Code review (verified against source + local runs on the PR head):

  • The change is exactly what the issue describes. proxyToolResultFailed (src/loop/core.ts:198) now reads the structured outcome when present — compress failed iff refused; every other tool failed iff failure — and keeps the first-line FAILED check only as the legacy fallback when no outcome is present. All 5 control-flow call sites (core.ts:1660/1680/1769/1890/1897) were updated to pass the whole result object; the proxyResults type carries outcome? and the push site populates it.
  • I traced the producer side: the dispatcher DOES populate outcome on every leaf path (compress→compressResult applied/partial/refused; decompress/absorb/…→toolOk/toolFail), so the structured outcome is authoritative in practice, not just in the type.
  • The behavior change is one-directional (more correct, no regression). The only flips are genuine failures whose first line lacks uppercase FAILED: decompress not-found [Block X not found] (src/decompress-shared.ts:601) and absorb's lowercase absorb failed (src/absorb.ts:117) — both now correctly read as failed. I found no reverse flip (no toolOk receipt carries a FAILED first line), and compress maps identically to the old behavior (refused receipts always carry FAILED; applied/partial never do). The cosmetic buildVisibilityMarker (core.ts:87) already checks not found, so it now agrees with the control flow instead of disagreeing.
  • The regression test is real. I ran it against the pre-fix base (parent + test only): it FAILS (re-requests to the loop limit). On the PR head it passes (turn ends after 1 upstream call). Full tests/plugin-protocol.test.ts: 24/24 pass. npm run typecheck: clean.

CI (the one open item): 3 test shards red — ubuntu-22 s2, ubuntu-24 s2, windows-22 s5. Two findings:

  1. The PR's own test file (plugin-protocol.test.ts) is only in windows s5. The ubuntu s2 failures (the same file set on both node 22 and node 24) are in other files — so they are not the PR's new test, and a failure on both node versions points to a real test issue in those files, not a node flake.
  2. I ran the behavior-relevant candidates in those shards locally on the PR head — issue847-compress-failure-guards, issue2204-decompress-to-file-outcome, issue2752-forward-partial-outcome, issue3013-refusal-terminal, issue5686-compaction-direct-refusal — 46/46 pass. This diff does not break them.

The PR is based on 4b77eab409, which is behind current master (548e396022) — a stale base. Combined with the above, the red is most consistent with an inherited/born-red test on the old base (or a flake), not this diff. Recommend: rebase onto current master and re-trigger (per #4599); if a shard is still red after a clean rebase, that's the one to chase. I did not merge anything.

8问:

  1. 是什么问题:proxy 工具结果的成败分类只靠首行 FAILED 关键字,双向误判——成功压缩回执里若引用了带 FAILED 的历史失败会被读成失败(误报);首行不带 FAILED 的真实失败(如 decompress not-found 的 [Block X not found])会被读成成功(漏报),导致 failed-only 轮空转到循环上限(10 次上游调用)。
  2. 什么原因导致的:dispatcher 其实已经算出了结构化 outcome(compress 的 applied/partial/refused,其余工具的 success/failure),但推入 proxyResults 时把它丢了,五个控制流谓词只看到文本,只能靠首行关键字猜。
  3. 影响范围与严重程度:仅影响插件/代理压缩循环里 compress/decompress/absorb 的失败判定;漏报方向会让真实失败被当成功、failed-only 轮空转到 10 次上游调用(浪费 + 延迟)。属"特定条件下有问题"(工具失败且首行无 FAILED 关键字),p2。
  4. 怎么修复的:让 outcome 随 proxyResults 透传,并在五个控制流点以它为权威(compress 仅 refused 算失败,其余工具仅 failure 算失败),首行 FAILED 检查降级为"无 outcome 时"的 legacy 兜底。
  5. 是否有回归问题:行为变更是单向的(更多正确判失败)。逐一核对无反向翻转(无 toolOk 回执带 FAILED 首行),compress 映射与旧行为完全一致,buildVisibilityMarker 本就检查 not found、改后与控制流一致。本地跑了 5 个行为相关候选测试 46/46 通过,未发现回归。
  6. 是否破坏缓存:不涉及;只改失败分类,不动消息字节、缓存键、重试与增量游标。
  7. 是否改了设计方向:不涉及架构转向;是把 dispatcher 已算好的结构化结果接回控制流,替代首行关键字启发式,方向更对。
  8. 是否引入了新的配置:不涉及;无新 env/config/默认值。

收工判定:代码正确、回归测试真实(修复前红、修复后绿)、typecheck 干净、行为变更单向安全——代码层面 approve。CI 三片红经本地排查与本 diff 无关(疑似 stale-base 继承红/flake),建议 rebase 到当前 master 重触发后再看;我未合并。

ci-check added 2 commits November 18, 2026 23:03
… shrinks (#7124)

The test compresses m00004-m00005 (~6 tokens) with a 24-char summary (~6 tokens); the no-net-shrink gate (src/stream.ts) refuses a compress whose summary is not strictly smaller than the range (net delta <= 0), so the compress was refused (outcome:refused). Pre-#7124 the first-line FAILED classifier missed it (the refused receipt's first line says 'refused', not 'FAILED'), so the test passed; #7124 makes the structured outcome authoritative, so the refused compress is now correctly failed and refreshFolded no longer runs. Shorten the summary to 'ok' (1 token) so the compress succeeds and the test exercises the post-compress re-request view it was written for.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

All 3 red test shards fixed on the PR branch; final head pushed (6f125b6, no [ework:ci-held] marker → full CI will run).

What was red:

  • ubuntu 22 s2 + ubuntu 24 s2 (same failure, both node versions): #564 path 2: compress re-request keeps Responses assistant run ordering valid after summary dedup → AssertionError: refreshFolded ran (otherwise the post-compress view is untested), 0 !== 1 at tests/issue564-responses-ordering.test.ts:200.
  • windows 22 s5: tests/server-process-listeners.test.ts killed its own child on win32 (PR base predates the portability fix).

Root cause — the #564 failure is a real regression from this PR's diff (not inherited red / flake):
The test compresses m00004–m00005 (bash_a call+result, ~6 tokens) with a 24-char summary ("Inspected files A and B.", also ~6 tokens). The no-net-shrink gate (src/stream.ts:1454) refuses any compress whose summary is not strictly smaller than the range (net delta ≤ 0), so the compress was refused with outcome: "refused". Before #7124 the first-line "FAILED" classifier missed this (the refused receipt's first line says "refused", not "FAILED"), so the compress read as NOT failed → refreshFolded ran → test passed. #7124 makes the structured outcome authoritative, so the refused compress is now correctly failed → refreshFolded does not run → test fails.

Fixes (2 commits + 1 merge):

  1. 0757ec1bd6 — tests/issue564-responses-ordering.test.ts: summary "Inspected files A and B." → "ok" (1 token < 6 tokens), so the compress actually succeeds and the test exercises the post-compress re-request view it was written for. Test-only; no change to the PR's code.
  2. Merge origin/master (548e396) — brings in the Windows shutdown-signal test fix (3186e54) + [PR] fix(lanes): demote the wire clients' auxiliary calls to the side lane (discussions/6884) #7083 sidecar changes. PR diff and master diff touch disjoint files; merge was clean.

Verified on merged head 6f125b6:

  • node --import tsx --test tests/issue564-responses-ordering.test.ts → 22 pass / 0 fail.
  • npm run typecheck → clean (exit 0).
  • server-process-listeners.test.ts now carries the sendSignal helper + win32 guard.

八问:

  1. 是什么问题: PR [PR] fix: classify proxy tool results by structured outcome, not just the FAILED keyword #7124 三个测试分片红——ubuntu 22/24 的 Compression breaks Responses assistant history ordering, causing repeated invalid_assistant_history 400s #564 path 2 断言失败(refreshFolded 没跑),windows 22 的 server-process-listeners 在 win32 上杀了自己的子进程。
  2. 什么原因导致的: Compression breaks Responses assistant history ordering, causing repeated invalid_assistant_history 400s #564 是 PR 自身 diff 引入的真实回归(非继承红):测试压缩 m00004–m00005(~6 token)配 24 字符摘要(也 ~6 token),no-net-shrink 门(src/stream.ts:1454)因净收缩 ≤ 0 拒绝压缩(outcome:refused);[PR] fix: classify proxy tool results by structured outcome, not just the FAILED keyword #7124 之前首行 "FAILED" 分类器漏判(拒绝回执首行是 "refused" 非 "FAILED")故测试侥幸过,[PR] fix: classify proxy tool results by structured outcome, not just the FAILED keyword #7124 让结构化 outcome 权威化后拒绝的压缩被正确判失败 → refreshFolded 不跑 → 挂。windows 红 = PR base 落后 master 两提交(缺 3186e54 的 Windows 移植修复)。
  3. 影响范围与严重程度: 仅测试层——Compression breaks Responses assistant history ordering, causing repeated invalid_assistant_history 400s #564 的压缩被 no-net-shrink 门拒(摘要不比范围小),非生产代码行为变更;PR 核心改动(outcome 权威分类)本身正确。windows 红是测试可移植性问题,不影响 Linux 用户。
  4. 怎么修复的: ① Compression breaks Responses assistant history ordering, causing repeated invalid_assistant_history 400s #564 测试摘要从 24 字符缩到 "ok"(1 token < 6 token),让压缩真正成功、测试回到本要验证的"压缩后重请求视图"路径——纯测试改动;② merge origin/master 带入 Windows 测试修复(3186e54)+ [PR] fix(lanes): demote the wire clients' auxiliary calls to the side lane (discussions/6884) #7083,PR diff 与 master diff 文件不相交,合并干净。
  5. 是否有回归问题: 无。Compression breaks Responses assistant history ordering, causing repeated invalid_assistant_history 400s #564 改动只动摘要字符串(断言不查摘要文本),测试从"侥幸过(压缩被拒但分类器漏判)"变"真过(压缩成功)";PR 核心代码(src/loop/core.ts outcome 分类)未动。merge 带入的 master 变更是已合入的既有修复,非本 PR 引入。
  6. 是否破坏缓存: 不涉及。no-net-shrink 门本就为保护前缀缓存(净收缩 ≤ 0 不提交),本修复让测试走成功路径,不影响缓存策略。
  7. 是否改了设计方向: 不涉及。PR 设计方向(outcome 权威分类)未变;本修复只是让一个测试跟上这个正确变化。
  8. 是否引入了新的配置: 不涉及。无新 env/config/默认值。

判定: 3 个红分片全部修复,最终 head(6f125b6)已推送、无 ci-held 标记,触发完整 CI;等 CI 绿即可合入。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Successful compression can end the turn when its summary contains FAILED

1 participant