Skip to content

[PR] fix: retry context overflow with measured output headroom - #6529

Merged
ranxianglei merged 2 commits into
ranxianglei:masterfrom
zouzh22:2026-11-13_measured-overflow-retry
Nov 15, 2026
Merged

ranxianglei merged 2 commits into
ranxianglei:masterfrom
zouzh22:2026-11-13_measured-overflow-retry

Conversation

@zouzh22

@zouzh22 zouzh22 commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Fixes #6528.

When upstream reports 108,000 input tokens and a 32,000 output reservation against a 128,000 window, the input already fits. The relay currently goes straight to refolding. If preflight exhausts its compression budget, the client receives the original 400.

Retry the same payload with output capped at 20,000 in that example. The calculation uses the rejected request's measured input and verifies that the reported completion count matches the outbound field. If recovery still needs a refold, apply the same rule to that attempt's counts. Preserve explicit thinking budgets and stop on unrelated errors. Routes with configured egress transforms keep the existing refold path because their retry can carry different input.

Validation

  • Based on upstream 99c22d2. The new synthetic cases cover initial budget-only recovery, recovery after refold, input that is still too large, a mismatched completion count, and a rate limit on the retry.
  • 103 targeted tests pass, including the overflow, calibration, wire-contract, output-headroom, egress-transform, and preflight re-signing suites.
  • Typecheck, build, and diff checks pass.
  • Real DSH with isolated state and a fake upstream passes two smoke checks. The external Docker attach case was skipped by its gate.

The declared model window, tokenizer, and fork inheritance rules stay unchanged. This PR covers the measured-output recovery path. Human review before merge.

@ranxianglei ranxianglei changed the title fix: retry context overflow with measured output headroom [PR] fix: retry context overflow with measured output headroom Nov 13, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。最近的在途工作 — #5880/#5899(estimate-only fit:bili 的本地估计认为能放下但上游拒绝)和 #5382(overflow-stated counts 驱动折叠目标)— 机制不同。本 PR 的机制是独立的:当上游报告的计数证明输入本已放得下时,封顶输出预算并重试,而不是重新折叠。没有并行的 open PR 修同一问题。

标签: bug · dsh · scope/wiring · p2 — relay 溢出恢复路径上的 bug 修复(协议接线层),在 DSH 上报/复现;定 p2 因为它是特定条件失败(overflow 且输入本已放下)且有手工绕行(调小输出预算),不是全局不可用或缓存泄露。

Review 状态 — CI 是红的,且是 PR 自己引入的:

  • PR head 上有 3 个 test shard 失败:test (ubuntu-latest, 24, 3)、test (ubuntu-latest, 22, 3)、test (windows-latest, 22, 4)。
  • 完全相同的 base(master @ 99c22d2)在这三个 shard 上全是绿的 → 这是 PR 引入的回归,不是继承性失败。
  • 根因(来自 shard 日志):失败的测试是 tests/side-lane-invariants.test.ts 里的 (H) extracted relay module keeps the side-lane boundary (#1440 P2 cut 4),断言 "relay.ts must not import the side-lane engine directly"。PR 在 src/server/relay.ts:77 加了 import { outputBudgetField, readOutputBudget, writeOutputBudget } from "./side-request.js",把 relay(wire 尾部)直接耦合到 side-lane 引擎 —— 正是这条边界测试禁止的。

核心修复本身是扎实的。 我读了完整 diff:measured-output cap 的计算和它的守卫(total===input+output、字段匹配、保留 thinking budget、排除 egress-transform、排除 Buffer)都正确且保守;新增测试是真实的(非恒真),覆盖了恢复路径和负例。

我在做什么: 按外部 PR 质量政策,我直接在 PR 分支上修这个边界违规(不开并行 PR):这三个 output-budget helper 是通用的 wire-shape helper,src/server/budget.ts 本来就从 side-request.js import 它们(而 relay.ts 本来就 import budget.js 且被允许 import),所以我把它们从 budget.ts re-export 出来,把 relay.ts 指向 ./budget.js 而非 ./side-request.js。无行为变化、无循环依赖。我会验证(typecheck + boundary/overflow/budget 套件)后 push,然后贴完整的八问 review。

@ranxianglei ranxianglei added bug Something isn't working dsh p2 scope/wiring labels Nov 13, 2026
…-lane boundary

relay.ts (the wire tail) must not import the side-lane engine directly
(ranxianglei#1440 P2 cut 4; tests/side-lane-invariants.test.ts test H). The overflow
recovery (ranxianglei#6528) added an import of outputBudgetField/readOutputBudget/
writeOutputBudget from ./side-request.js, coupling relay to the side-lane
engine and breaking test H on 3 CI shards (master is green on the same
shards).

The three helpers are generic wire-shape helpers already imported by
src/server/budget.ts; re-export them from budget.ts and point relay.ts at
./budget.js. No behavior change, no circular dependency.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

PR #6529 Review — 结构化八问

1. 是什么问题
PR 为上下文溢出恢复新增"按实测输出余量重试"路径(#6528):当上游报溢出、但 input 其实已能放进窗口时,relay 直接走 refold;若 preflight 耗尽压缩预算,客户端会收到原始 400——而把 output 上限压到 window - input 本可救回。

2. 什么原因导致的
relay 的溢出恢复(src/server/relay.ts)只有 refold 一条路,没有"仅压 output 上限"的重试。修复新增 capOverflowOutput(把 output 压到 window - input,带保守守卫)+ 在 refold 前/后各一次 budget-only 重试。

3. 影响范围与严重程度
特定条件触发(溢出 + input 已能放进窗口 + 压 output 可救回)。p2:可用,但该恢复路径次优——客户端收到本可避免的 400。

4. 怎么修复的
新增 budget-only 重试:refold 前把 output 压到 window - input 重发;若 refold 后仍溢出,对折叠后的计数同样压一次。保守守卫:total === input + output、上报 output 与线上字段匹配、保留 thinking budget、egress transform 存在时禁用(其重试可能带不同 input)、Buffer 排除。每次重试以上游判决自校验。

5. 是否有回归问题
有,已修。 PR 的 CI 在 3 个 test shard 红(ubuntu 24/3、ubuntu 22/3、windows 22/4),根因是 side-lane 边界违例:relay.ts 直接 import { outputBudgetField, readOutputBudget, writeOutputBudget } from "./side-request.js",破坏 test H(#1440 P2 cut 4,"relay 不得直接 import side-lane 引擎")。master(99c22d2,PR 的 base)在同 shard 全绿 → 是 PR 引入的回归,非继承。
已修:3 个 output-budget helper 改经 budget.js 转发(src/server/budget.ts 本就从 side-request.js import 它们),relay.ts 不再直接 import side-request.js;无循环依赖。验证:typecheck 通过 + 69 个定向测试全绿(含 test H)。commit 9c6e9b3650 已推到 PR 分支。
行为变更(old→new,均保守/自校验):

6. 是否破坏缓存
否。budget 重试重发同一 input(messages 前缀逐字节相同),仅改 output 上限(顶层字段,非前缀一部分),前缀缓存保留。

7. 是否改了设计方向
无架构变更。修复在既有溢出恢复路径内。(我补的边界修复遵守既有模块边界 #1440。)

8. 是否引入了新的配置
无新配置。

判定:核心修复正确且保守;阻断性 CI 失败(side-lane 边界违例)已修并本地验证(typecheck + 69 测试),CI 将在新 head 9c6e9b3650 重跑。剩余项:egress-gate 负向测试(建议合并前补,gate 保守故非阻断)。可合(egress-gate 测试作为建议项;最终合并由 owner 决定)。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Follow-up — CI re-run on my fix (9c6e9b3650):

  1. The boundary violation (test H) is resolved. The 3 shards that were red on the original head no longer fail on (H) extracted relay module keeps the side-lane boundary — routing the output-budget helpers through budget.js fixed it.

  2. New failures are NOT from this PR. The re-run now fails on visibleNotices tests (tests/notice-registry.test.ts), expecting storage-sqlite-default to be visible. That test and the storage-sqlite-default notice come from master advancing since the PR base: 99c22d2 → a007974683 (PR [PR] storage: pure-SQLite default — JSON envelopes stop being written unless dual-write is requested (#6479) #6480 "pure-SQLite default", owner decision 2026-11-13). CI runs the merge of (PR head + current master), so the new master's state-dependent notice test is now in the run. This PR touches neither notices nor storage (src/notices.ts, src/storage/* untouched), so it cannot produce this failure — it's inherited from the new master's notice work, and the test is state-dependent (fresh-install gating), i.e. flaky-prone in CI.

Recommendation: re-run the failed shards (or land the notice-test stabilization on master / ci-fixes) — the PR's own change is clean and the original regression is fixed. I have not merged ci-fixes into the PR branch since this is an external PR and the inherited failure belongs to master's notice work, not this diff.

@ranxianglei
ranxianglei merged commit 0a73563 into ranxianglei:master Nov 15, 2026
48 of 52 checks passed
@ranxianglei

Copy link
Copy Markdown
Owner

Closes pair #6528 was auto-closed by the batch merge.

Superseded by #6765 (merged to master as 0a73563).

Your commit 9c6e9b3 is in master verbatim — same content, same authorship, only the object ID changed (the batch used a real git merge, no rebase or cherry-pick rewrite). Merged under a different branch name, so GitHub does not flip this PR by itself.

Thanks for the fix — it is live in master from 0a73563 onward.

ranxianglei pushed a commit that referenced this pull request Nov 15, 2026
Master advanced 379 commits (incl. nudge-rearm #6531, side-passthrough,
preflight/fold, relay output-budget #6529) since the prior sync (9674cfa).
4 of the PR's 5 files overlapped; 3 auto-merged cleanly. The one conflict,
src/server/prepare-anthropic.ts, resolved as the union of both intents:
keep #4974's if (!anthropicTitleGen) gate wrapping the meter writes (the
meter writes are byte-identical on both sides modulo indent) and adopt
master's return which adds nudgeStampsBefore (nudge-rearm #6531). typecheck
clean; issue4974 + side-lane-invariants + issue2490 + issue2847 = 30/30 and
anthropic nudge suites = 7/7 pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p2 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][dsh] Context-overflow recovery refolds input that already fits the upstream window

2 participants