Skip to content

[PR] fix(preflight): fold an over-window payload while epoch confirmation is pending (#6069) - #6807

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
Maziar123:fix/epoch-pause-over-window
Nov 18, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
Maziar123:fix/epoch-pause-over-window

Conversation

@Maziar123

Copy link
Copy Markdown
Contributor

Problem

#6069: once the epoch-confirm latch (#3641) is set and the history grows past the model window, every turn is refused with preflight_compress_failed and the session cannot recover.

Fix

The pause now holds only while the verbatim payload fits the window. The gate moves below the entry estimate in preflightCompress, so it judges fit on the same calibrated scale as the fold loop (payloadEstimate + viewDeltaCorrection < limit). When the payload does not fit, preflight logs one warn line and folds as usual.

Nothing else changes: fold-reconcile still sets and clears the latch, the snapshot is kept, and a fitting payload still passes through unfolded.

Evidence

Reproduction through the proxy (the new e2e test): fold a history, resend an unrelated history three times under the same session id, then grow it past the window.

master (e7a42babe) this branch
after 3 passes of the new history latch set, 5 blocks, 0 active same
over-window turn 502 preflight_compress_failed, 0 summary calls 200, 5 summary calls, 5 new active blocks
latch after that turn still set, drift streak keeps growing cleared on the same pass
next turn 502 again 200

Tests

  • node --import ./tests/preload-storage-baseline.mjs --import tsx --test tests/issue6069-epoch-pause-over-window.test.ts → 3 pass. On master the two over-window tests fail (0 summary calls; 502).
  • The same runner over all 53 test files that reference preflightCompress, the latch or preflight_compress_failed → 351 pass, 0 fail.
  • tests/fold-anchor-doctrine.test.ts: the existing pause test now gives its fake session an empty stats object, because the gate runs after the calibration factor is read. Its assertions are unchanged.
  • npm run typecheck, npm run check:unused-exports, npm run check:test-import-graph → clean.
  • node tools/gen-config-docs.mjs check-coverage, node tools/check-architecture-docs.mjs, node scripts/config-surface-guard.mjs, node scripts/lifecycle-ledger-guard.mjs → clean.

Not run: the full test suite and the Windows legs (left to CI).

Notes

Refs #6069

…is pending (ranxianglei#6069)

The ranxianglei#3641 epoch-confirm latch pauses preflight folding and passes the
request through verbatim. An over-window payload cannot pass through, so
the paused preflight reported 0 compressible ranges and every turn was
refused with preflight_compress_failed. The latch clears only after a
turn is forwarded, so the session could not recover.

Keep the pause while the verbatim payload fits the window, judged on the
same calibrated scale as the fold loop, and fold as usual once it does not.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复. Searched open+closed issues/PRs for the epoch-pause over-window deadlock (#6069). #6807 is the only PR fixing this exact problem. Closest neighbor #5275 (open) is complementary, not duplicate: it improves the refusal message for the drift-pause state and fixes a separate empty-processedMessages forwarding bug (#5223); it does not touch the epoch-confirm pause gate. The two are coordinated — this PR's body already accounts for #5275's message.

Recommended labels (suggestion only — not applied, per the no-source-issue-mutation rule): bug · scope/kernel (the preflight fold gate is core context management) · p1 (session unusable: every turn 502-refused, no recovery path).

Review

Direction (①): Correct. The deadlock is real and I reproduced it: on master (e7a42babec) the new e2e test fails 2/3 (over-window turn → 502 preflight_compress_failed, 0 summary calls); on this branch it passes 3/3. The mechanism is sound — once the #3641 latch is set, the only two auto-clears (old history back on the wire, or a fresh fold) are both unreachable while preflight is paused and the turn is refused, so the refusal repeats forever. The fix restores the "fresh fold on the new history" recovery path, which is exactly the "new epoch" outcome #3641's own error message offers.

Fix quality (③): The gate moves from the top of preflightCompress (raw scale, <=) to after the calibrated entry estimate, judging fit with the same expression as the fold loop's exit bar (result.payloadEstimate + viewDeltaCorrection < limit, src/preflight.ts:1814-1820 vs :3275). It is byte-consistent with the loop by construction — the pause holds exactly while the verbatim payload fits, and falls through to the normal fold otherwise. Minimal, single-purpose, root-cause (fixes the producer of the refusal, not a mask).

Regression inventory (②): I walked the existing behaviors the touched path carries:

Tests (④): Real teeth. The new e2e (tests/issue6069-epoch-pause-over-window.test.ts) asserts both gate directions (#3617 discipline): fits → 0 summary calls / latch stays; over-window → ≥1 summary call / fold / latch clears. It fails on master, passes here. The doctrine test change is minimal (adds stats: {} because the gate now runs after the calibration-factor read; assertions unchanged). Local: all 50 preflight-related test files 267/267, 7 fold-mechanism files 132/132, typecheck clean. Full-suite verdict is the CI lanes (still running, 0 failures so far).

Diff cleanliness (⑤): 6 files, +280/−18, single-purpose, no unrelated changes, no whitespace churn. Docs updated in all 3 languages (en/zh-CN/ja) consistently.

#5275 interaction: verified consistent — with this change the over-window state folds instead of refusing, so #5275's "Folding is currently paused…" message is reached only when the fold itself fails, exactly as the PR body states. No textual conflict (different regions of server.ts).

结构化八问

  1. 是什么问题:epoch 确认闩锁(Snapshot as session reconstruction: identity claim order takes priority, message hash demoted to reference signal (root- | 快照即会话重构:身份认领顺序优先,消息哈希降级为参考信号(根治 #3588/#3640 一类误判) #3641)置位后,若重发的历史超过模型窗口,每一轮都被 preflight_compress_failed(502)拒绝,会话永久卡死无法恢复。
  2. 什么原因导致的:闩锁置位时 preflight 在函数顶部直接返回 0 个可折叠区间(逐字透传),但超窗负载无法透传,调用方只能拒绝;而闩锁只有「旧历史回到线上」或「新一轮折叠」两条自动清除路径,两者在「暂停+拒绝」状态下都不可达,形成死锁。
  3. 影响范围与严重程度:触发条件 = 同一 session id 下历史被整体替换(宿主原生压缩 / 新会话复用 id)且连续 3 轮漂移触发闩锁 + 重发历史超过模型窗口。严重度 p1:会话完全不可用(每轮 502),无自愈路径。
  4. 怎么修复的:把暂停判定从函数顶部(原始刻度)下移到校准后的入口估计之后,用与折叠循环退出条件完全相同的表达式(payloadEstimate + viewDeltaCorrection < limit)判定「逐字负载是否装得下窗口」;装得下才暂停透传,装不下就照常折叠。折叠覆盖新历史后,下一轮 reconcile 见已覆盖 id → 干净通过 → 闩锁清除(即 Snapshot as session reconstruction: identity claim order takes priority, message hash demoted to reference signal (root- | 快照即会话重构:身份认领顺序优先,消息哈希降级为参考信号(根治 #3588/#3640 一类误判) #3641 自己命名的「新 epoch」结果)。
  5. 是否有回归问题:审过既有行为——装得下的暂停路径(刻度从原始→校准,更保守;已知基线会话走调用方自己的判定,不变)、hadFoldableCandidates(暂停路径两版皆 false,仅 growth-armed 回退消费)、[Bug][preflight] Folding pass expansion asymmetry: view meets target but outbound reconstruction exceeds window, only ~5 | [Bug][preflight] 折叠 pass 展开不对称:视图达标但出站重建超窗,每轮只推进 ~5k token (#4543-①) #5974 校正二次进入(超窗照常再折叠,与非闩锁一致)、两种压缩模式(门在共享 preflightCompress,handle.ts:3242 不按 pluginMode 分流)、latched+limit<=0 角落(与非闩锁同一退化路径一致,实际不可达)。无回归。
  6. 是否破坏缓存:不涉及——不改缓存/重试/增量游标/去重键;折叠走既有持久化路径。
  7. 是否改了设计方向:不改架构/边界/数据流;只是把 Snapshot as session reconstruction: identity claim order takes priority, message hash demoted to reference signal (root- | 快照即会话重构:身份认领顺序优先,消息哈希降级为参考信号(根治 #3588/#3640 一类误判) #3641 暂停的「唯一恢复路径」从「等人工 epoch 确认(实际无 UI)」落实为「超窗即折叠 = 新 epoch」,与 Snapshot as session reconstruction: identity claim order takes priority, message hash demoted to reference signal (root- | 快照即会话重构:身份认领顺序优先,消息哈希降级为参考信号(根治 #3588/#3640 一类误判) #3641 错误信息自己给出的出口一致。
  8. 是否引入了新的配置:不涉及——无新 env/config/默认值。

判定:可合 (mergeable),pending CI 全量测试车道(目前 0 失败,ubuntu/windows × node 22/24 分片仍在跑)。方向正确、修法打到根因、回归清点干净、测试有牙、diff 干净。以推送 head 的 CI test 车道为最终绿/红判定。

@Maziar123

Copy link
Copy Markdown
Contributor Author

One CI leg is red: test (ubuntu-latest, 24, 2). The failing test is #3380 P2 integration: failures escalate on the ladder, then heal clean (tests/issue3380-dsh-rereg-selflock.test.ts:285, retryAt sits on the 80ms ladder step (got -49ms)).

It compares a wall-clock delta against an 80 ms retry step, so a stalled runner fails it. This change touches only src/preflight.ts, its docs and two tests, and that test file imports none of them. The other 3,770 tests in that leg pass, and the same test is already known as a timing flake (#4599; seen on #5597 and #5178).

Could a maintainer re-run that leg? I cannot re-run it from a fork.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — 该红是已知时序 flake 的一次新发生(档案:#3571 已关闭的追踪 issue + #4599 CI 总控),不是新问题;无在途修复 PR(已查 open PR 提及 3380 的 4 条,均无关),稳定化修复已开 PR #6821。

标签: bug / scope/kernel / p1 — 按 PR 本体定性(p1 可用性修复,preflight fold 门 = 内核域),与楼上 review 建议一致。

Done — re-ran the leg and it is green:

Since this is now the 5th+ cross-PR occurrence (the re-eval condition #4599 set for stabilization), I also opened PR #6821 (test-only, zero production change): it implements #3571's checklist item 1 — the mock records the throw time (which lands in the same event-loop turn as the production arm at src/agent/dsh-native.ts:1041), and the assertion checks retryAt - armTime against the ladder step from _regBackoffForTest(failStreak). Immune to runner stalls; still has teeth (a flat or off-by-one ladder fails by ≥60ms). Merging it makes this leg stop flaking on any PR.

Verdict: CI is now fully green on your head; the earlier review's "mergeable, pending CI" stands — ready for the owner's merge.


八问

  1. 是什么问题:PR [PR] fix(preflight): fold an over-window payload while epoch confirmation is pending (#6069) #6807 的 CI 腿 test (ubuntu-latest, 24, 2) 红,唯一失败是已知时序 flake #3380 P2 integration(tests/issue3380-dsh-rereg-selflock.test.ts:285,got -49ms),与本 PR 代码无关。
  2. 什么原因导致的:该断言用 snap.retryAt - Date.now() 采样 [30, 95]ms 窗口——量的是「arm 到观察」的观察延迟,不是退避值本身;runner 负载下,第 3 次失败 arm(retryAt = now + 80ms)到读快照之间漂了 129ms,窗口被击穿。根因在测试写法(墙钟采样),不在生产代码。
  3. 影响范围与严重程度:仅 CI 假红,不影响任何用户/运行时行为;触发条件 = 该 shard 恰好负载高。严重程度:体验瑕疵(挡合并、浪费重跑),flake 本身 p3 级;PR 本体是 p1 修复(标签按 PR 本体打)。
  4. 怎么修复的:两层——①当场:重跑该腿(rerun-failed-jobs,run 43698036798 attempt 2)→ 已绿,全 run success;②根治:PR [PR] test: make the #3380 P2 ladder assertion state-based, immune to CI runner stalls #6821(仅测试,零生产改动)把断言改为状态式——在 mock 的 registerError 回调里记录 throw 时刻(与生产 arm 同一 event-loop turn,src/agent/dsh-native.ts:1041),断言 retryAt - armTime 落在 _regBackoffForTest(failStreak) 的紧窗口内,对 runner 停顿免疫;即 [no-dispatch] Windows CI shard intermittently red: two real-time-sensitive tests fail under runner load (seen twice on #3540) #3571 清单第 1 项。
  5. 是否有回归问题:无。重跑不改任何代码;[PR] test: make the #3380 P2 ladder assertion state-based, immune to CI runner stalls #6821 只改一个测试文件的断言,断言仍有牙(阶梯错一步 20/40/160ms 都会以 ≥60ms 差距失败),生产行为零变更。
  6. 是否破坏缓存:不涉及(纯测试断言改写,无缓存/重试/游标逻辑)。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

收工判定:CI 全绿,PR #6807 可合并(等 owner);#6821 合入后该 flake 不再挡任何 PR。

@ranxianglei ranxianglei added bug Something isn't working p1 scope/kernel labels Nov 15, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Re-verified live this run: run 43698036798 attempt 2 is fully green — all 28 jobs success, including test (ubuntu-latest, 24, 2); run conclusion success; mirror cache: 0 failed / 0 pending. This event reports the attempt-1 red, which the earlier wake already fixed via rerun-failed-jobs (see the comment above — root cause was the known #3380 wall-clock flake, stabilization in PR #6821). Nothing to fix; no further work.

八问

  1. 是什么问题:本次唤醒报告的红腿 test (ubuntu-latest, 24, 2) 是 attempt 1 的已知 Plugin tool registration self-lock: after a failed proxy respawn, re-registration is blocked by its own prior registrations — identity headers lost forever (wire mode) → 413 preflight_compress_failed on large sessions | [Bug] 插件工具注册状态自锁:proxy respawn 失败后重注册被自己挡死,永久失去身份头(wire 模式)→ 大会话 413 preflight_compress_failed #3380 时序 flake,非本 PR 代码问题。
  2. 什么原因导致的:测试以墙钟采样断言退避窗口([30, 95]ms),runner 负载下 arm 到观察漂移 129ms 击穿窗口;生产代码无缺陷。
  3. 影响范围与严重程度:仅 CI 假红,零用户/运行时影响;已消除。
  4. 怎么修复的:上一轮已 rerun-failed-jobs,attempt 2 全绿;根治在 PR [PR] test: make the #3380 P2 ladder assertion state-based, immune to CI runner stalls #6821(状态式断言,仅测试)。
  5. 是否有回归问题:无——本轮零代码改动,仅核实状态。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

收工判定:PR #6807 全部 check 绿,可合并(等 owner)。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

@ranxianglei
ranxianglei merged commit 6c1832d into ranxianglei:master Nov 18, 2026
115 of 154 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p1 scope/kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants