Repository navigation
[PR] fix(preflight): scope the usage baseline and k̂ estimate to the model that measured them - #6467
ranxianglei wants to merge 6 commits into
Conversation
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.10982-pr.6467.9fb168bThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr6467.tgz
npm install -g packageInstall instructions are refreshed on each push. |
5289944 to
19ab036
Compare
…switch e2e The #6430 model-scoped baseline demotion is a third path into the baselineFloor===0 state the #2313 forward-once escape anticipates: after a mid-lane model switch the cross-model baseline no longer drives the trigger, so the first switched-model request is forwarded to acquire the new model's usage evidence (in the real world the over-window forward 400s and arms the overflow baseline) and the second request folds. The test now asserts both legs: req2 forwards without summarization and settles a fresh same-model baseline; req3 folds before forwarding. Found in PR #6467 review: the old test asserted a fold on the first switched-model request, which the intended #6430 behavior (stated in the PR body) no longer does.
2e64cc3 to
ad1ff55
Compare
…switch e2e The #6430 model-scoped baseline demotion is a third path into the baselineFloor===0 state the #2313 forward-once escape anticipates: after a mid-lane model switch the cross-model baseline no longer drives the trigger, so the first switched-model request is forwarded to acquire the new model's usage evidence (in the real world the over-window forward 400s and arms the overflow baseline) and the second request folds. The test now asserts both legs: req2 forwards without summarization and settles a fresh same-model baseline; req3 folds before forwarding. Found in PR #6467 review: the old test asserted a fold on the first switched-model request, which the intended #6430 behavior (stated in the PR body) no longer does.
(ai: vllm-qwen/qwen3.8-27b) CI 冲突修复完成(PR #6467,分支 七问:
判定:冲突已解决、分支已同步 master,可进入 CI 重跑。 |
(ai: vllm-qwen/qwen3.8-27b) 七问 summary — PR #6467 red-CI fix (attempt 1/6)
|
33822b5 to
e1a7fbc
Compare
…switch e2e The #6430 model-scoped baseline demotion is a third path into the baselineFloor===0 state the #2313 forward-once escape anticipates: after a mid-lane model switch the cross-model baseline no longer drives the trigger, so the first switched-model request is forwarded to acquire the new model's usage evidence (in the real world the over-window forward 400s and arms the overflow baseline) and the second request folds. The test now asserts both legs: req2 forwards without summarization and settles a fresh same-model baseline; req3 folds before forwarding. Found in PR #6467 review: the old test asserted a fold on the first switched-model request, which the intended #6430 behavior (stated in the PR body) no longer does.
(ai: vllm-qwen/qwen3.8-27b) CI fix: aligned the #663 model-scoping test with the PR's #6430 cross-model demotion — pushed What I did this run:
结构化八问(中文)
收工判定:修复已推送( |
(ai: vllm-qwen/qwen3.8-27b) No action taken — the reported conflict is stale; verified live this run.
Stopping here per the re-verify-first rule. If the in-progress CI pass comes back red, the next wake will handle it. |
ranxianglei
left a comment
There was a problem hiding this comment.
APPROVE
The fix correctly scopes the preflight usage baseline and k̂ estimate to the model that measured them. Both incident cases (#6430: cross-model baseline driving spurious preflight, stale k̂ over-correction) are addressed by a single coherent mechanism: model-scoped baseline demotion (Fix A) and a fresh-measurement cap on the calibrated estimate (Fix B). The tests pin both directions plus the control (no regression). The behavior changes are disclosed in the PR body.
8问
- 是什么问题: 预检大小模型(本地估计 × k̂ + 使用基线)在两个方向上偏离了实测使用量,导致虚假折叠。
- 什么原因导致的: 使用基线只携带路由(origin)来源,没有模型来源;跨模型基线(同路由)在模型切换后仍然驱动触发器。过时的 k̂>1 会膨胀本地估计,超过新鲜的目标以下基线。
- 影响范围与严重程度: 受影响的用户面: 在会话中途切换模型的会话。触发条件: 基线模型与当前请求模型不同,或过时的 k̂>1 膨胀估计。严重程度: 特定条件下有问题(虚假预检折叠, 13.7s-303s 延迟)。
- 怎么修复的: Fix A: 记录测量基线的模型,在触发器中,当基线模型与当前请求模型不同时,将基线降级为 0。Fix B: 当新鲜(同路由+同模型)基线存在且负载的原始估计不大于它时,将校准估计限制在基线以内。
- 是否有回归问题: 审过的既有行为: #663 模型范围流程(已更新测试以反映新的两请求自愈行为), preflight-compress e2e 测试(已更新), golden 文件(已重新生成)。没有发现意外的回归。
- 是否破坏缓存: 不涉及。
- 是否改了设计方向: 不涉及。基线现在也按模型范围化,这与 k̂ 读取门的模型维度一致。
- 是否引入了新的配置: 不涉及。
lastInputTokensModel是一个内部统计字段(与lastInputTokensOrigin一起持久化),不是配置旋钮。
合并判定: 可以合并。
74305b7 to
710a36f
Compare
…switch e2e The #6430 model-scoped baseline demotion is a third path into the baselineFloor===0 state the #2313 forward-once escape anticipates: after a mid-lane model switch the cross-model baseline no longer drives the trigger, so the first switched-model request is forwarded to acquire the new model's usage evidence (in the real world the over-window forward 400s and arms the overflow baseline) and the second request folds. The test now asserts both legs: req2 forwards without summarization and settles a fresh same-model baseline; req3 folds before forwarding. Found in PR #6467 review: the old test asserted a fold on the first switched-model request, which the intended #6430 behavior (stated in the PR body) no longer does.
(ai: vllm-qwen/qwen3.8-27b) Conflict resolved and rebased — new head This wake's conflict was real (unlike the previous one): master moved to
|
(ai: vllm-qwen/qwen3.8-27b) No action taken — this conflict wake is stale; the rebase from the previous run already cleared it. Verified live this run:
Stopping here per the re-verify-first rule; if the in-flight pass on |
ranxianglei
left a comment
There was a problem hiding this comment.
APPROVE
Re-review after rebase: content is identical to the previously reviewed head (same 6 commits, same 14-file diff, +356/-32). The fix correctly scopes the preflight usage baseline and k̂ estimate to the model that measured them. Both incident cases (#6430: cross-model baseline driving spurious preflight, stale k̂ over-correction) are addressed by a single coherent mechanism: model-scoped baseline demotion (Fix A) and a fresh-measurement cap on the calibrated estimate (Fix B). The tests pin both directions plus the control (no regression). The behavior changes are disclosed in the PR body.
8问
- 是什么问题: 预检大小模型(本地估计 × k̂ + 使用基线)在两个方向上偏离了实测使用量,导致虚假折叠。
- 什么原因导致的: 使用基线只携带路由(origin)来源,没有模型来源;跨模型基线(同路由)在模型切换后仍然驱动触发器。过时的 k̂>1 会膨胀本地估计,超过新鲜的目标以下基线。
- 影响范围与严重程度: 受影响的用户面: 在会话中途切换模型的会话。触发条件: 基线模型与当前请求模型不同,或过时的 k̂>1 膨胀估计。严重程度: 特定条件下有问题(虚假预检折叠, 13.7s-303s 延迟)。
- 怎么修复的: Fix A: 记录测量基线的模型,在触发器中,当基线模型与当前请求模型不同时,将基线降级为 0。Fix B: 当新鲜(同路由+同模型)基线存在且负载的原始估计不大于它时,将校准估计限制在基线以内。
- 是否有回归问题: 审过的既有行为: #663 模型范围流程(已更新测试以反映新的两请求自愈行为), preflight-compress e2e 测试(已更新), golden 文件(已重新生成)。没有发现意外的回归。
- 是否破坏缓存: 不涉及。
- 是否改了设计方向: 不涉及。基线现在也按模型范围化,这与 k̂ 读取门的模型维度一致。
- 是否引入了新的配置: 不涉及。
lastInputTokensModel是一个内部统计字段(与lastInputTokensOrigin一起持久化),不是配置旋钮。
合并判定: 可以合并。
Re-review complete on rebased head 710a36f: content identical to previously reviewed head (same 6 commits, same diff). APPROVE posted (comment-type, self-approval blocked by GitHub). 8问 in Chinese included. CI green (30 checks, 0 failed). No small findings. No kernel/ changes. Verdict: mergeable. |
[ework] CI fix verified green ✓ |
1 similar comment
[ework] CI fix verified green ✓ |
… that measured them (#6430) The preflight size model diverged from the measured usage in both directions, firing spurious folds: 1. Cross-model baseline: a usage baseline measured under a prior model on the same route carried no model provenance, so it survived the #1933 F2 route demotion and drove the trigger while the current payload's own estimate read far below the target. 2. Stale k̂ over-correction: a fresh same-route+same-model usage baseline (below the target) was overridden by a stale k̂>1 that inflated the local estimate past the target. Both are one mechanism: the baseline floor and the k̂-scaled estimate must be scoped to the model that measured them, and a fresh measurement must win over a stale calibration. Fix A: record the model that measured the usage baseline (lastInputTokensModel) and demote the baseline floor on model mismatch, mirroring the k̂ read gate's model dimension. Fix B: when a fresh (same route+model) baseline is present and the payload's raw estimate is not bigger than it, cap the calibrated estimate at the baseline — the k̂-scaled excess is over-correction. Fixes #6430
… (session.ts) The #6430 change shifted the existing lastInputTokensOrigin field and the #1933 F1 comment line one space right (8→9 / 9→10), leaving the #1933 F1 comment internally inconsistent. Restore the original indentation and indent the new field/comment to the block's field level (8/9), matching the neighboring #1933 F2 comment. Type-only change; typecheck clean.
…switch e2e The #6430 model-scoped baseline demotion is a third path into the baselineFloor===0 state the #2313 forward-once escape anticipates: after a mid-lane model switch the cross-model baseline no longer drives the trigger, so the first switched-model request is forwarded to acquire the new model's usage evidence (in the real world the over-window forward 400s and arms the overflow baseline) and the second request folds. The test now asserts both legs: req2 forwards without summarization and settles a fresh same-model baseline; req3 folds before forwarding. Found in PR #6467 review: the old test asserted a fold on the first switched-model request, which the intended #6430 behavior (stated in the PR body) no longer does.
…tInputTokensModel field The new stats field changes the persisted session state, so the storage-render goldens' state digests drift. Regenerated with scripts/update-storage-render-goldens.ts (intentional persist contract change — the field is the PR's own). Diff is surgical: per wire only stateDigest, stateDigestAligned and sectionDigests.stats changed; all per-turn wire digests are unchanged, so the outbound wire is untouched.
The PR's Fix A demotes a usage baseline measured under a different model (cross-model baselines are no longer inherited — tokenizers bill differently across models). The #2189 test's setup incidentally relied on a cross-model baseline (300K settled under claude-big, small payload under claude-small) to fire preflight on r2; under the new behavior that cross-model baseline is correctly demoted, so preflight no longer fires and the 4 billing/429 tests fail. Change r1's model from claude-big to claude-small (same as r2) so the 300K baseline is same-model and not demoted. Preflight still fires; the test's real purpose (billing-attribution block + 429 handling) is preserved.
The PR's Fix A demotes a usage baseline measured under a different model. In the #663 model-scoping test, r2 (same session, model B, grown over-window) now hits exactly the state the #2313 forward-once escape is designed for: the trigger fires on the payload's own estimate while the cross-model baseline is demoted to untrusted (baselineFloor === 0 with a nonzero lastInputTokens), so the proxy forwards once to acquire model B's evidence instead of folding first. The mock upstream accepts model B, settles a fresh model-B usage baseline, and no summary call is made on r2 — the old expectation ('second request triggered preflight summaries') no longer holds. Extend the flow by one request: r2 asserts the evidence forward (no summary call), r3 (same model, still over-window) folds under the fresh trusted model-B baseline and its summaries keep the 32768 cap — the test's real purpose (model A's learned rejection must not strip the cap from model B's summaries) is preserved and still pinned.
710a36f to
b885591
Compare
(ai: vllm-qwen/qwen3.8-27b) Re-rebased onto the new master — new head This wake's conflict was real: master moved to
|
[ework] CI fix verified green ✓ |
ranxianglei
left a comment
There was a problem hiding this comment.
APPROVE
Third re-review: content identical to the two previously reviewed heads (same 6 commits, same 14-file diff, +356/-32). Rebase onto newer master only — no code changes. Fix A (model-scoped baseline demotion) and Fix B (fresh-measurement cap) verified correct. Tests pin both directions plus control. Behavior changes disclosed. CI green (30/30).
8问
- 是什么问题: 预检大小模型(本地估计 × k̂ + 使用基线)在两个方向上偏离了实测使用量,导致虚假折叠。
- 什么原因导致的: 使用基线只携带路由(origin)来源,没有模型来源;跨模型基线(同路由)在模型切换后仍然驱动触发器。过时的 k̂>1 会膨胀本地估计,超过新鲜的目标以下基线。
- 影响范围与严重程度: 受影响的用户面: 在会话中途切换模型的会话。触发条件: 基线模型与当前请求模型不同,或过时的 k̂>1 膨胀估计。严重程度: 特定条件下有问题(虚假预检折叠, 13.7s-303s 延迟)。
- 怎么修复的: Fix A: 记录测量基线的模型,在触发器中,当基线模型与当前请求模型不同时,将基线降级为 0。Fix B: 当新鲜(同路由+同模型)基线存在且负载的原始估计不大于它时,将校准估计限制在基线以内。
- 是否有回归问题: 审过的既有行为: #663 模型范围流程, preflight-compress e2e, golden 文件。没有发现意外的回归。
- 是否破坏缓存: 不涉及。
- 是否改了设计方向: 不涉及。
- 是否引入了新的配置: 不涉及。
合并判定: 可以合并。
[ework] CI fix verified green ✓ |
Summary
The preflight size model (local estimate × k̂ + usage baseline) diverged from the measured usage in both directions, firing spurious folds (#6430):
Both are one mechanism: the baseline floor and the k̂-scaled estimate must be scoped to the model that measured them, and a fresh measurement must win over a stale calibration.
Fix
Fix A — model-scoped baseline demotion. Record the model that measured the usage baseline (
lastInputTokensModel, set insettleUsageReportand both overflow arms). In the trigger, after the #1933 F2 origin demotion, demote the baseline floor to 0 when the baseline's model differs from the current request's model (both known). Mirrors the k̂ read gate's model dimension (currentCalibrationFactor).Fix B — fresh-measurement cap on the calibrated estimate. When a fresh (same route+model, not demoted) baseline is present and the payload's raw estimate is not bigger than it, cap the calibrated estimate at the baseline. Rationale: the raw estimate is a direct measurement of the current payload; if it is not bigger than what upstream just billed on this exact route+model, the payload has not genuinely grown — the k̂-scaled excess is over-correction. Genuine growth (raw > baseline) leaves the estimate untouched and still folds. A demoted baseline floors at 0 so the cap is a no-op.
Tests
tests/issue6430-preflight-baseline-model.test.ts(new): Fix A (cross-model baseline demoted → no spurious preflight), Fix B (fresh same-model baseline caps a stale k̂ over-correction → no spurious preflight), control (same-model baseline above the window still drives preflight — no regression).tests/issue2129-restart-measurement-state.test.ts: addedlastInputTokensModelto the persist round-trip assertion.Behavior change
Config surface
None.
lastInputTokensModelis an internal stats field (persisted alongsidelastInputTokensOrigin), not a config knob.Closes #6430