Repository navigation
fix(preflight): calibrate local estimates against provider usage, provenance-gate the baseline, scale the fold budget (#1933) - #1940
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.1968-pr.1940.e698d77This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr1940.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 PR review 完成(拉分支逐段核对代码 + 机制验证 + CI 全量确认)。结论:可合,CI 15/15 全绿,合并由人来。 审核方式说明:不是只读 diff——把 结构化七问
测试有效性(④关)
CI 与 diff 整洁度
判定:可合。 行为变更已在 PR 描述声明(刻度 max(raw,usage)→max(k̂·raw,usage)、基线跨路由降级、预算 16→≤32、新日志行),均为有意变更且 legacy 路径保守回退。合并操作留给人:#1940 |
|
Reviewed against issue #1933's four defects + master hot-path context. Verdict: high quality, fixes defects 1–2 cleanly, softens 3, defers 3/4 — one real gap found. Verified locally: calibration suite 10/10, preflight-round-budget + multi-range green, typecheck clean; CI 15/15. What's sound
Finding 1 (real gap): Sequence: usage settles on route A ( Fix is one line per branch: Finding 2 (scope): Nits
|
a5e68d2 to
706a96c
Compare
[bot] 🏷 Review round 2 findings 已全部在本分支直接修掉并推送(rebase 到当前 master Finding 1(arm-origin gap)— 确认成立,已修。 你的序列分析完全准确: 回归 e2e(calibration suite 新增 F2b,全真实 proxy 链路):A 上 settle usage → 切 B 带全量历史重发 → B 对未折叠 payload 返回带窗口的 overflow 400 → 断言 ①救援生效(refold 折叠后重发,client 见 200 而非裸 400)②基线 origin === B(sharp pin,pre-fix 停在 A)③救援后流量继续流动(无 502 锁死——即事件终态的回归守卫)。已验证 red-without-fix(去掉两行 → 400 passthrough 断言失败)、green-with-fix。测试设计过程中还排掉两个坑:mock 若按内容 marker 拒绝会撞上 kernel 的 minSummaryLength 与范围分组机制;最终用「payload 是否含折叠载体」作拒绝判据,与 #1195 的产出物精确对齐、无算术边界。 Nits: ①死参数 七问逐项:
Scope:#1950(你已立)承接 Evidence-D 剩余(对象不匹配→任何预算不收敛→fail-fast 硬失败),PR 正文已更新引用,#1933 的 auto-close 不丢范围。CI 正在新 head 上跑,绿了我会再报一句。合并仍由人来。 |
[bot] 🏷 CI 全绿:head 当前状态汇总:
PR ready for merge — 合并由人来:#1940 |
706a96c to
c110edb
Compare
|
追加收紧(0ea4a17):k̂ 校准改为单向 —— clamp [0.25,4] → [0.25,1],外加 apply 站点无效因子防护。 改了什么
为什么单向
验证:clamp/cross-route 单测改期望(k̂>1 → 1),apply 站点新增 0/null/NaN/∞/负 用例;e2e F1(0.5 放气路径)不受影响仍绿。typecheck ✔、校准相关 5 个套件 86 用例全绿 ✔、build ✔。( |
…venance-gate the baseline, scale the fold budget (#1933) F1: learn a per-route estimate-calibration factor (k-hat) from settled pairs of the local billed-caliber estimate and the upstream usage report. A sample is admitted only when plausible (estimate >= 2000 tok, ratio in [0.2, 5] — outside it the report demonstrably does not correspond to this payload); up to 3 same-route samples are kept in a ring and the factor publishes only once two agree within x2 (clamped mean, [0.25, 4]). Disagreement or a route switch clears it back to raw behavior, so degenerate billing (placeholder reports, relay echoes, mocks) can never poison a judgment. Applied at every preflight judgment site that uses the local estimate (gate trigger/fit, folder per-round exit/final fit, post-fold fit) so all sites share one scale. F2: record which upstream origin measured lastInputTokens; when the request now routes to a different origin the usage-baseline floor is demoted to untrusted (info log) instead of firing preflight on another provider's scale. F3: scale the preflight summarization/round budgets with entry overshoot (base 16, cap 2x) so a payload far over the window gets the folds it needs instead of exhausting a fixed 16-call budget mid-shrink into a fail-fast 502. F4: the trigger log carries both measurement scales (usage-baseline, calibrated local est, raw est + k-hat) for field diagnosis. Fixes #1933
…apply-site factor (#1933) Calibration may now only DEFLATE the estimate (fire later than the raw proxy would), never inflate it: routes billing above the local estimate (samples > 1) publish k̂ = 1 — legacy raw behavior, still covered by the overflow arm / learn-on-failure ladder. This eliminates the class "calibration itself causes an earlier trigger" outright; the #1933 damage was over-triggering, and the opposite error already has backstops. applyEstimateCalibration now degrades invalid factors (null / 0 / NaN / ±Infinity, e.g. corrupted persisted session.stats) to no-correction at the single shared apply site — raw × 0 would blind the estimate arm.
0ea4a17 to
75f70a8
Compare
[bot] 🏷 追加收紧( 审核要点
七问逐项:
状态:rebase 到 master |
What / why
Issue #1933: on DSH (OpenAI-compatible relay, full-history resend every turn) the gate logged
context ~257K tokens reached preflight target 256000while upstream usage measured only 151–161K (59–63% of the window); after a mid-session provider switch a stale cross-provider usage baseline fired preflight even on small payloads; worst case (issue Evidence D): a 1.5M-token resend was folded for 12 rounds until the fixed 16-call summarization budget exhausted → fail-fast 502 on a payload the provider had just billed at ~50K.Root causes (verified against v0.1.180 source
dd7631c1; the issue's code quote matches master exactly):max(usage-baseline, local-estimate), but the local chars/4 estimator's ratio to real billing varies per upstream — observed in the issue log: 1.30–2.47× on one relay vs ~1.0× on another in the same session. On a relay that bills below the estimator, the raw estimate alone crosses the target while real usage sits far under it.lastInputTokenscarries no record of which upstream measured it, so it survives provider/window switches and fires the gate on another provider's scale.Changes (F1–F4)
settleUsageReportrecords which upstream origin measuredlastInputTokens; when the request now routes to a different origin, the usage-baseline floor is demoted to untrusted (info log) and the gate judges by this payload's own calibrated estimate.Config surface
No new config fields, env vars, or CLI flags. Only additive session-stats persistence fields (
lastInputTokensOrigin,calibratedEstimate,calibratedEstimateOrigin,calibrationRing,lastLocalTextEstimate+lastLocalTextEstimateOrigin) — all optional, absent on legacy sessions, ignored when missing. No config schema change, no persistence format version bump.Behavior changes (old → new)
max(raw-local-estimate, usage-baseline)→max(k̂·raw-local-estimate, usage-baseline). k̂ ∈ [0.25, 1] — one-way by design (Review round 3: the correction may only deflate the estimate, never inflate it), published only after ≥2 consistent same-route samples (each: est ≥ 2000 tok, ratio ∈ [0.2, 5]; ring ≤ 3, spread ≤ ×2), cleared on disagreement or route change. Until k̂ publishes (legacy sessions, first turns, unknown/mismatched routes) behavior is byte-identical to today.Not implemented (tracked separately): the Evidence-D remainder — trigger object ≠ fold object on protected-heavy resends, where no budget converges and fail-fast still hard-fails — is filed as #1950 (suggested direction: generalize the #496 forward-once-then-learn pattern with evidence gating). Downgrading fail-fast to forward-with-warning and last-resort emergency truncation remain owner decisions inside that issue.
Relation to #1849 / #1929
Orthogonal lines: #1849/#1929 fix the window denominator (host window delivery + credibility visibility); this PR fixes the trigger numerator (what the gate measures). This incident's window was explicitly configured (
compress.modelContextLimit), so the #1929 path does not touch it. No shared code regions (#1929: window ladder, stats panel, dsh-native, client-config; here: preflight gate, cache-ledger, prefolder). If #1929 merges first I will rebase onto it.Tests
tests/issue1933-preflight-calibration.test.ts(11 tests): calibration-ring unit pins (publish-at-two-consistent, cross-route clearing, plausibility band, clamps, no-upstream safety, compaction-boundary reset) plus three e2e regressions through the real proxy: (a) a relay billing at 0.5× the estimator teaches k̂≈0.42, after which an over-raw payload that would have fired preflight on master forwards with zero summary calls; (b) a route switch demotes the stale baseline so a small post-switch request is not folded; (c) overflow-arm provenance (Review round 2): usage settles on route A, the session switches to B, B rejects an unfolded payload with a stated window — the arm stamps B's origin so the baseline stays authoritative and the 使用pi 压缩后就提示prompt is too long #1195 same-request refold rescue delivers 200 to the client.tests/preflight-round-budget.test.ts/tests/preflight-multi-range.test.ts: expectations updated to the scaled budgets, fixtures sized so the raised cap is genuinely exercised.tests/preflight-max-output-tokens.test.tsandtests/preflight-compress.test.tspass unchanged.Pre-flight (local, targeted per repo testing discipline — full suite runs in CI): typecheck ✅ · build ✅ · targeted runs: all
tests/preflight-*.test.ts+ new file + metering family (cache-ledger, nudge, usage frame/abort/totals/observability) = 225/225 pass.Review round 2 (2026-10-03)
Owner review findings addressed directly on this branch:
armOverflowShrinkwrotelastInputTokens+source="overflow-arm"but neverlastInputTokensOrigin. After a mid-session provider switch, the F2 gate compared the arm against the previous route's origin, demoted this route's own rejection evidence to untrusted, and the 使用pi 压缩后就提示prompt is too long #1195 in-request refold rescue never fired — the client got the raw overflow 400 on exactly the scenario the arm protects. Both arm branches now stampnormalizeUpstreamOrigin(route)alongside the baseline write (unknown route degrades to the legacy unprovenanced behavior). New regression e2e (F2b in the calibration suite): usage settles on A → switch to B → B rejects an unfolded payload with a stated window → armed baseline stays authoritative on B → same-request refold summarizes, re-sends, client sees 200; asserts the baseline origin is B and post-rescue traffic keeps flowing. Verified red without the fix, green with it.settleUsageReport'slocalTextEstimateargument had no caller passing it (prepare* records the pair directly) — dropped; the roll-forward became an explicit clear of the pending pair, which is the actual invariant (one pair per turn, consumed or cleared by exactly one report).countSystemAndToolsTokens(system, tools)while the gate applies the factor toestimateWireOverhead(protocol, body)— a constant additive offset that shrinks on large payloads and is floored byCALIBRATION_MIN_ESTIMATE = 2000on small ones; accepted as-is per review.a863e935, includes fix(web): protectedTools row must address compress.protectedTools (#1948) #1949 — web-only, zero overlap); targeted suites re-run green (calibration 11/11 incl. new F2b, preflight round-budget + multi-range, overflow/refold family 30/30, cache-ledger + zero-usage 45/45), typecheck clean. Full regression runs in CI.Review round 3 (2026-10-03)
Owner-driven tightening (
0ea4a174), verified and rebased onto current master (f2b18022, includes #1956 — itssrc/server.tshunks sit at lines ~108–2336, zero overlap with this PR's regions; clean rebase, no conflicts):CALIBRATION_CLAMP_MAX4 → 1. The published factor can now only deflate the estimate (trigger no earlier than the raw proxy would); routes whose billing runs above the local estimate publish k̂=1 = exactly legacy behavior, still covered by the overflow arm / learn-on-failure ladder. Rationale: risk asymmetry — every observed DSH: 切换模型后,preflight 在窗口远未满时提前触发压缩 #1933 harm was over-triggering, while late triggering already has backstops. This eliminates the failure class "calibration itself causes an earlier trigger" outright. Learning side (plausibility band [0.2,5], ring ≤3, ≥2 agreement within ×2) unchanged.applyEstimateCalibrationdegradesnull / 0 / NaN / ±Infinity / negativeto no-correction at the single shared choke point — verified all five k̂ consumers go through it (gate triggersrc/server.ts:5365, final fitsrc/server.ts:5583, fit sitessrc/preflight.ts:855/:955). A corrupted persistedsession.stats.calibratedEstimatecan no longer blind (raw×0) or poison (NaN) the estimate arm. Healthy pipelines cannot produce such values (clamp + admission guarantee [0.25,1]), so this is pure hardening against data corruption.Fixes #1933