Repository navigation
[PR] fix: 429-hold answered locally before ingesting large request bodies (#5795) - #5801
Conversation
…5795) The #4204 egress hold refused requests only AFTER the full ingest (JSON.parse + openaiToCore + snapshot pinning + processTurn), so every host retry of a held send still paid ~5.5-7.2s of proxy CPU and a hundreds-of-MB allocation spike per attempt on a 51 MiB session. A read-only peek of the governor (upstreamThrottlePeekReject — rejects only, never mints probe tokens) now answers >=1 MiB POSTs up front in the dispatch path; sub-MiB traffic, forge/classifier local intercepts, passthrough-mark lanes, chained-bili hop markers and model-mismatched model-scoped holds all keep their exact pre-change paths, with the post-ingest gate unchanged as backstop. The single reject-response renderer (answerLocalThrottleReject) keeps both sites byte-identical.
Unit: peek/gate parity incl. terminal 424, peek read-only probe-slot guarantee, model-scoped conservatism, credential isolation, provisional groups, disabled governor, threshold boundary, exact 429/424 response shape. E2E hermetic proxy: held big turn = zero egress + local 429 with Retry-After, clean big turn egresses, small turns during a hold keep the post-ingest refusal shape, keyring model-scoped hold does not block other models' big turns.
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.8369-pr.5801.92efb5bThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5801.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) unused-exports 已修复并推到 PR 分支:commit 结构化八问
验证过程备注
当前状态
从本车道角度收工:唯一指派的失败检查已修复且全链路验证通过;合并由人执行。 |
With a first-response fixture set, n++ sat only on the 200 path, so the fixture replayed its first response forever: the model-scope row's egressing turn received another 429 and its 200 assertion could never pass (deterministic fail, reproduced locally). Counter now advances on every request; fixture behavior for the hold/clean rows is unchanged.
…askUrlsInText) These e2e requests mint fresh ANONYMOUS sessions, whose provenance path masks the ENTIRE first message through maskUrlsInText before truncating to the preview cap. The #4191 any-scheme regex ([a-z][a-z0-9+.-]*://) backtracks O(n^2) over long scheme-char runs (measured: 7.4s @ 100k chars, exact 4x per doubling), so a megabyte of 'x' froze the event loop for minutes and hung this suite in CI. Whitespace-separated words keep the byte size identical while keeping the scan linear. The underlying masker defect is pre-existing and tracked separately.
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。按失败签名(429-hold + pre-ingest fastpath)检索 open+closed 的 issue/PR,仅本 PR 针对 #5795 的 bili 侧修复;dsh 侧根因仍在 #5795 的 🔶 待决评论下等 owner 拍板(跨仓库,不在本 PR 范围)。 标签: bug、dsh、scope/wiring、p2(与源 issue #5795 对齐;不打 needs-decision——该决策项属 dsh 侧,已在 #5795 跟踪)。 结论:可合并(等推送后的新 CI 转绿)——机制本身正确且保守;但审核开始时 head 上 CI 是红的,且新增测试文件有两个确定性缺陷(其一会让任何 runner 挂死),均已修复并推送到 PR 分支。 审核过什么(方向关 / 回归关)
审核发现的三个问题(均已修复并推送)
|
ranxianglei
left a comment
There was a problem hiding this comment.
评审结论:建议可合并(需人审) — 独立复核通过,无阻塞项。以下为本地复核证据与两条非阻塞备注。
逐点复核(在 PR head feb8be9e1 上,基于 master c45ac21)
- peek 与 gate 的 REJECT 分支逐行一致:两个循环(holds + provisionalGroups)、跳过条件(
!state.provisional && state.model !== undefined && state.model !== requestModel、slotBlocked(state, now)、strongerReject)完全相同;唯一刻意的差异是 peek 不铸造 probe token、不记 hasApplicable —— 即文档承诺的"只读对照"。新增套件里有专门的非变异钉(peek 后 probe slot 仍在、streak 不漂移)。 - 线上形状逐字节不变:post-ingest 原内联渲染被替换为共享
answerLocalThrottleReject,字段、状态码、retry-after(仅 429 带)一字未动;日志标签沿用 session.id / "passthrough",新路径加 "pre-ingest"。 - 保守边界都对:GET/HEAD、protocol=null、<1MiB、模型不匹配的 scoped hold(#4776 keyring 行覆盖)、governor 关闭时 inert、凭据覆写车道(#2336,raw header 指纹不可见→落旧路径)——逐一核对代码位置成立。
CREDENTIAL_HEADER_NAMES提为共享常量是对的:hold key 与 peek 读同一组头名,不会再漂。 requestModelFromBody不构成隐性 ingest:它是顶层结构性扫描(skipJsonValue 不物化值),O(n) 无分配,相对省下的 parse+convert+pin(~5.5–7.2s)可忽略;"零 ingest"合同成立(请求体本就在内存缓冲里)。- 新测试文件对 master 自动失效(import 的
tryAnswerEarlyHold/upstreamThrottlePeekReject在 master 不存在),回归钉天然有效。
本地验证
tests/issue5795-early-hold-fastpath.test.ts+tests/issue4204-throttle-governor.test.ts+tests/issue4776-keyring-model-hold.test.ts:42/42 绿;另跑tests/issue4725-loop-429-observe.test.ts+tests/issue4743-degenerate-refetch-429.test.ts:10/10 绿。npm run typecheck(tsc --noEmit)干净。- CI(plan/gate/build-artifact/static-gates 矩阵)该 head 全 SUCCESS。
两条非阻塞备注(都不需改动,记录在案)
① forge 车道的角落行为变化:fast path 位于 prepare 之前(handle.ts:512),而 codexForge/autoModeForge 判定在 prepare 内(~2100 行起)。若某次 forge/classifier 请求体真 ≥1 MiB 且当时 origin+credential 正被 hold,宿主将收到本地 429/424 而非锻造响应。PR 已论证 forge 形状为 KB 级,且方向安全(零出口、机器可读的退避信号、窗口过后重试即可拿到锻造结果);要在 forge 之前排除它们就必须先解析体,等于自废"免 ingest"目的——维持现状是正确的取舍,留档即可。
② 阈值硬编码 1 MiB(EARLY_HOLD_GATE_MIN_BYTES):当前合理(<1MiB 走旧路径保 shape),若将来想按会话大小调参再提 knob,现在不加是对的(无配置面)。
宿主侧(dsh)的重建+重传大头仍按 issue 线程里的 🔶 待决事项处理,与本 PR 正交。
ranxianglei
left a comment
There was a problem hiding this comment.
补充更正(对上述备注①):我重新核对了代码,之前"forge 形状为 KB 级"的表述不准确——那是转述 PR 描述,与代码事实不符,三条本地应答车道的请求体都随会话历史增长:
- dsh compaction 应答/拒绝(handle.ts:2113
isDshCompactionCall,openai/anthropic/responses 三协议):注释明说"the compaction call is a full-budget request",同文件注释记录了 7.5MB 量级的单消息包(#2490);且 #2490/#3013 记录了该调用在压力下会高频重试(现场出现过 5051 次拒绝的风暴)。→ 该组合不罕见。 - auto-mode classifier 锻造(handle.ts:2205):
isAutoModeClassifier要求全转录 span(maxTranscriptSpan >= MIN_TRANSCRIPT_SPAN)——它本来就是 "O(n²) full-transcript-resend lane"(原注释),大会话下轻松超 1 MiB。 - codex forge(prepare-responses.ts 构建,本地应答 trigger/endpoint/prompt 三种)。
即:在这些车道被 hold 命中的窗口内,宿主会先收到本地 429/424(Retry-After)而非原本的本地锻造/拒绝应答。复核方向性后维持"不改"的判断,理由:(a) 三条车道本来就零出口、零计费,变成带退避语义的 429 不引入新的出口/数据风险;(b) hold 窗口是瞬态的,过期后即恢复原本行为;(c) 要在 fast path 之前识别这些形状就必须解析请求体——精确等于本 PR 要省掉的 O(history) ingest 成本,改动等于自废目标;(d) 若 owner 认为 dsh compaction 风暴在过载窗口内不应被 hold,正确的修法是把这三个本地应答块上提到 tryAnswerEarlyHold 之前(它们需要 parsed,代价已被 prepare 摊销),那是一次有意的区域重排,值得单独讨论而不是顺手改。请人审时知悉此窗口的存在。
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.1065-master.21c950e # this exact build
npm install -g billion-context@master # always the newest merged build
|
…ge gap) Picked up from PR #4428 for the CI/test-signal small-fixes batch. Resolved against current master (post-#5801/#4828): guards keep their new timeout-minutes; kernel-guard keeps the #4828 lockfile-exemption and now also skips only the PR-body declaration check on owner-fired workflow_dispatch.
…ge gap) Picked up from PR #4428 for the CI/test-signal small-fixes batch. Resolved against current master (post-#5801/#4828): guards keep their new timeout-minutes; kernel-guard keeps the #4828 lockfile-exemption and now also skips only the PR-body declaration check on owner-fired workflow_dispatch.
Closes #5795
What changed
The #4204 egress hold refuses held sends locally (429 backoff window / 424 terminal) without egress — but the refusal was produced only AFTER full ingest, so every host retry of a held send still paid the proxy's entire ingest cost (JSON.parse + openaiToCore + snapshot pinning + processTurn). On the reporter's 51.2 MiB session that is ~5.5–7.2 s of proxy CPU plus a hundreds-of-MB allocation spike PER ATTEMPT, multiplied by the host's retry ladder.
This adds a pre-ingest fast path: a read-only peek at the governor (
upstreamThrottlePeekReject) answers ≥1 MiB non-GET/HEAD requests up front in the dispatch path with the exact same local refusal; everything else keeps its existing path, and the post-ingest gate stays unchanged as backstop.Behavior change (disclosed per policy)
Retry-Afteron 429 only) and JSON error body (code: bili_throttle_hold|bili_throttle_terminal,retryable,stage, optional scope/model). Both sites now render through one shared helper (answerLocalThrottleReject), so the post-ingest path's bytes are byte-identical to before.Why a separate peek function instead of calling the gate twice
upstreamThrottleGateMINTS a probe token when it admits an expired-window hold. Calling it pre-prepare would consume the probe slot and make the post-prepare call REJECT the very probe request it is supposed to let through. The peek duplicates the reject loop read-only; the gate loop and its mutation stay atomic and single-call-site for admission.Tests
New
tests/issue5795-early-hold-fastpath.test.ts(16 rows):retryAfterSec: null), peek non-mutation (probe slot survives a peek; mid-window both reject without minting), model-scoped conservatism (exact match rejects / other model passes / absent model passes), credential isolation, provisional groups, disabled governor inert, threshold boundary, exact 429/424 response shape +[pre-ingest]log label;Adjacent suites green locally:
tests/issue4204-throttle-governor.test.ts+tests/issue4776-keyring-model-hold.test.ts(29/29). Full-suite verdict = CI on this head.Config surface
None. No new knobs/env/config fields — reuses the existing #4204 governor knobs (
throttle429*).Out of scope (tracked in-thread)
The dominant cost — host-side rebuild + re-upload of the full 51 MiB per retry, dsh ignoring
Retry-Afterand classifying the local 424 as retryableSERVER— lives in the dsh repo. That part is a pending owner decision in the issue thread (🔶 需要决策 comment): recommended package = honor Retry-After, reuse the last serialized body on retry, treat local-hold 424/bili_throttle_terminalas non-retryable.