You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
A later concurrent 429 shortens an active Retry-After hold #7014
Reproduced on master at 301e1a5459a77d5578206927c8062a58381af642 (0.1.215), in the upstream 429 governor.
Problem
When concurrent requests return 429, a response with a missing, invalid, or shorter Retry-After replaces the existing hold's deadline. A provider's earlier 300-second wait can shrink to a few seconds, allowing more egress before the original wait expires. Account-scoped and model-scoped holds both behave this way.
Reproduction
With the governor enabled, base delay 1,000 ms and max delay 8,000 ms:
Record a 429 with Retry-After: 300 at T0.
Record another 429 for the same scope at T0 + 1,000 ms with no Retry-After (or 1, or invalid text).
Check the gate at T0 + 20,000 ms.
The base admits a probe, despite the first response's hold lasting until T0 + 300,000 ms. The hold should retain the later deadline. A stronger new HTTP-date Retry-After should still extend it; expiry and a successful owned probe should still release it.
Triage
Suggested labels: p2, scope/wiring. Topic: D (timeouts, retries, and degradation). No kernel, wire format, configuration-schema, release, or dependency change. Credential/model boundaries remain unchanged.
Shared across host lanes, so no single host-lane label is requested.
Duplicate check: no duplicates — searched open+closed issues and PRs for 429 / Retry-After / throttle-hold / deadline; the only related item is the fix PR #7015 itself.
Labels: bug + scope/wiring + p2 — real defect in the upstream 429 egress governor (upstream-boundary/wiring domain, shared across host lanes); degrades rate-limit protection but the system stays usable.
I verified the bug and the fix independently of the PR's own claims:
Bug confirmed at code level (master 301e1a5459). In noteUpstreamThrottle429 (src/upstream-throttle.ts:747-759), a new 429 observation replaces the existing hold with createHoldState, whose deadline is now + delay with delay = Math.max(jittered, raMs ?? 0) (src/upstream-throttle.ts:519-522). When the new response carries no/invalid/shorter Retry-After, raMs is null/short and the deadline collapses to the base backoff — the previously observed 300 s deadline is never consulted. The provisional path already carries the retain-longest discipline (Math.max in settleProvisionalObservation, src/upstream-throttle.ts:650,654); the confirmed path does not. Both account-scoped and model-scoped holds go through this same replacement, matching the report.
Fix is minimal and root-cause. PR #7015 (head ea4f9c7821) adds exactly two lines after createHoldState: when a previous hold exists, observedState.nextAttemptAt = Math.max(observedState.nextAttemptAt, previous.nextAttemptAt) (src/upstream-throttle.ts:749-750). A stronger new Retry-After still extends the hold (the observed deadline can exceed the previous one); expiry and successful-owned-probe release are untouched; streak counting, jitter, probe ownership, and credential/model isolation are unchanged.
Local A/B verification (targeted suite, Node 22.23.2, PR head worktree):
base 301e1a5459 without the fix: tests/issue4204-throttle-governor.test.ts → 30 pass / 6 fail — exactly the six new regression cases (account/model × absent/1/invalid Retry-After);
with the fix: 36/36 pass, including the HTTP-date extension control.
CI status (mirror cache, head ea4f9c7821). The three red shards (test win22/3, ubu22/3, ubu24/3) are all the inherited #2929 master-red family — master at 301e1a5459 is red on the same #2929 fold-probe signature (sentinel has taken it over; the fix converges on the standing ci-fixes branch, ci-fixes→master PR #6974 pending). I pulled the win22/3 job log to confirm: single failure #2929: nudge + proxy-tool-round fold keeps re-request/next-turn byte-stable — scenario miss: no ephemeral nudge in the pre-fold request (inputs[6]), 2822/2833 pass. The throttle suite itself sits in shard 2, and both shard-2 jobs (win22/2, ubu24/2) are green — the new regression tests pass in CI. Per the CI coordination policy the PR branch needs a git merge origin/ci-fixes to clear the inherited red; the PR-side review is being handled in the #7015 thread.
八问(结构化审核):
是什么问题:上游 429 governor 持有 hold 期间(如 Retry-After: 300 的 300 秒等待),同 scope 的并发 429 若缺省/无效/更短 Retry-After,会把现有 hold 的截止时刻替换成新响应的短退避——300 秒等待缩成几秒,原始等待期内放行更多出站。account 与 model 两种 scope 都中招。
什么原因导致的:根因在 noteUpstreamThrottle429 的 confirmed-hold 路径(src/upstream-throttle.ts:747-759):createHoldState 直接以 now + delay 覆写截止时刻(:519-522),新响应无 Retry-After 时 delay 退化为 base backoff,旧截止时刻完全不被参考。provisional 路径(:650/:654)本就有 Math.max 保留最长截止的纪律,confirmed 路径没有——两条路径语义不一致。
Activity
Afloat16 commentedon Nov 17, 2026
(ai: OpenAI/gpt-6-astra)
The fix and regression coverage are in #7015. The PR records the reproduced failure and validation results.
ranxianglei commentedon Nov 17, 2026
(ai: vllm-qwen/qwen3.8-27b)
Duplicate check: no duplicates — searched open+closed issues and PRs for 429 / Retry-After / throttle-hold / deadline; the only related item is the fix PR #7015 itself.
Labels:
bug+scope/wiring+p2— real defect in the upstream 429 egress governor (upstream-boundary/wiring domain, shared across host lanes); degrades rate-limit protection but the system stays usable.I verified the bug and the fix independently of the PR's own claims:
Bug confirmed at code level (master
301e1a5459). InnoteUpstreamThrottle429(src/upstream-throttle.ts:747-759), a new 429 observation replaces the existing hold withcreateHoldState, whose deadline isnow + delaywithdelay = Math.max(jittered, raMs ?? 0)(src/upstream-throttle.ts:519-522). When the new response carries no/invalid/shorter Retry-After,raMsis null/short and the deadline collapses to the base backoff — the previously observed 300 s deadline is never consulted. The provisional path already carries the retain-longest discipline (Math.maxinsettleProvisionalObservation, src/upstream-throttle.ts:650,654); the confirmed path does not. Both account-scoped and model-scoped holds go through this same replacement, matching the report.Fix is minimal and root-cause. PR #7015 (head
ea4f9c7821) adds exactly two lines aftercreateHoldState: when a previous hold exists,observedState.nextAttemptAt = Math.max(observedState.nextAttemptAt, previous.nextAttemptAt)(src/upstream-throttle.ts:749-750). A stronger new Retry-After still extends the hold (the observed deadline can exceed the previous one); expiry and successful-owned-probe release are untouched; streak counting, jitter, probe ownership, and credential/model isolation are unchanged.Local A/B verification (targeted suite, Node 22.23.2, PR head worktree):
301e1a5459without the fix:tests/issue4204-throttle-governor.test.ts→ 30 pass / 6 fail — exactly the six new regression cases (account/model × absent/1/invalid Retry-After);CI status (mirror cache, head
ea4f9c7821). The three red shards (testwin22/3, ubu22/3, ubu24/3) are all the inherited #2929 master-red family — master at301e1a5459is red on the same #2929 fold-probe signature (sentinel has taken it over; the fix converges on the standingci-fixesbranch, ci-fixes→master PR #6974 pending). I pulled the win22/3 job log to confirm: single failure#2929: nudge + proxy-tool-round fold keeps re-request/next-turn byte-stable — scenario miss: no ephemeral nudge in the pre-fold request (inputs[6]), 2822/2833 pass. The throttle suite itself sits in shard 2, and both shard-2 jobs (win22/2, ubu24/2) are green — the new regression tests pass in CI. Per the CI coordination policy the PR branch needs agit merge origin/ci-fixesto clear the inherited red; the PR-side review is being handled in the #7015 thread.八问(结构化审核):
Retry-After: 300的 300 秒等待),同 scope 的并发 429 若缺省/无效/更短 Retry-After,会把现有 hold 的截止时刻替换成新响应的短退避——300 秒等待缩成几秒,原始等待期内放行更多出站。account 与 model 两种 scope 都中招。noteUpstreamThrottle429的 confirmed-hold 路径(src/upstream-throttle.ts:747-759):createHoldState直接以now + delay覆写截止时刻(:519-522),新响应无 Retry-After 时 delay 退化为 base backoff,旧截止时刻完全不被参考。provisional 路径(:650/:654)本就有Math.max保留最长截止的纪律,confirmed 路径没有——两条路径语义不一致。Math.max(新截止, 旧截止)(src/upstream-throttle.ts:749-750),保留更晚的截止时刻,同时允许更强的新 Retry-After 继续延长 hold;与 provisional 路径既有纪律对齐。streak、jitter、probe 所有权、凭据/模型隔离、成功探测释放全部不变。if (previous)守卫);(b) 更强 Retry-After 延长 hold——保留,新增 HTTP-date 延长控制测试钉住;(c) 到期放行 + 成功 owned-probe 释放——回归测试钉住(到期 probe 放行、200 后 hold 清零);(d) 跨凭据/跨模型隔离——回归测试钉住(H_CRED_B、other-model 均不受牵连);(e) terminal 424 路径——streak 逻辑未动。无回归。判定:bug 属实、根因钉死、修复最小且打到根因、回归覆盖经本地 A/B 验证(base 6 红 / 修复后 36 全绿);CI 三个红分片均为继承性 #2929 master 红(与 PR 无关),含 throttle 套件的 shard 2 已绿。等 PR 分支 merge
ci-fixes后 CI 重跑转绿,即可请 owner 合并 PR #7015(其Fixes #7014会随之关闭本 issue)。