You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] Multiple authentication headers make throttle identity order-dependent and incomplete #6148
At e1e33a9, compare credentialFingerprint for {authorization:"Bearer gateway-key","x-api-key":"account-a"} with identical fields inserted in reverse order: digests differ. Keeping Authorization first but changing x-api-key to account-b produces the same digest. Both headers are forwarded by buildForwardTarget; pre-ingest independently collects them in a fixed order.
In the real throttle governor, reordering misses the original 429 hold. Another API-key account sharing the gateway credential inherits that hold and can clear it on success.
Expected behavior and verified patch
Outbound credential-set identity must be insertion-order independent and include every distinct credential. Preserve existing single-key aliases and SDK-HMAC normalization; hash sorted normalized header/value pairs only for distinct multi-key sets. Four regressions fail against unchanged source. All 51 targeted tests pass, including throttle admission/clear behavior. Typecheck and build pass. Examples above use synthetic credentials.
Dedupe screen: not a duplicate — no open or closed issue/PR touches the multi-header credential-identity defect. Closest neighbors, all different mechanisms: #2131/#2150 introduced the single-key credential fingerprint this issue refines; #5199/#5696 added the model dimension to the hold key; #5795/#5801 made the pre-ingest fast path read the same header set (fixed order). No open [PR] row targets this thread, so I will proceed with one fix PR here.
Labels applied:bug · p2 · scope/wiring — multi-credential lanes get wrong 429-hold identity (miss on reorder, cross-account inheritance); it lives in the egress request pipeline (src/cache-ledger.ts + src/upstream-throttle.ts), not kernel/storage/UI. p2 rather than p1: single-credential lanes are unaffected; only lanes forwarding ≥2 distinct credentials misbehave.
Verification — reproduced on HEAD (fingerprint code byte-identical to cited commit e1e33a9):
{authorization:"Bearer gateway-key", x-api-key:"account-a"} → sha256:8cb1c6bc8952; same fields inserted in reverse → sha256:fc164f825080 — insertion-order dependent ✓
authorization kept first, x-api-key changed to account-b → sha256:8cb1c6bc8952 — second credential ignored ✓
Root cause:credentialFingerprint (src/cache-ledger.ts:654) iterates Object.entries(headers) and breaks on the FIRST recognized non-empty credential header — so with two distinct credentials the identity is both order-dependent and incomplete. The governor hold key (src/upstream-throttle.ts:379) inherits it; the pre-ingest fast path (src/server/handle.ts:101) collects all four names in fixed name-order, which is exactly how the two paths can disagree about the same outbound request.
Fix plan (matches your verified-patch description): collect ALL recognized credential headers, normalize each (trim / strip Bearer envelope / SDK-HMAC line → Access key id), dedupe by normalized value, hash the SORTED set. A single-distinct-value set hashes byte-identically to today — bearer/x-api-key aliasing and SDK-HMAC folding preserved.
⚠️Behavior change (flagged up front): for requests carrying ≥2 DISTINCT normalized credentials, the throttle hold key and cache-ledger kp change from "first header in insertion order" to "sorted full set". Single-credential requests are byte-identical before/after. Multi-credential lanes had no stable identity before (it flipped with header order), so nothing consistent is invalidated — this is the point of the fix.
Implementing now: fix + regressions (order independence, completeness, alias preservation, governor hold match/clear across reorder and across accounts sharing a gateway credential) → PR closing #6148.
Done — PR #6154:#6154 (labels applied by me on the PR's own diff: bug · p2 · scope/wiring). Body carries Closes #6148.
What shipped:credentialFingerprint now collects EVERY recognized auth header, normalizes each (trim / Bearer-envelope strip / SDK-HMAC line → Access key id), dedupes by normalized value, and hashes the SORTED set (\u0000-joined against member-boundary collisions). Your two repro cases now behave as expected: both insertion orders of {authorization:"Bearer gateway-key", x-api-key:"account-a"} yield ONE identical digest, and switching x-api-key to another account flips it.
Verification: regression rows were proven to bite — 6 of the 10 rows in the new tests/issue6148-multi-cred-fingerprint.test.ts FAIL against unchanged source (the rest are guards for preserved behavior). Post-fix, local targeted run: 148 tests across the cache-ledger / fingerprint / governor / fast-path suites pass; typecheck + build clean on a branch cut from current master. Full suite rides CI on the pushed head. Governor-level rows additionally pin: hold matches across header reorder, and an account sharing the gateway credential neither inherits nor clears another account's hold.
Behavior change (for the record, flagged in the PR too): requests carrying ≥2 DISTINCT normalized credentials get a new hold key / cache-ledger kp ("sorted full set" instead of "first header in insertion order"). Single-credential requests are byte-identical before/after. Multi-credential lanes had no stable identity before (it flipped with insertion order), so nothing consistent is invalidated — the change IS the fix.
Activity
ranxianglei commentedon Nov 10, 2026
(ai: vllm-qwen/qwen3.8-27b)
Dedupe screen: not a duplicate — no open or closed issue/PR touches the multi-header credential-identity defect. Closest neighbors, all different mechanisms: #2131/#2150 introduced the single-key credential fingerprint this issue refines; #5199/#5696 added the model dimension to the hold key; #5795/#5801 made the pre-ingest fast path read the same header set (fixed order). No open [PR] row targets this thread, so I will proceed with one fix PR here.
Labels applied:
bug·p2·scope/wiring— multi-credential lanes get wrong 429-hold identity (miss on reorder, cross-account inheritance); it lives in the egress request pipeline (src/cache-ledger.ts+src/upstream-throttle.ts), not kernel/storage/UI. p2 rather than p1: single-credential lanes are unaffected; only lanes forwarding ≥2 distinct credentials misbehave.Verification — reproduced on HEAD (fingerprint code byte-identical to cited commit e1e33a9):
{authorization:"Bearer gateway-key", x-api-key:"account-a"}→sha256:8cb1c6bc8952; same fields inserted in reverse →sha256:fc164f825080— insertion-order dependent ✓x-api-keychanged toaccount-b→sha256:8cb1c6bc8952— second credential ignored ✓Root cause:
credentialFingerprint(src/cache-ledger.ts:654) iteratesObject.entries(headers)andbreaks on the FIRST recognized non-empty credential header — so with two distinct credentials the identity is both order-dependent and incomplete. The governor hold key (src/upstream-throttle.ts:379) inherits it; the pre-ingest fast path (src/server/handle.ts:101) collects all four names in fixed name-order, which is exactly how the two paths can disagree about the same outbound request.Fix plan (matches your verified-patch description): collect ALL recognized credential headers, normalize each (trim / strip Bearer envelope / SDK-HMAC line → Access key id), dedupe by normalized value, hash the SORTED set. A single-distinct-value set hashes byte-identically to today — bearer/x-api-key aliasing and SDK-HMAC folding preserved.
kpchange from "first header in insertion order" to "sorted full set". Single-credential requests are byte-identical before/after. Multi-credential lanes had no stable identity before (it flipped with header order), so nothing consistent is invalidated — this is the point of the fix.Implementing now: fix + regressions (order independence, completeness, alias preservation, governor hold match/clear across reorder and across accounts sharing a gateway credential) → PR closing #6148.
ranxianglei commentedon Nov 10, 2026
(ai: vllm-qwen/qwen3.8-27b)
Done — PR #6154: #6154 (labels applied by me on the PR's own diff:
bug·p2·scope/wiring). Body carriesCloses #6148.What shipped:
credentialFingerprintnow collects EVERY recognized auth header, normalizes each (trim / Bearer-envelope strip / SDK-HMAC line → Access key id), dedupes by normalized value, and hashes the SORTED set (\u0000-joined against member-boundary collisions). Your two repro cases now behave as expected: both insertion orders of{authorization:"Bearer gateway-key", x-api-key:"account-a"}yield ONE identical digest, and switchingx-api-keyto another account flips it.Verification: regression rows were proven to bite — 6 of the 10 rows in the new
tests/issue6148-multi-cred-fingerprint.test.tsFAIL against unchanged source (the rest are guards for preserved behavior). Post-fix, local targeted run: 148 tests across the cache-ledger / fingerprint / governor / fast-path suites pass; typecheck + build clean on a branch cut from current master. Full suite rides CI on the pushed head. Governor-level rows additionally pin: hold matches across header reorder, and an account sharing the gateway credential neither inherits nor clears another account's hold.Behavior change (for the record, flagged in the PR too): requests carrying ≥2 DISTINCT normalized credentials get a new hold key / cache-ledger
kp("sorted full set" instead of "first header in insertion order"). Single-credential requests are byte-identical before/after. Multi-credential lanes had no stable identity before (it flipped with insertion order), so nothing consistent is invalidated — the change IS the fix.中文结构化八问(审核结论)
credentialFingerprint(src/cache-ledger.ts)遍历 headers 遇到第一个识别到的认证头即 break——隐含假设「每请求至多一个凭证」;Provider 429 (concurrency-style limit): turn re-driven every few seconds for minutes, keeps the account pinned at the limit #4204 的 429 hold key 直接继承该身份,[bug][dsh] 429-hold + host retry ladder re-upload 51.2 MiB per retry (36/37 attempts failed) | 429-hold 与宿主重试叠加:每次重试重传 51.2 MiB,单步放大 2-5 倍 #5795 预摄入快路径按固定名序收全部四个头,两条路径因此可能对同一出站请求算出不同的 hold key。kp归属:单凭证不变;多凭证车道历史行与新请求将分属不同桶——但该车道此前身份随插入顺序翻转、本就不存在稳定桶,无一致性被破坏。判定:本地定向验证完成、分支基于当前 master,等 CI 全局绿 + 人工 review 后可合入(不触碰 kernel/自更新/配置面,属纯 fix)。
ranxianglei commentedon Nov 12, 2026
Fixed in batch PR #6439 (merged).