Skip to content

[Bug] Multiple authentication headers make throttle identity order-dependent and incomplete #6148

Closed

Description

@Afloat16

Reproduction

At e1e33a9, compare credentialFingerprint for {authorization:"Bearer gateway-key","x-api-key":"account-a"} with identical fields inserted in reverse order: digests differ. Keeping Authorization first but changing x-api-key to account-b produces the same digest. Both headers are forwarded by buildForwardTarget; pre-ingest independently collects them in a fixed order.

In the real throttle governor, reordering misses the original 429 hold. Another API-key account sharing the gateway credential inherits that hold and can clear it on success.

Expected behavior and verified patch

Outbound credential-set identity must be insertion-order independent and include every distinct credential. Preserve existing single-key aliases and SDK-HMAC normalization; hash sorted normalized header/value pairs only for distinct multi-key sets. Four regressions fail against unchanged source. All 51 targeted tests pass, including throttle admission/clear behavior. Typecheck and build pass. Examples above use synthetic credentials.

Activity

  1. ranxianglei commented on Nov 10, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    Dedupe screen: not a duplicate — no open or closed issue/PR touches the multi-header credential-identity defect. Closest neighbors, all different mechanisms: #2131/#2150 introduced the single-key credential fingerprint this issue refines; #5199/#5696 added the model dimension to the hold key; #5795/#5801 made the pre-ingest fast path read the same header set (fixed order). No open [PR] row targets this thread, so I will proceed with one fix PR here.

    Labels applied: bug · p2 · scope/wiring — multi-credential lanes get wrong 429-hold identity (miss on reorder, cross-account inheritance); it lives in the egress request pipeline (src/cache-ledger.ts + src/upstream-throttle.ts), not kernel/storage/UI. p2 rather than p1: single-credential lanes are unaffected; only lanes forwarding ≥2 distinct credentials misbehave.

    Verification — reproduced on HEAD (fingerprint code byte-identical to cited commit e1e33a9):

    • {authorization:"Bearer gateway-key", x-api-key:"account-a"} → sha256:8cb1c6bc8952; same fields inserted in reverse → sha256:fc164f825080 — insertion-order dependent ✓
    • authorization kept first, x-api-key changed to account-b → sha256:8cb1c6bc8952 — second credential ignored ✓

    Root cause: credentialFingerprint (src/cache-ledger.ts:654) iterates Object.entries(headers) and breaks on the FIRST recognized non-empty credential header — so with two distinct credentials the identity is both order-dependent and incomplete. The governor hold key (src/upstream-throttle.ts:379) inherits it; the pre-ingest fast path (src/server/handle.ts:101) collects all four names in fixed name-order, which is exactly how the two paths can disagree about the same outbound request.

    Fix plan (matches your verified-patch description): collect ALL recognized credential headers, normalize each (trim / strip Bearer envelope / SDK-HMAC line → Access key id), dedupe by normalized value, hash the SORTED set. A single-distinct-value set hashes byte-identically to today — bearer/x-api-key aliasing and SDK-HMAC folding preserved.

    ⚠️ Behavior change (flagged up front): for requests carrying ≥2 DISTINCT normalized credentials, the throttle hold key and cache-ledger kp change from "first header in insertion order" to "sorted full set". Single-credential requests are byte-identical before/after. Multi-credential lanes had no stable identity before (it flipped with header order), so nothing consistent is invalidated — this is the point of the fix.

    Implementing now: fix + regressions (order independence, completeness, alias preservation, governor hold match/clear across reorder and across accounts sharing a gateway credential) → PR closing #6148.

  2. ranxianglei commented on Nov 10, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    Done — PR #6154: #6154 (labels applied by me on the PR's own diff: bug · p2 · scope/wiring). Body carries Closes #6148.

    What shipped: credentialFingerprint now collects EVERY recognized auth header, normalizes each (trim / Bearer-envelope strip / SDK-HMAC line → Access key id), dedupes by normalized value, and hashes the SORTED set (\u0000-joined against member-boundary collisions). Your two repro cases now behave as expected: both insertion orders of {authorization:"Bearer gateway-key", x-api-key:"account-a"} yield ONE identical digest, and switching x-api-key to another account flips it.

    Verification: regression rows were proven to bite — 6 of the 10 rows in the new tests/issue6148-multi-cred-fingerprint.test.ts FAIL against unchanged source (the rest are guards for preserved behavior). Post-fix, local targeted run: 148 tests across the cache-ledger / fingerprint / governor / fast-path suites pass; typecheck + build clean on a branch cut from current master. Full suite rides CI on the pushed head. Governor-level rows additionally pin: hold matches across header reorder, and an account sharing the gateway credential neither inherits nor clears another account's hold.

    Behavior change (for the record, flagged in the PR too): requests carrying ≥2 DISTINCT normalized credentials get a new hold key / cache-ledger kp ("sorted full set" instead of "first header in insertion order"). Single-credential requests are byte-identical before/after. Multi-credential lanes had no stable identity before (it flipped with insertion order), so nothing consistent is invalidated — the change IS the fix.


    中文结构化八问(审核结论)

    1. 是什么问题:出站凭证指纹只取「插入顺序里的第一个认证头」,多凭证车道(网关 key + 账号 key)的身份既随头顺序翻转、又不包含第二个凭证。
    2. 什么原因导致的:credentialFingerprint(src/cache-ledger.ts)遍历 headers 遇到第一个识别到的认证头即 break——隐含假设「每请求至多一个凭证」;Provider 429 (concurrency-style limit): turn re-driven every few seconds for minutes, keeps the account pinned at the limit #4204 的 429 hold key 直接继承该身份,[bug][dsh] 429-hold + host retry ladder re-upload 51.2 MiB per retry (36/37 attempts failed) | 429-hold 与宿主重试叠加:每次重试重传 51.2 MiB,单步放大 2-5 倍 #5795 预摄入快路径按固定名序收全部四个头,两条路径因此可能对同一出站请求算出不同的 hold key。
    3. 影响范围与严重程度:仅转发 ≥2 个不同凭证的车道受影响——头重排后 429 hold 失配(限流保护失效、重试放大可能延续)、同网关下另一账号继承并可在他账号成功时清除其 hold。单凭证车道零影响。定级 p2:能用,但该类车道存在明确的保护性错误。
    4. 怎么修复的:收集全部识别头 → 逐个归一化(去 Bearer 信封 / SDK-HMAC 折到 Access key id)→ 按归一化值去重 → 对排序后的集合做 sha256。排序保证插入顺序无关,全量收集保证完整;单值集合的哈希与修复前逐字节一致,大量缓存接缝事件导致缓存命中率大幅度降低 #2131 的 bearer/x-api-key 别名与 bili 改写带 body 签名的模型请求,CodeArts(华为 APIG SDK-HMAC-SHA256)全部 401 #1884 的 SDK-HMAC 折叠原样保留。
    5. 是否有回归问题:逐项审过既有行为——单凭证请求指纹逐字节不变(bearer/x-api-key 统一、SDK-HMAC 折叠、空/未识别输入返回 undefined 均有回归行钉住);Provider 429 (concurrency-style limit): turn re-driven every few seconds for minutes, keeps the account pinned at the limit #4204/429 governor hold is keyed by origin+credential with no model dimension — one model's quota exhaustion freezes the whole lane (healthy models get local 429/424) #5199 governor 全套与 [bug][dsh] 429-hold + host retry ladder re-upload 51.2 MiB per retry (36/37 attempts failed) | 429-hold 与宿主重试叠加:每次重试重传 51.2 MiB,单步放大 2-5 倍 #5795 快路径一致性测试本地通过。高危模式逐项过:无阈值/时序漂移、无输出格式变化、无 fail-fast 变吞错;唯一语义变化即本 PR 目标本身,已在上文与 PR 描述中披露(旧值→新值+原因)。
    6. 是否破坏缓存:前缀缓存不受影响(不改 wire/消息形状)。缓存账本 kp 归属:单凭证不变;多凭证车道历史行与新请求将分属不同桶——但该车道此前身份随插入顺序翻转、本就不存在稳定桶,无一致性被破坏。
    7. 是否改了设计方向:没有。仍是「归一化凭证值 → sha256 短指纹」同一机制,只是从「第一个头」改为「全体不同凭证的排序集」。
    8. 是否引入了新的配置:没有。无新 env/config/默认值。

    判定:本地定向验证完成、分支基于当前 master,等 CI 全局绿 + 人工 review 后可合入(不触碰 kernel/自更新/配置面,属纯 fix)。

  3. ranxianglei commented on Nov 12, 2026

    @ranxianglei
    Owner

    Fixed in batch PR #6439 (merged).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions