Skip to content

[PR] fix(server): exclude the foreign protocol's conversation field from the guard meter - #6840

Open
sx4im wants to merge 4 commits into
ranxianglei:masterfrom
sx4im:fix-6828-guard-meter-double-count-spare1
Open

sx4im wants to merge 4 commits into
ranxianglei:masterfrom
sx4im:fix-6828-guard-meter-double-count-spare1

Conversation

@sx4im

@sx4im sx4im commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

What broke

The #554 guard meter walks every string field of the outbound body, so when a body carries the conversation under two protocol shapes at once (messages plus Responses' input), it counts the history twice. The reporter's fuse reading was 1,815,699 tokens against about 380K actually billed, so the fuse blocked a body that fits.

What changed

  • estimateRawBodyBreakdown and estimateRawBodyTokens take an optional protocol and skip the other protocol's conversation fields at the top level only.
  • measuredOutbound and the preflight measured-body tiers pass prepared.protocol through.
  • With no protocol, the old fail-closed count-everything behavior stays.

How verified

  • New tests/issue6828-guard-meter-dual-shape.test.ts: 5 tests, red to green; 50/50 with the neighboring suites.
  • Typecheck clean in the changed files. Full suite is CI's call.

Fixes #6828

ranxianglei#554 guard meter (ranxianglei#6828)

A dual-shape body carries the conversation under two top-level keys: the
lane's native field (messages[] on openai/anthropic, input[] on responses)
plus another protocol's field (a Responses-native host such as dsh sends
`input` alongside `messages`). The upstream bills only the native field, so
the protocol-blind meter counted the conversation twice and the egress fuse
false-tripped on the phantom total (1,815,699 measured vs ~380K billed,
4.77x), while preflight's protocol-aware estimate saw only ~135K and never
compressed.

Thread the lane's protocol through measuredOutbound and the preflight
measured-body doom tiers so the meter skips the foreign conversation fields
for that lane, top level only. Without a protocol the meter keeps its
fail-closed count-everything behavior.

Regression test: tests/issue6828-guard-meter-dual-shape.test.ts
@ranxianglei ranxianglei changed the title fix(server): exclude the foreign protocol's conversation field from the guard meter [PR] fix(server): exclude the foreign protocol's conversation field from the guard meter Nov 15, 2026
…e-request guard meter (ranxianglei#6828)

Review supplement to the PR: the egress fuse (relay) and the preflight
measured-body tiers were made protocol-aware, but sideRequestGuard() still
metered the raw body protocol-blind even though the lane protocol is a
parameter of the function. A dual-shape body's foreign conversation field
double-counted there too, false-blocking the compress/nudge side call —
the very request that would shrink the session.

- sideRequestGuard: pass the lane protocol to estimateRawBodyTokens
- regression test: a dual-shape body must not false-block the side request
  (verified red without the fix, green with it)
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — Searched open + closed issues/PRs for "guard meter" / "egress fuse": the only open PR fixing #6828 is this one (#6840); #4849 is closed and unrelated. Sibling #6844 is the same symptom family (the measured-body egress fuse counting a model-hidden field in the other bucket) but a different field (dsh_session_log) and a different mechanism — it is not fixed by this PR (see "Not addressed" below).

Review of PR #6840 (fixes #6828)

Verdict: the fix is correct and safe. I pushed one additive commit (3d16a8a643) completing its intent at a call site the PR missed.

What the PR does right

  • Fail-closed preserved. estimateRawBodyTokens/estimateRawBodyBreakdown take an optional protocol; with no protocol the old count-everything behavior is byte-identical. No caller is silently changed.
  • Top-level only. FOREIGN_CONVERSATION_FIELDS skips the foreign conversation field at the top level only; nested same-named keys still count (test 4 pins this). Correct — the double-count is a top-level shape artifact, not a nested one.
  • Protocol passthrough at the two outbound-body call sites (relay egress fuse src/server/relay.ts:1016, preflight measured-body tiers src/server.ts ~4496) — both already had prepared.protocol in scope.
  • Tests are real. 5 tests, red-to-green, including a reporter-scale repro that reproduces the 4.77× unscoped and shows the openai-lane total fits under 1,048,576. Not tautologies.

Root-cause judgment

The dual-shape body is the trigger, but the defect is the meter: it is protocol-blind and counts every top-level conversation field regardless of which one the lane actually bills. The PR fixes the meter at the right layer (the estimate function + its outbound call sites), not by filtering payload in transit — that is the correct root-cause fix, not a mask. (The foreign field still rides the wire — bandwidth waste — but that is a separate wire-fidelity concern, out of scope here.)

Additive commit I pushed (3d16a8a643)

The PR made the egress fuse and preflight tiers protocol-aware, but sideRequestGuard() (src/server/side-request.ts:931) still metered the raw body protocol-blind even though the lane protocol is a parameter of the function. A dual-shape body's foreign field double-counted there too — false-blocking the compress/nudge side call, which is the very request that would shrink the session. I passed protocol through (1 line) and added a red-to-green regression test (verified: fails without the fix, passes with it; issue6828 suite 6/6, side-request-isolation 40/40, typecheck clean). This is the same mechanism as the PR's fix, so it belongs in the same PR, not a new one.

Not addressed (out of scope, noted for the thread)

CI

I pushed 3d16a8a643 to the PR head. At the time of writing the mirror's check cache was still showing the previous run (old SHA 1e3ebba3a4); I have not verified green on the new head yet — the full-suite verdict is CI's on 3d16a8a643.

八问(结构化审核)

  1. 是什么问题:side request(max_tokens<=200)绕过 preflight 且不检查载荷体积,携带全量历史直转上游导致 400 #554 出站 guard 计量器遍历出站 body 的每个字符串字段;当 body 同时以两种协议形状携带会话(dual-shape,如 dsh 等 Responses 原生宿主在 messages[] 旁再带 input[])时,历史被数了两遍,熔断器按虚高总量(实测 1,815,699 vs 实际计费 ~380K,4.77×)误触发,把放得下的 body 拦下(413)。
  2. 什么原因导致的:计量器是 protocol-blind 的——它不知道哪条会话字段是本车道的原生字段,于是把每个顶层会话字段都计入。dual-shape body 是触发条件,计量器的盲目遍历才是缺陷本体。
  3. 影响范围与严重程度:任何 dual-shape body、任意车道(dsh/Responses 原生宿主在 messages 旁带 input)——出站熔断对放得下的 body 误触发 413;且(在我补的 commit 之前)compress/nudge 侧请求也会被误拦——而那正是会缩小会话的请求。严重度 p2(条件性):会话仍可用,但受影响的宿主在大请求上被拦、需绕行;对 dual-shape 宿主可视为 p1。
  4. 怎么修复的:给计量器传入车道 protocol,只跳过顶层的外来会话字段。estimateRawBodyTokens/estimateRawBodyBreakdown 增加可选 protocol 参数;FOREIGN_CONVERSATION_FIELDS 把每个协议映射到其它协议的会话字段;三个出站 body 计量点(relay 熔断、preflight 实测档、side-request guard)都传入 prepared.protocol/protocol。无 protocol 时保持旧的 fail-closed 全量计数。
  5. 是否有回归问题:无。无 protocol 时行为与之前逐字节一致(fail-closed 全量计数);有 protocol 时只跳过顶层外来字段——原生字段、tools、system、所有嵌套字段仍计入,所以它只会少算虚高的重复份,绝不会少算真实负载(嵌套同名键仍计入,测试 4 钉死)。我审计了全部 5 个计量点:3 个出站 body 计量点已传 protocol;入站 GC 信号(handle.ts 的 rawInputTokens)按设计保持 protocol-blind(保守方向,多算无害);threads-lane 的合成 body 是单形状、无需 protocol。
  6. 是否破坏缓存:否。计量器是熔断/守卫用的只读估算,不触碰前缀缓存、重试或任何去重键。
  7. 是否改了设计方向:否。只是让既有的 side request(max_tokens<=200)绕过 preflight 且不检查载荷体积,携带全量历史直转上游导致 400 #554 计量器在已有 protocol 作用域的调用点变得 protocol-aware;无新数据流、无新模块。
  8. 是否引入了新的配置:否。无新 env/config/默认值。

判定:可合并(待新 head CI 绿)——修复正确、安全(fail-closed 不变)、且经我的补充 commit 后覆盖了全部三个出站计量点。合并动作仍由 owner 执行。

…de-lane suite (ranxianglei#2170 lane-guard)

The lane-guard gate (ranxianglei#2170) requires that a change to
src/server/side-request.ts land alongside one of the dedicated side-lane
suites. The previous commit put the ranxianglei#6828 dual-shape regression in
tests/issue6828-guard-meter-dual-shape.test.ts (the meter's file), which
does not satisfy the gate. Move it to tests/side-request-isolation.test.ts,
the canonical home for sideRequestGuard tests.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI 跟进(新 head d143fbd8b5):我上一条补充 commit(3d16a8a643)触发了 lane-guard 门禁(#2170)——它要求改动 src/server/side-request.ts 时必须同步更新四个专用 side-lane 套件之一,而我当时把回归测试放在了计量器的文件 tests/issue6828-guard-meter-dual-shape.test.ts 里,不满足门禁。已把该测试移到 tests/side-request-isolation.test.ts(sideRequestGuard 测试的规范归属地,commit d143fbd8b5),lane-guard 现已转绿。

当前 CI(live 核实):lane-guard ✅、全部静态门禁/guards/docs/build ✅、无失败;22 个 test 分片仍在跑(in_progress/queued)。test 分片全绿后我会再确认一次最终状态。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI 最终确认(head d143fbd8b5,live 核实):全部 45 项 check 完成 —— 29 success / 16 skipped / 0 failure,含 lane-guard ✅ 与全部 22 个 test 分片 ✅。新 head 全绿。

修复正确、安全(无 protocol 时 fail-closed 行为逐字节不变)、且经补充 commit 后覆盖了全部三个出站计量点。合并动作由 owner 执行:#6840

Resolve conflicts in src/preflight.ts: keep the PR's ranxianglei#6828 foreign-conversation-field skip and main's isNonModelTopLevelKey skip side by side in estimateRawBodyTokens and estimateRawBodyBreakdown. src/server.ts and src/server/side-request.ts merged clean.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants