Skip to content

[PR] fix(#4452): surface body-rewriting-gateway overflow starvation in relay log (observability stopgap) - #4455

Closed
ranxianglei wants to merge 1 commit into
masterfrom
wt-4452-ework-daemon
Closed

ranxianglei wants to merge 1 commit into
masterfrom
wt-4452-ework-daemon

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What & why

An intermediate gateway that rewrites every reject into one generic body (Cherry Studio et al.) strips the token counts / window from each overflow 400, so inspectContextOverflow returns isOverflow:false and bili's whole overflow self-heal silently never engages — the route starves of calibration and re-sends the same oversized payload forever (#4452).

This PR lands the decision-free observability slice of #4452 so that silent starvation becomes diagnosable now, independent of the behavioral decision still pending with the owner.

Changes

Purely additive logging. No behavior change when isOverflow is true; no new config surface.

Scope / what is NOT here

The behavioral recovery (meter-corroborated arm/refold, or route degradation after N consecutive marker-less rejects) is a preflight/metering behavior change awaiting owner sign-off on thresholds (#4452 decision request). That work will land on this same branch / PR (one PR per issue). This PR is intentionally mergeable standalone as the stopgap.

Refs #4452 (not closing — behavioral fix pending).

…#4452)

An intermediate gateway that rewrites every reject into one generic body
(Cherry Studio et al.) strips the token counts / window from each overflow
400, so inspectContextOverflow returns isOverflow:false and bili's whole
overflow self-heal silently never engages — the route starves of calibration
and re-sends the same oversized payload forever.

Decision-free observability slice of #4452 (behavioral recovery awaits owner
sign-off on thresholds):
- add isMarkerlessOverflowReject() (util.ts): a 4xx overflow-shaped reject
  whose body matches no CONTEXT_OVERFLOW_PATTERNS (excludes 5xx, which already
  self-heal via the local estimate, and any body that carries a marker).
- the relay logs a loud warn naming the starvation + the last meter reading
  (settled input vs effective window) instead of only the opaque
  `<- upstream 400` line.

Purely additive logging — no behavior change when isOverflow is true, no new
config surface.
@ranxianglei ranxianglei changed the title fix(#4452): surface body-rewriting-gateway overflow starvation in relay log (observability stopgap) [PR] fix(#4452): surface body-rewriting-gateway overflow starvation in relay log (observability stopgap) Oct 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: wt-4452-ework-daemon (6906dfa)

Latest build — install this exact version

npm install -g billion-context@0.0.5351-pr.4455.0a9e787

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
6906dfa billion-context@0.0.5351-pr.4455.0a9e787 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4455.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。本条是我自己为 #4452 落的「可观测性止血」PR(#4455,分支 wt-4452-ework-daemon),非外部提交。行为修复(meter 佐证 refold / 连续 N 次 marker-less reject 后路由降级)仍待你拍板阈值,会落在同一分支;本 PR 只交付无决策依赖的日志切片,可独立合入。

按「自己的 PR → 核对代码 + 确认 CI + 报结论」处理,未重做。核对结论(结构化八问):

  1. 是什么问题:Cherry Studio 这类网关把每个 overflow 400 重写成同一个通用 body、剥掉 token 数/窗口,使 inspectContextOverflow 判 isOverflow:false,bili 整套 overflow 自愈静默不触发,路由饿死在校准、永远重发同一份超大 payload(Upstream 400 bodies rewritten to a generic error by an intermediate gateway starve overflow calibration — repeated 400s persist on those routes (v0.1.203+) #4452)。本 PR 只是让这种「静默饿死」现在就能从日志诊断出来。
  2. 什么原因导致的:根因在网关侧剥标记(不在 bili 能修的层);bili 侧可修点是——marker-less 的 4xx overflow 形态拒绝此前只打一条不透明的 ← upstream 400,看不出「标记被剥了所以自愈没接上」。本 PR 补这个可见性缺口。
  3. 影响范围与严重程度:纯日志增强,零行为变更(isOverflow:true 路径完全不变)、零新配置面。之前这类饿死在日志里不可见,之后打一条点名 Upstream 400 bodies rewritten to a generic error by an intermediate gateway starve overflow calibration — repeated 400s persist on those routes (v0.1.203+) #4452 的 warn(带最近一次 meter 读数)。严重度 p3——不修不影响功能,只把隐性故障显性化。
  4. 怎么修复的:src/util.ts 加纯谓词 isMarkerlessOverflowReject(status, bodyText) =「400/413 且 body 不匹配任何 CONTEXT_OVERFLOW_PATTERNS」;src/server/relay.ts 在 else if(非 overflow 分支)命中就打 warn,附 last settled input vs effective window。刻意排除 5xx(arm emergency shrink on upstream failure to break relay-5xx deadlock #604 已走本地估计自愈)和带标记的 body(正常走自愈)。
  5. 是否有回归问题:无。改动全在新增分支/新增函数,info.isOverflow 命中时完全绕开新逻辑;顺带把同一 errBody 第二次 .toString("utf8")(reasoning_content 正则处)复用成已提取的 errText,语义等价。引用符号逐一核对过真实源码:s.stats.lastInputTokens、s.metadata.effectiveContextLimit(与 relay.ts:601/1102 同款守卫写法)、log("warn",…)(同 relay.ts:174 既有形状);CONTEXT_OVERFLOW_PATTERNS 全部非 global 正则,.test() 无 lastIndex 状态坑,inspectContextOverflow 与新谓词各调用一次互不干扰。
  6. 是否破坏缓存:不涉及——不碰任何缓存/游标/去重键。
  7. 是否改了设计方向:不涉及——仍是「观测先行、行为决策留给你」,与 PR 声明一致;行为恢复(refold/降级阈值)明确不在这里。
  8. 是否引入新配置:无。

一个已知边界(不拦合并,供你定夺,随行为切片一起细化):谓词是「400/413 且无标记」的宽口径,所以非 overflow 的 400(典型如 strict-reasoning-echo 首次 400,body 带 reasoning_content 但无 overflow 标记)也会触发这条 warn,属误报噪声。方向上这是刻意的——宁可多报也不漏掉被剥标记的真 overflow;若要收窄(如排除含 reasoning_content/text 的 body)属于 #4452 的决策面,留到行为切片在同一分支一起定。

测试:tests/issue4452-markerless-reject.test.ts 按 #3617 双向钉死——通用 body(400/413)→flagged;带 OpenAI / CJK 标记→not flagged;5xx / 2xx / 其他 4xx→not flagged。断言有牙(bug 回归抓得住)。

CI:pushed head 正在跑(当前 success 25 / skipped 13 / in-progress+queued 12,暂无红)。全量回归以远程 CI test lanes 为准;转绿我再回一句确认。

判定:代码核对通过、范围干净(仅 3 文件、纯增量日志)、可独立合入作止血;等 CI 转绿即可由你合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 转绿:pushed head(6906dfa)全部 check-run 收尾——37 success / 14 skipped,0 failure,含 ubuntu/windows × node22/24 的全量 test lanes 与 static-gates。

结论不变:代码核对通过、范围干净(仅 3 文件、纯增量日志、零行为变更、零新配置)、tests/issue4452-markerless-reject.test.ts 双向钉死且已在 CI 跑过。可独立合入作为 #4452 的止血;行为恢复切片待你拍板阈值后落同一分支。合并动作按规矩留给你:#4455

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

已并入合并列车 #4551(master 合并,merge commit 保留原作者署名)。本 PR 关单收尾;回滚点为 #4551 单提交 revert。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant