Skip to content

[PR] Learn upstream per-prompt image-count caps and converge outbound images — no more 'Too many images' 400 loops - #4496

Closed
ranxianglei wants to merge 1 commit into
masterfrom
2026-10-24_image-count-cap-learning
Closed

ranxianglei wants to merge 1 commit into
masterfrom
2026-10-24_image-count-cap-learning

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Part of #4124 (implements option B, the learning side; option A — the config surface — stays open there pending owner sign-off on the shape).

What

Upstreams with a hard per-prompt image-count wall (e.g. vLLM --limit-mm-per-prompt) reject the whole request (Too many images in request: 165 > 30). bili had no count dimension anywhere: the reject vocabulary only knew token/context shapes, so the 400 was not even classified as a capacity rejection and was forwarded verbatim every round until the session dead-ended. Field evidence (#4124): 8 occurrences across 3 sessions/providers on one upstream model service; the cascade reached a second wall (gateway byte limit) before manual compressions freed the session.

This PR makes bili learn the cap from the rejection itself and converge:

  1. Detection — inspectContextOverflow gains an independent image-count channel (imageCountCap/imageCountSeen) for the two known rejection shapes (vLLM's exact format, including the > escaped as \u003e wire form seen in field logs, plus an At-most form). isOverflow keeps its token-only meaning: a count wall can never mint a token-space arm (parseOverflowWindow/toTokenNumber reject <1000 anyway, and the no-window arm branch would shrink at a payload estimate).
  2. Convergence — new shedWireImagesToCount: count-space twin of shedWireImages (rung A only — downscaling cannot reduce count; the protected tail YIELDS to the hard cap; no MAX_SHED_IMAGES slice since archiving is a cheap string swap). Oldest-first archive behind stable placeholder refs — originals stay in session storage, recoverable via decompress({imageRef}). Byte-identical passthrough when under cap.
  3. Same-request rescue — on the counted 400, converge the outbound copy to cap−1 and re-send once within the same request (the existing 使用pi 压缩后就提示prompt is too long #1195 seam); the client sees success, never the 400.
  4. Per-origin persistence + proactive pre-send — the learned cap is stored in session.stats.imageCountCaps keyed by normalized origin (survives restarts). Later requests to the same route converge BEFORE send when their count exceeds cap−1 — no rejected round-trip.
  5. pck probe skip — the prompt_cache_key A/B probe no longer white-runs a key-less ablation against a content-wall 400 (it fails identically); the image rescue owns the single retry and the key survives.

Behavior changes (old → new)

old new why
counted 400 forwarded verbatim every round; session dead-ended until manual compression same-request shed+retry; client sees success (+1 upstream round-trip on first hit) availability over performance; the rejection names the exact fix
learned cap none persisted per-origin, no TTL staleness is harmless: a stale cap only over-sheds (recoverable), never under-sheds
later requests to a capped route sent as-is, re-rejected once history regrows proactively converged pre-send when count > cap−1 kills the loop at the source; model sees fewer OLDER images than before — recoverable via decompress({imageRef}), placeholders stable
pck A/B probe on content-wall 400 ran a key-less ablation that failed identically skipped by inspection one fewer wasted round-trip; key-withholding outcome unchanged

Requests under the cap are byte-identical to before (asserted both directions, gate discipline #3617). No change to text-side metering or fold triggering.

Config surface

None added. Option B is config-free by design (learning from the rejection is the whole point — the reporter cannot pre-declare a shared gateway's cap). Option A (flat imageCountCap route/global/env mirroring the imageTokenCap precedent, default off) remains a checklist item in #4124 awaiting owner shape sign-off; it can join this PR pre-merge if approved now.

kernel-change

none (no files under kernel/ touched)

Tests

New tests/issue4124-image-count-cap.test.ts (17 tests): detection matrix incl. escaped-wire form, status gate, negatives (token overflow stays token-only, Bedrock throttle dark on both channels, degenerate caps); shedding units per protocol (openai/anthropic/responses/google) with determinism, under-cap byte-identity, tail-yields-to-cap, maxImages=0, exempt outputs untouched, unparseable noop; relay integration (reactive rescue, proactive pre-send, pck skip) against a scripted mock upstream. Targeted neighbors green locally: context-overflow, cjk-context-overflow, side-request-isolation, issue2484, openai-pck-identity, upstream-error-log, issue3701-ws-byte-budget, issue1195/issue3787 overflow-refold, fix-arm-headroom-clamp (136 tests total). Full suite = CI.

…d images — #4124

Upstreams with a hard per-prompt image-count wall (e.g. vLLM
--limit-mm-per-prompt) reject the whole request ("Too many images in
request: 165 > 30"); bili forwarded that 400 verbatim every round and
the session dead-ended. The reject vocabulary only knew token/context
shapes, so the error was not even classified as a capacity rejection.

- inspectContextOverflow detects image-count rejections on an
  independent channel (imageCountCap/imageCountSeen); isOverflow keeps
  its token-only meaning so no token-space arm can be minted from a
  count wall (parseOverflowWindow/toTokenNumber would reject <1000).
- shedWireImagesToCount: count-space twin of shedWireImages (rung A
  only — downscaling cannot reduce count; tail yields to the hard cap;
  no MAX_SHED_IMAGES slice) converging the outbound copy to cap-1.
  Originals stay in session storage behind stable placeholder refs
  (decompress({imageRef}) recovers them).
- Relay: same-request rescue (converge + re-send once; the client sees
  success), the learned cap persists per-origin in
  stats.imageCountCaps, later requests to the same route converge
  pre-send (no rejected round-trip), and the pck A/B probe skips
  content-wall 400s instead of white-running a key-less ablation that
  fails identically.
@ranxianglei
ranxianglei force-pushed the 2026-10-24_image-count-cap-learning branch from 7e8594e to 632b28a Compare October 24, 2026 07:55
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-24_image-count-cap-learning (632b28a)

Latest build — install this exact version

npm install -g billion-context@0.0.5410-pr.4496.b99b047

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
632b28a billion-context@0.0.5410-pr.4496.b99b047 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4496.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

冲突已解决并推送:7e8594ecf → 632b28a15(rebase 到 origin/master 02f544da0,force-push 到 PR 头分支)。无设计决策需求,纯机械冲突。

八问:

  1. 是什么问题:本 PR 分支与 master 产生 merge_conflict——master 在 PR 基线之后前进了(seq-ledger [PR] storage: seq ledger foundations — block interval bounds + ledgerSeq ref stamping (#4316 layer 3, phase 1) #4346 等,288 个文件),其中 src/server/relay.ts 的 import 块被双方各改了一行。
  2. 什么原因导致的:master 侧给 external-summary-surface.js import 加了 conversationPromptSections,PR 侧给 egress-shed.js import 加了 shedWireImagesToCount,同一文本区域双方各自插入 → 纯文本冲突,无语义分歧。
  3. 影响范围与严重程度:仅本 PR 分支的合并状态;rebase 本身不改变任何用户可见行为。PR 内容(图片数量上限学习)未动。
  4. 怎么修复的:rebase 到最新 master;唯一冲突按「两侧并集」解(两个符号在合并后文件中均有使用:relay.ts:569/:1274 调 shedWireImagesToCount,relay.ts:1984 用 conversationPromptSections)。逐 hunk 核对了全部 5 个文件对 master 的最终 diff,与原始 PR diff 完全一致(5 files, +638/−9)。
  5. 是否有回归问题:无。逐段审了 auto-merged 区域(session.ts 的 imageCountCaps stats 字段、util.ts 的 image-count 检测通道、relay.ts 四处接入点)在新 master 代码上语义完整;本地 typecheck + build 干净;定向测试 86/86 绿(PR 自身 17 个 + context-overflow / cjk-context-overflow / issue1195-refold / issue3787-refold / image-request-budget)。全量回归以远程 CI 为准。
  6. 是否破坏缓存:不涉及——rebase 未触碰任何缓存相关逻辑;PR 自身的缓存稳定性质(确定性 shed 走查、低于上限时字节级直通)保持原样。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及(rebase 未新增;PR 本身也不引入配置面——上限是运行时学习、按会话持久化的)。

判定:冲突已清,CI 将在新 head 上跑全量;绿即可合。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 —— 这是 #4124 的 option B(学习侧);option A(配置面)仍挂在 #4124 等 owner 定形状。相关 image 分支(#4150 responses-image-drop、learned-image-cost-ceiling、outbound-image-reserve)机制不同(丢弃/成本上限/预留),不是同一个 count-cap 学习机制。

标签: bug · scope/wiring · p2(会话死锁但有手动压缩绕行,非「无法恢复」类,故 p2 非 p1)。

核实而非重做:已拉分支 2026-10-24_image-count-cap-learning @ 632b28a15,已 rebase 到当前 master(可合并、无冲突)。以下是独立代码审查(未照抄 issue 结论)。

八问:

  1. 是什么问题:上游有硬性的「每 prompt 图片数」墙(vLLM --limit-mm-per-prompt),整请求拒("Too many images in request: 165 > 30");bili 没有任何 count 维度,把 400 原样每轮转发,直到会话死锁、级联撞第二道墙(gateway byte limit)才靠手动压缩脱困。

  2. 什么原因导致的:bili 的拒绝词表只认 token/context 形状;图片数量从来不是一个计量维度——图片折算进 token,但没有任何 fold/shed 决策看 count,所以这个 400 连「容量拒绝」都没被分类出来,直接被当普通错误转发。

  3. 影响范围与严重程度:任何向「带 count 墙的上游」发超过上限图片数的会话。触发面窄(需要 count 墙 + 大量图片),但命中即整会话不可用、需手动压缩绕行。现场证据:No per-prompt image count cap: upstream "Too many images in request: 165 > 30" is forwarded as-is | 缺少按 prompt 的图片张数上限:上游 "Too many images in request: 165 > 30" 被原样转发 #4124 里 8 次 / 3 会话 / 1 个上游模型服务。

  4. 怎么修复的:从拒绝本身学出上限——inspectContextOverflow 加独立 imageCountCap/imageCountSeen 通道(保留 isOverflow 的 token-only 语义,count 墙不会铸出 token-space arm);用 count 空间的 shedWireImagesToCount(rung A only,最旧优先归档到稳定占位 ref,原件留会话存储、decompress({imageRef}) 可恢复)把出站副本收敛到 cap−1;同一请求内救一次重发(使用pi 压缩后就提示prompt is too long #1195 缝,客户端看到成功、见不到 400);按 origin 持久化进 stats.imageCountCaps;后续同路由请求 pre-send 主动收敛;pck A/B probe 对 content-wall 400 跳过(不再白跑一次必然同样失败的 key-less ablation)。

  5. 是否有回归问题:审了触碰路径的既有行为(relay 请求管道 / overflow 判定 / egress shed / session state)。低于上限的请求字节级不变(确定性遍历 → 前缀缓存稳定),两个方向都断言(Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617 门禁纪律);isOverflow 保持 token-only,文本侧计量与 fold 触发未动;pck 的 key-withholding 结果不变;Buffer body / 无 protocol / 无可归档图片时落回原样透传(且仍学 cap)。未发现回归。

  6. 是否破坏缓存:否。低于上限直通字节不变;确定性最旧优先归档保持占位稳定,前缀缓存不受影响;原件留在会话存储。只在超上限时触发 shed。

  7. 是否改了设计方向:在既有 token 维度旁加了一个正交的 count 维度,无架构转向;复用既有 shed/archive/占位机制与 使用pi 压缩后就提示prompt is too long #1195 的同请求缝。

  8. 是否引入了新的配置:无。设计上免配置(从拒绝学习正是要点——报障人无法预声明共享 gateway 的上限)。session.stats.imageCountCaps 是内部持久化状态,非用户可见旋钮。option A(flat imageCountCap route/global/env)仍在 No per-prompt image count cap: upstream "Too many images in request: 165 > 30" is forwarded as-is | 缺少按 prompt 的图片张数上限:上游 "Too many images in request: 165 > 30" 被原样转发 #4124 挂着等 owner 拍板。

代码一致性核查:normalizeUpstreamOrigin(util.ts:88)、refForItemBytes(image-rescale.js)、v1Wrap(relay.ts:305)、readStreamToBuffer(stream-io.js)、markDirty(session.js)全部 import/定义到位;无 kernel 变更(diff 仅 src/{egress-shed,server/relay,session,util}.ts + tests);测试 17 个断言有牙(转义 wire 形 \u003e、status 门禁 200/418/413/502、双向字节一致、tail-yields-to-cap、maxImages=0、exempt output 不动、unparseable noop、relay 集成断言「一次拒发+一次收敛重发、客户端收 200」)。

CI:static-gates ✅、mysql-contract ✅,test 分片在跑(目前全绿),e2e 按 gate 跳过。判定:代码审查通过,可合并;等全套 CI 绿后我确认收尾。

@ranxianglei ranxianglei added bug Something isn't working p2 scope/wiring labels Oct 24, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 跟进(接上一条):全套 test 分片目前红,但红点不在本 PR 的功能上。

诊断(拉了失败分片 ubuntu-latest/node24/shard1 的日志——这个分片恰好包含本 PR 的 tests/issue4124-image-count-cap.test.ts):

结论:没有证据表明这是本 PR 引入的回归;红点与「CI 高并发负载下的时序 flake」或「既有的 #3250 flake」一致。

一个状态提醒:07:55 那次跑全套的分片红了之后,08:09 又触发了几轮——其中一轮把真正的 test matrix / static-gates skip 了(只跑了 guard job),另两轮被 cancel。所以当前这个 head(632b28a15)还没有一次「干净跑完全套且绿」的记录。分支近期有 rebase/re-trigger 痕迹(我处理期间 head 从 7e8594ecf 变到 632b28a15),像是有人在并行推进。

建议:对失败的 test 分片做一次干净的 re-run 确认转绿即可,功能代码无需改动。需要的话我可以触发 re-run-failed-jobs;考虑到分支疑似有并行操作,先跟你确认再动,免得互相踩。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

收工小结(详见上面两条:八问代码审查 + CI 诊断)。

  • 代码审查:通过。 已拉分支 2026-10-24_image-count-cap-learning @ 632b28a15(已干净 rebase 到 master),通读全部 5 个改动文件、核对所有 import、验证 17 个测试断言有牙。设计成立:count 维度的 shed(与既有 byte-budget shed 并列)、独立的 imageCountCap 通道(不动 token-only 的 isOverflow)、learned cap 的 pre-send 主动收敛、使用pi 压缩后就提示prompt is too long #1195 缝的同请求救援、pck probe 对 content-wall 400 跳过。无 kernel 改动、未新增配置面。
  • CI:红,但非本 PR 引入。 全套 test 分片红在 6 个 shard;含本 PR 功能测试的那个 shard 里,唯一失败是 tests/issue3250-wal-checkpoint.test.ts:291([bug] index.db-wal 在长驻进程内永不 checkpoint —— STORAGE_PRAGMAS 无 journal_size_limit、master 无 wal_checkpoint 调用,WAL 单调增长到 55 MiB 不回收 #3250 SQLite WAL autocheckpoint 截断差 ~4%)——时序敏感 flake,与本 PR 无关。本 PR 的 80+ image/count 断言全过。
  • 标签:bug · scope/wiring · p2。
  • 待办:对失败 test 分片做一次干净 re-run 确认转绿即可,功能代码无需改。分支近期有并行 rebase/re-trigger 痕迹,我暂未自行触发 re-run,等你确认再动。

判定:代码可合并,等一次干净的 CI 全绿收尾。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

已并入合并列车 #4551(master 合并,merge commit 保留原作者署名)。本 PR 关单收尾;回滚点为 #4551 单提交 revert。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant