Repository navigation
[PR] Learn upstream per-prompt image-count caps and converge outbound images — no more 'Too many images' 400 loops - #4496
ranxianglei wants to merge 1 commit into
Conversation
…d images — #4124 Upstreams with a hard per-prompt image-count wall (e.g. vLLM --limit-mm-per-prompt) reject the whole request ("Too many images in request: 165 > 30"); bili forwarded that 400 verbatim every round and the session dead-ended. The reject vocabulary only knew token/context shapes, so the error was not even classified as a capacity rejection. - inspectContextOverflow detects image-count rejections on an independent channel (imageCountCap/imageCountSeen); isOverflow keeps its token-only meaning so no token-space arm can be minted from a count wall (parseOverflowWindow/toTokenNumber would reject <1000). - shedWireImagesToCount: count-space twin of shedWireImages (rung A only — downscaling cannot reduce count; tail yields to the hard cap; no MAX_SHED_IMAGES slice) converging the outbound copy to cap-1. Originals stay in session storage behind stable placeholder refs (decompress({imageRef}) recovers them). - Relay: same-request rescue (converge + re-send once; the client sees success), the learned cap persists per-origin in stats.imageCountCaps, later requests to the same route converge pre-send (no rejected round-trip), and the pck A/B probe skips content-wall 400s instead of white-running a key-less ablation that fails identically.
7e8594e to
632b28a
Compare
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.5410-pr.4496.b99b047This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr4496.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 冲突已解决并推送: 八问:
判定:冲突已清,CI 将在新 head 上跑全量;绿即可合。 |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复 —— 这是 #4124 的 option B(学习侧);option A(配置面)仍挂在 #4124 等 owner 定形状。相关 image 分支(#4150 responses-image-drop、learned-image-cost-ceiling、outbound-image-reserve)机制不同(丢弃/成本上限/预留),不是同一个 count-cap 学习机制。 标签: bug · scope/wiring · p2(会话死锁但有手动压缩绕行,非「无法恢复」类,故 p2 非 p1)。 核实而非重做:已拉分支 八问:
代码一致性核查: CI:static-gates ✅、mysql-contract ✅,test 分片在跑(目前全绿),e2e 按 gate 跳过。判定:代码审查通过,可合并;等全套 CI 绿后我确认收尾。 |
(ai: vllm-qwen/qwen3.8-27b) CI 跟进(接上一条):全套 test 分片目前红,但红点不在本 PR 的功能上。 诊断(拉了失败分片 ubuntu-latest/node24/shard1 的日志——这个分片恰好包含本 PR 的
结论:没有证据表明这是本 PR 引入的回归;红点与「CI 高并发负载下的时序 flake」或「既有的 #3250 flake」一致。 一个状态提醒:07:55 那次跑全套的分片红了之后,08:09 又触发了几轮——其中一轮把真正的 test matrix / static-gates skip 了(只跑了 guard job),另两轮被 cancel。所以当前这个 head( 建议:对失败的 test 分片做一次干净的 re-run 确认转绿即可,功能代码无需改动。需要的话我可以触发 re-run-failed-jobs;考虑到分支疑似有并行操作,先跟你确认再动,免得互相踩。 |
(ai: vllm-qwen/qwen3.8-27b) 收工小结(详见上面两条:八问代码审查 + CI 诊断)。
判定:代码可合并,等一次干净的 CI 全绿收尾。 |
Part of #4124 (implements option B, the learning side; option A — the config surface — stays open there pending owner sign-off on the shape).
What
Upstreams with a hard per-prompt image-count wall (e.g. vLLM
--limit-mm-per-prompt) reject the whole request (Too many images in request: 165 > 30). bili had no count dimension anywhere: the reject vocabulary only knew token/context shapes, so the 400 was not even classified as a capacity rejection and was forwarded verbatim every round until the session dead-ended. Field evidence (#4124): 8 occurrences across 3 sessions/providers on one upstream model service; the cascade reached a second wall (gateway byte limit) before manual compressions freed the session.This PR makes bili learn the cap from the rejection itself and converge:
inspectContextOverflowgains an independent image-count channel (imageCountCap/imageCountSeen) for the two known rejection shapes (vLLM's exact format, including the> escaped as \u003ewire form seen in field logs, plus an At-most form).isOverflowkeeps its token-only meaning: a count wall can never mint a token-space arm (parseOverflowWindow/toTokenNumberreject <1000 anyway, and the no-window arm branch would shrink at a payload estimate).shedWireImagesToCount: count-space twin ofshedWireImages(rung A only — downscaling cannot reduce count; the protected tail YIELDS to the hard cap; noMAX_SHED_IMAGESslice since archiving is a cheap string swap). Oldest-first archive behind stable placeholder refs — originals stay in session storage, recoverable viadecompress({imageRef}). Byte-identical passthrough when under cap.session.stats.imageCountCapskeyed by normalized origin (survives restarts). Later requests to the same route converge BEFORE send when their count exceeds cap−1 — no rejected round-trip.Behavior changes (old → new)
Requests under the cap are byte-identical to before (asserted both directions, gate discipline #3617). No change to text-side metering or fold triggering.
Config surface
None added. Option B is config-free by design (learning from the rejection is the whole point — the reporter cannot pre-declare a shared gateway's cap). Option A (flat
imageCountCaproute/global/env mirroring theimageTokenCapprecedent, default off) remains a checklist item in #4124 awaiting owner shape sign-off; it can join this PR pre-merge if approved now.kernel-change
none (no files under kernel/ touched)
Tests
New
tests/issue4124-image-count-cap.test.ts(17 tests): detection matrix incl. escaped-wire form, status gate, negatives (token overflow stays token-only, Bedrock throttle dark on both channels, degenerate caps); shedding units per protocol (openai/anthropic/responses/google) with determinism, under-cap byte-identity, tail-yields-to-cap, maxImages=0, exempt outputs untouched, unparseable noop; relay integration (reactive rescue, proactive pre-send, pck skip) against a scripted mock upstream. Targeted neighbors green locally: context-overflow, cjk-context-overflow, side-request-isolation, issue2484, openai-pck-identity, upstream-error-log, issue3701-ws-byte-budget, issue1195/issue3787 overflow-refold, fix-arm-headroom-clamp (136 tests total). Full suite = CI.