Skip to content

[PR] fix: exempt the dsh plugin lane and compaction calls from the 32 MiB ingest band (#6908) - #6933

Closed
ranxianglei wants to merge 2 commits into
masterfrom
2026-11-16_ingest-dsh-exemption
Closed

ranxianglei wants to merge 2 commits into
masterfrom
2026-11-16_ingest-dsh-exemption

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

What this fixes

The 32 MiB ingest band (network.ingestBodyBytes) measured the pre-folding inbound body, but its premise ("any parseable 32 MiB+ text body is ≥ ~8M tokens, beyond every window") equates INBOUND BYTES with MODEL INPUT. That holds for the outbound/incremental lanes, not for the plugin lanes, which re-send the host's ENTIRE raw ledger every round on the full-resend wire (dsh on the OpenAI wire, no incremental lane there, #3433; opencode V2 the same): the body is the host's history, not the model input, and bili's job is to fold it (dsh field: 32.0 MiB inbound vs 446.4 KiB outbound, 70x; opencode field #7426: 33.5 MiB inbound vs ~391K tokens of actual context).

A healthy plugin-lane session that crossed 32 MiB was 413'd at ingest forever — and the dsh compaction call (the ONLY action that shrinks the host's ledger via bili, #3733) is itself an over-band body (rc.2 replays the full shadowed region, #2193), so the self-heal self-locked: the one action that could shrink the ledger was the one the gate refused.

The fix

Two dimensionally-motivated exemptions at the gate (src/server/handle.ts):

Plus two observability fixes on the same line:

  • inboundBytesOf (src/agent/dsh-native.ts) reported UTF-16 code units (raw.length) for strings; the host-json-parse inbound=…B field now reports UTF-8 bytes, matching the gate's byteLength (an 8%+ unit gap hid how close the body was to the wall).
  • The ingest 413 log now names the ACTUAL decoded byte count next to the cap (was limit-only).

⚠️ Behavior change (old → new)

Tests

5 new e2e cases in tests/issue5794-ingest-cap.test.ts:

All 8 cases in the file pass; typecheck clean.

Closes #6908
Closes #7426

@ranxianglei ranxianglei added bug Something isn't working dsh p2 scope:wiring labels Nov 16, 2026
@ranxianglei ranxianglei changed the title fix: exempt the dsh plugin lane and compaction calls from the 32 MiB ingest band (#6908) [PR] fix: exempt the dsh plugin lane and compaction calls from the 32 MiB ingest band (#6908) Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-16_ingest-dsh-exemption (aa2ba04)

Latest build — install this exact version

npm install -g billion-context@0.0.11130-pr.6933.d17678b

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
aa2ba04 billion-context@0.0.11130-pr.6933.d17678b build
9707b15 billion-context@0.0.10768-pr.6933.f82e15f build
1e9af39 billion-context@0.0.10381-pr.6933.9676e07 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr6933.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷

Reviewed PR #6933 (exempt the dsh plugin lane and compaction calls from the 32 MiB ingest band, closes #6908).

I pulled the branch (2026-11-16_ingest-dsh-exemption), read the full diff against current master (merge-base de895aa), and verified the code, the tests, and the typecheck locally.

What I verified:

Minor observations (non-blocking):

  1. dshCompactionSniff is computed eagerly (one .includes scan per request) while countImagesInRawBody is lazy. The cost is negligible for the common small-body case and bounded (one scan) for over-band bodies, so it's acceptable — but a lazy computation would be marginally cleaner.
  2. Test (A) verifies the gate exemption (200 + 1 upstream call) but not that the fold actually shrank the upstream body. The gate exemption is the fix under test, so this is acceptable, but asserting the upstream body size would make the "folded" claim airtight.

CI: 0 failed, 1 pending (mysql-contract — unrelated to this PR, which doesn't touch storage). I'll confirm the final verdict when it's green.

8问 (中文):

  1. 是什么问题: 32 MiB 摄入成本带量的是折叠前的入站体,把「入站字节」等同于「模型输入」。dsh 插件车道每轮全量重发宿主历史(无增量车道),入站体是宿主历史而非模型输入,折叠完全正常(出站 446 KiB)的会话一旦入站越过 32 MiB 就被永久 413;且唯一能缩小宿主历史的 dsh 压缩调用本身也是超带体,自锁。
  2. 什么原因导致的: 闸的前提「32 MiB+ 可解析文本体 ≥ ~8M token」对出站/增量车道成立,对全量重发车道量纲错误;闸在 JSON.parse 之前抛,超带请求永远到不了 isDshCompactionCall。
  3. 影响范围与严重程度: 仅 dsh 插件车道(及带压缩标记的超带请求);入站体越过 32 MiB 后每轮 413,会话完全不可用(自锁,无法自愈)。属 p1(可用性缺陷)。
  4. 怎么修复的: 在闸上加两个量纲动机的豁免——(A) x-bili-plugin: dsh(体是折叠底物)、(B) 48 字节压缩标记嗅探(自愈必须到达 isDshCompactionCall);嗅探只是放行,不是决策,完整标记判定仍在 parse 后跑。另修两处观测:inboundBytesOf 改报 UTF-8 字节(对齐闸的 byteLength)、413 日志带上实际解码字节数。
  5. 是否有回归问题: 非 dsh 超带体仍 413(负向测试钉住);slot 预算/接收上限/解码上限全部保留;dsh 车道超带体现在付 parse 成本(结构成本,低于悬崖时每轮都在付,闸只是在 32 MiB 处制造了不连续)。无回归。
  6. 是否破坏缓存: 不涉及(纯摄入闸 + 观测日志,不触碰缓存/重试/游标/去重键)。
  7. 是否改了设计方向: 不涉及(闸的机制不变,只加了两个豁免;出站/增量车道行为不变)。
  8. 是否引入了新的配置: 不涉及(复用现有 network.ingestBodyBytes 与 x-bili-plugin 头,无新 env/config/默认值)。

判定: 代码层面可合(定向测试全绿、typecheck 干净、无回归、行为变更已披露)。CI 0 failed / 1 pending(mysql-contract,与本 PR 无关),全量判定待绿后确认。

…ingest band (#6908)

The ingest band (network.ingestBodyBytes, default 32 MiB) measured the
PRE-folding inbound body, but its premise ("any parseable 32 MiB+ text
body is >= ~8M tokens, beyond every window") equates INBOUND BYTES with
MODEL INPUT. That holds for the outbound/incremental lanes, not for the
dsh plugin lane, which re-sends the host's ENTIRE raw ledger every round
on the OpenAI wire (no incremental lane there, #3433): the body is the
host's history, not the model input, and bili's job is to fold it (field:
32.0 MiB inbound vs 446.4 KiB outbound, 70x). A healthy session that
crossed 32 MiB was 413'd at ingest forever, and the dsh compaction call
(the ONLY action that shrinks the host's ledger via bili) is itself an
over-band body, so the self-heal self-locked.

Two dimensionally-motivated exemptions:
  (A) x-bili-plugin: dsh -> the body is fold substrate, exempt.
  (B) the dsh compaction marker present -> the self-heal must reach
      isDshCompactionCall (#3733 answers it locally with the folded
      snapshot); rc.2 replays the full shadowed region (#2193), so it
      crosses the band and must pass the gate. A 48-byte sniff decides;
      a false positive (a turn that merely quotes the marker) just pays
      the parse cost, identical to today's under-cliff behavior — the
      full marker-decisive check still runs after parse.

Also fixes two observability gaps on the same line:
  - inboundBytesOf reported UTF-16 code units (raw.length) for strings;
    the host-json-parse inbound=...B field now reports UTF-8 bytes,
    matching the gate's byteLength (an 8%+ unit gap hid how close the
    body was to the wall).
  - the ingest 413 log now names the ACTUAL decoded byte count next to
    the cap (was limit-only), so operators can see the margin.

Tests: 3 new e2e cases in issue5794-ingest-cap.test.ts — (A) a dsh
plugin-lane over-band full-resend body is folded and forwarded (200 + 1
upstream call), (B) a compaction-marker body passes the gate and reaches
the compaction guard (422), and the negative: a plain over-band body with
no dsh header/marker is still refused 413 (the band is not widened).
@ranxianglei
ranxianglei force-pushed the 2026-11-16_ingest-dsh-exemption branch from 1e9af39 to 9707b15 Compare November 18, 2026 07:24
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Conflict resolved — the PR is re-based onto current master and mergeable again.

What I did this run:

  1. Verified live state: PR [PR] fix: exempt the dsh plugin lane and compaction calls from the 32 MiB ingest band (#6908) #6933 head 1e9af39a71 on branch 2026-11-16_ingest-dsh-exemption, base master at 548e396022, merge state dirty, 142 commits behind.
  2. Rebased the PR branch onto origin/master. Exactly ONE conflict, in the import block of src/server/handle.ts: master had added isMcodeOneShotAuxRequest, isMcodeTitleRequest, isOpencodeTitleRequest to the ./side-request.js import on the same lines the PR modified (the PR added DSH_COMPACTION_MARKER_SNIFF to the ./dsh-compaction-guard.js import). Resolved as the union of both — master's new side-request imports kept, the PR's sniff import kept. Verified DSH_COMPACTION_MARKER_SNIFF exists in the rebased tree's src/server/dsh-compaction-guard.ts:143.
  3. Verified the auto-merged hunks are intact: the two [Bug][dsh][ingest] The 32 MiB ingest cost is measured against the pre-folding inbound body: a normal session folded down | [Bug][dsh][ingest] 32 MiB 摄入成本带量的是折叠前入站体:折叠到 446 KiB 的正常会话被永久 413 #6908 exemptions at src/server/handle.ts:561-565 (x-bili-plugin: dsh header + 48-byte compaction-marker sniff), the 413 log with actual decoded byte count, and the UTF-8 byte fix in src/agent/dsh-native.ts (inboundBytesOf).
  4. Verified on the rebased head: npm run typecheck clean; targeted tests/issue5794-ingest-cap.test.ts 6/6 pass (incl. the 3 new [Bug][dsh][ingest] The 32 MiB ingest cost is measured against the pre-folding inbound body: a normal session folded down | [Bug][dsh][ingest] 32 MiB 摄入成本带量的是折叠前入站体:折叠到 446 KiB 的正常会话被永久 413 #6908 cases: dsh plugin-lane over-band body folded+forwarded, compaction-marker body reaches the guard, plain over-band body still 413). The rebased diff vs master is byte-identical to the original PR's stat (3 files, +139/−4) — no content drift from the rebase.
  5. Fetched first (remote head unchanged, no concurrent work), then force-pushed the rebased head 9707b15125. Live re-check: mergeable: true, state blocked (= conflict gone; only CI/approval requirements remain). The final head carries no [ework:ci-held] marker, so this push is the branch's one full CI pass.

八问:

  1. 是什么问题:PR 分支落后 master 142 个提交,src/server/handle.ts 的 import 区块与 master 新增的 side-request 导入冲突,PR 处于 dirty 无法合并状态。
  2. 什么原因导致的:PR 开发期间 master 合入了新工作(mcode/opencode title-request 检测),恰好改在 PR 触碰的同一组 import 行上,git 无法自动合并——纯时序冲突,非设计分歧。
  3. 影响范围与严重程度:仅该 PR 分支的合并能力,无用户面影响;PR 本身功能未变。
  4. 怎么修复的:rebase 到当前 master,唯一冲突按「双方并集」解决(保留 master 新导入 + 保留 PR 的 DSH_COMPACTION_MARKER_SNIFF 导入),force-push 后 PR 恢复 mergeable。
  5. 是否有回归问题:无。rebase 后 diff 与原 PR 完全一致(3 文件 +139/−4);typecheck 干净;定向测试 6/6 通过;未动任何 PR 之外的行为。
  6. 是否破坏缓存:不涉及——未触碰缓存/重试/增量游标/去重逻辑。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

收工判定:冲突已解,分支已就绪,等 CI 全量结果即可。

#7426)

The #6908 exemption was dsh-only (x-bili-plugin === "dsh"), so the
opencode lane — which has the SAME full-resend structure (the host
re-sends its ENTIRE raw ledger every round; bili's job is to fold it) —
was still 413-locked at the 32 MiB band (field: 33.5 MiB inbound vs
~391K tokens of actual context, a healthy 390-block session).

The criterion generalizes to ANY plugin lane: the x-bili-plugin header
announces "my body is my ledger, fold it," which is dimensionally true
for every cooperative plugin lane. Incremental-lane turns carry small
deltas and never reach the band, so the wider net is harmless there.

Both directions pinned: opencode lane passes the gate (folded +
forwarded), non-plugin over-band bodies are still refused 413.
@ranxianglei

Copy link
Copy Markdown
Owner Author

Superseded by #7443. Decision for #7426: lift the ingest band for ALL lanes (default 0 = unlimited), not just exempt the dsh/plugin lanes. With the band off by default there is nothing left to exempt, so this scoped approach is no longer needed. BILI_INGEST_BODY_BYTES remains as the opt-in re-arm path if a specific deployment hits the pre-fold parse CPU.

ranxianglei added a commit that referenced this pull request Nov 21, 2026
)

#5794's ingest band proxied 'beyond every model window' by raw decoded bytes,
which is invalid on the re-send lanes (opencode/dsh re-send the whole raw ledger
every turn and bili folds it) — it 413'd a 33.5 MB opencode turn whose folded
context was tiny. Default -> 0 (unlimited, all lanes); the sibling receive (5 GiB)
/ decode (10 GiB) transport guards still bound pathological bodies.
BILI_INGEST_BODY_BYTES re-enables the band. Supersedes #6933.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment