Repository navigation
[PR] fix: preserve unknown anthropic top-level blocks verbatim in the anthropic wire codec - #3547
ranxianglei wants to merge 1 commit into
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.4755-pr.3547.2e08447This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3547.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重:不重复。 与在途 #3412(opaque-media-reserve)有交集但互补——#3412 预留了 opaque-media 的窗口配额,但当时 anthropic codec 会丢 document 块所以没给它计价;本 PR 让 document 块存活,因此必须在计价侧补上(共享 walker 取并集,PR 正文已标 sequencing)。两者修的是不同层(#3412 = reserve 计量 / 本 PR = codec 保留 + 计价),非同一改动的重复。 审核独立性: 以下结论由我独立读 diff + 本地复现得出,未照抄 PR 描述。kernel 侧逐行读了 结构化八问
行为变更(旧→新,披露)
代码整洁(外部提交从严)命名达意(unknownAnthropicBlocks / warnUnknownAnthropicBlocks / WireDropReport)、无死代码、无重复(WireDropReport 抽成两 detector 共用)、无魔法数(ANTHROPIC_KNOWN_BLOCKS 具名 Set、REMOTE_IMAGE_TOKENS 具名常量)、注释讲为什么、风格跟仓库一致、零新依赖。全项过。diff 10 文件全部 on-purpose,无无关 churn / 空行噪音。 测试有效性真测试,断言有牙:detector deepEqual 精确报告、warn 去重矩阵(同/异 type-set、跨 session、known-only 静默、null 静默、reset 后重报)、parity 断言(同字节 base64 document 与 image 计费恒等)、cost/count 符号锁步(每种 shape)、raw probe 同时吃 compact + spaced JSON、image 回归锚。非 tautology。注意:这些 host/kernel 测试此前因 lock-sync 红而全部 SKIPPED,从未在 CI 真正跑过——「41/41」「1296/1296」是作者本地值,非 CI 验证;现已解锁,正在 CI 执行。 CI 状态 + 我已清的阻塞
判定
|
(ai: vllm-qwen/qwen3.8-27b) CI 跟进(新 head
|
(ai: vllm-qwen/qwen3.8-27b) CI 全绿(最终确认,head 收工判定:
|
(ai: vllm-qwen/qwen3.8-27b) 协调通知:本 PR 的 kernel hunk 已被 #3961 吸收,请收窄为 host-only。 #3938/#3939 内核专项把「kernel 侧 wire 保真」收进统一节奏后,本 PR 的以下部分已逐字搬入 #3961(仅适配 master 现行
Closes #3410 链接保留在本 PR 上不变。 |
anthropicToCore's block switch had no default case: unknown top-level block types (in the wild today: document/PDF) fell out silently and their payload vanished on every rebuild with no warning. Unknown blocks now ride rawAnthropicBlock exactly like image/redacted_thinking — coreToAnthropic re-emits them byte-exact (cache_control included) and never renders the label text; protected.ts classifies them as media AND unrecoverable so a fold can never destroy a payload the host archives nowhere. Absorbs the kernel-side hunks of PR #3547 (adapted to master's #3642 canonical identity); #3547 keeps its host-side notice/billing surface. kernel-change: wire保真(#3938 / #3939⑥, 吸收#3547)
anthropicToCore's block switch had no default case: unknown top-level block types (in the wild today: document/PDF) fell out silently and their payload vanished on every rebuild with no warning. Unknown blocks now ride rawAnthropicBlock exactly like image/redacted_thinking — coreToAnthropic re-emits them byte-exact (cache_control included) and never renders the label text; protected.ts classifies them as media AND unrecoverable so a fold can never destroy a payload the host archives nowhere. Absorbs the kernel-side hunks of PR #3547 (adapted to master's #3642 canonical identity); #3547 keeps its host-side notice/billing surface. kernel-change: wire保真(#3938 / #3939⑥, 吸收#3547)
anthropicToCore's block switch had no default case: unknown top-level block types (in the wild today: document/PDF) fell out silently and their payload vanished on every rebuild with no warning. Unknown blocks now ride rawAnthropicBlock exactly like image/redacted_thinking — coreToAnthropic re-emits them byte-exact (cache_control included) and never renders the label text; protected.ts classifies them as media AND unrecoverable so a fold can never destroy a payload the host archives nowhere. Absorbs the kernel-side hunks of PR #3547 (adapted to master's #3642 canonical identity); #3547 keeps its host-side notice/billing surface. kernel-change: wire保真(#3938 / #3939⑥, 吸收#3547)
anthropicToCore's block switch had no default case: unknown top-level block types (in the wild today: document/PDF) fell out silently and their payload vanished on every rebuild with no warning. Unknown blocks now ride rawAnthropicBlock exactly like image/redacted_thinking — coreToAnthropic re-emits them byte-exact (cache_control included) and never renders the label text; protected.ts classifies them as media AND unrecoverable so a fold can never destroy a payload the host archives nowhere. Absorbs the kernel-side hunks of PR #3547 (adapted to master's #3642 canonical identity); #3547 keeps its host-side notice/billing surface. kernel-change: wire保真(#3938 / #3939⑥, 吸收#3547)
Host-side half of #3410: price document/other unknown top-level blocks in the billing reserve (probe + cost/count walkers) and warn once per shape that they ride verbatim but stay out of folding. The kernel-side codec fix (sidecar passthrough) ships separately in #3961 — merge #3961 first so priced bytes == forwarded bytes. Pin updated: #3398 document case now expects opaque-media pricing.
95eb5f5 to
4ffb236
Compare
(ai: vllm-qwen/qwen3.8-27b) 收窄完成,已按协调方案重推。此前冲突 worker 停摆(分支停留在旧 base 做了什么
验证:定向本地跑 51/51 绿(issue3410 新增套件、issue3398 更新后、issue1843、fix-1205、issue767 像素计费族);typecheck 干净。CI 已在 |
|
(ai: vllm-qwen/qwen3.8-27b) 关闭为 superseded by #3981。owner 在 #3938(2026-10-20)已把 PR-C 的 host 车道定为 #3981(stacked on #3961),本 PR 与其范围完全重合(unknownAnthropicBlocks 检测器、每-session NOTICE、计费对齐、同名测试文件)——一个 issue 只留一条 PR。 本 PR 今日收窄后的 head
合并顺序不变:#3961 → #3981。若 owner 希望保留本 PR 而非 #3981,reopen 即可,head 是干净的单 commit。 |
kernel-change: the anthropic wire codec's block switch has no default case — unknown top-level block types (in the wild today: document/PDF) are silently dropped inside the kernel's parse→core projection, which only the kernel owns (the drop point plus the protected-predicate semantics). Host side cannot fix it. Owner approval trail: #3487 (P0, "default mandatory fix" + repair direction "fallback passthrough for unknown blocks, at least warn before dropping") and the full implementation plan posted in the #3410 thread.
Closes #3410
What was broken
Rebuilding an anthropic request body ran every content block through a switch over the six known types (text/tool_use/tool_result/thinking/image/redacted_thinking) with no default — any other top-level block type was simply not emitted. In practice that means user-sent PDF
documentblocks vanished on every turn: no warning, no log, silent content loss (same severity class as fold-drop-text).Fix (one mechanism, kernel + host)
kernel/src/wire/anthropic.ts: newdefaultcase rides unknown blocks on the existingrawAnthropicBlocksidecar, exactly like image/redacted_thinking.coreToAnthropicalready re-emits sidecars verbatim (block-internalcache_controlincluded), so the round trip is byte-exact — verified by golden-free deepEqual regression tests.kernel/src/protected.ts: unknown anthropic blocks now count as media AND unrecoverable. Their payload exists only in the sidecar and bili cannot archive it to disk (image-restore handles images only), so folding would lose it permanently. Archivable images (建议:compress.stripImages默认开启——否则带图消息不可压缩会形成只升不降的输入地板撑爆上下文 #2607) and redacted_thinking keep their existing behavior.warnUnknownAnthropicBlocks, same bounded FIFO as the [Bug] OpenAI 线解析/重建无条件丢弃非 text/image_url 的 content part(如 DeepSeek Files APIfilepart)——不压缩也丢图 #1205 drop warning). It is a NOTICE, not a drop warning — nothing is dropped anymore; it tells users their client sent PDFs and that such blocks are excluded from folding.src/image-tokens.ts): document blocks are now priced like the openai file precedent (inline base64 by payload size, url refs flat 4096, bare/malformed-source flat 4096) across the cost walker, the lockstep count walker, and the raw-body probe. Before this PR the bytes were dropped so there was nothing to price; now that they are preserved, the fold gate must see them or it undercounts.Behavior change (old → new)
Overlap with in-flight #3412 (opaque-media-reserve)
#3412 deliberately left anthropic documents unpriced in
src/image-tokens.tsbecause the codec dropped them at the time. This PR makes them survive, so they must be priced here. If #3412 merges first, rebase takes the union of intent in the shared walker — flagged for sequencing.Verification
tests/issue3410-unknown-anthropic-block.test.ts(detector shapes, warn dedup matrix, three billing shapes, cost/count lockstep, probe admission incl. spaced JSON, image regression anchors). Targeted run with fix-1205/issue767/issue1843 siblings: 41/41 green.npm run typecheckclean;npm run buildgreen (dist-external-zod gate OK).Kernel change — human-only review tier per the Kernel Boundary.