Repository navigation
[PR] Fix #5618 usage wall: remote-URL / file-ref media becomes foldable, its re-access handles ride the summary - #5629
Conversation
… stored summary (#5620) The #5620 usage wall: in image-heavy workflows every message carrying remote-URL images / DeepSeek Files API file refs was permanently pinned by hasUnrecoverableMediaPayload, turn integrity pinned its partners, and the context floor climbed to ~70% of the window while folds kept pace with growth but could never lower it. Split the class (conservative enumeration): - new kernel/src/media-refs.ts: extractMediaRefs pulls known re-access handles (non-data: URLs, file ids, fileUris) from all five wire sidecar families; hasRefRecoverableMediaPayload = at least one handle; unknown opaque shapes keep pinning until their semantics are known - hasUnrecoverableMediaPayload media branches now return !hasRefRecoverableMediaPayload(msg); control-plane structured system branches (#3992/#4166) and refless shapes are untouched - apply path appends a deterministic MEDIA REFS section to the stored block summary (applySingleRange union incl. consumed-block carry; applyRefolds per-block carry), so handles survive folds/merges/ distillations mechanically — never model discretion (on the wire a URL part is often invisible to the summarizer); decompress restores the original losslessly either way - compression-rules: MEDIA REFS preservation added to T1/T2/T3 rules; golden regenerated (justification: #5620 root fix, three prompt fields extended additively, COMPRESS_PHILOSOPHY unchanged) - cosmetic: normalize one stray leading space on the tier-2 Open-objectives bullet (pre-existing, touched while inserting the sibling line) Legacy sessions self-heal without migration code: pre-#5620 BLOCKED refs on now-foldable media migrate to fresh numbers via the existing #3805 deferred-render path (carrier render deferred to next fold boundary). kernel version 0.0.179 -> 0.0.180
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.8064-pr.5629.904d4e5This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5629.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 查重: 不重复 — 按失败签名搜了 open+closed 的 issue/PR(#5618、usage wall、ref-recoverable media):本 PR 是 #5618 唯一在途修复。搜索还命中 #4629(正文含 "5618"),但其内容是 #4600 的 codex forged-summary ledger,机制无关。 标签: bug + dsh + scope/kernel + p1 — 图片密集的 dsh 工作流里上下文地板钉在窗口 ~70% 只升不降 = 上下文失控增长类,按 owner 2026-10-19 修订一律 p1。 审查已开始:分支 |
…rule line
static-gates was red on both legs: surface-config.test.ts ("default
builders are byte-stable against checked-in fixtures") compares
buildCompress{,Text,Hybrid}SystemPrompt() against kernel/tests/fixtures/,
and the #5620 prompt change added one line ("Media re-access handles -
...") to all three default prompts without regenerating those fixtures.
Regenerated all three from the current builders; each fixture diffs by
exactly that one added line - no other drift. Local: kernel suite
1576/1576, pi-subagents 263/263, build + thin-install smoke OK,
wire-contract golden 23/23.
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Review 完成:代码通过,CI 红已定位并修复(独立小修 commit CI 红根因与修复:static-gates 双腿(windows/22、ubuntu/24)红在 结构化八问
测试有效性:新回归文件 8 个测试全有牙(五族 wire 提取矩阵含负例、幂等/陈旧孪生替换、apply e2e 断言段落落位、merge 双源句柄逐字存活、legacy BLOCKED→新号迁移);carrier-advertise-validate 与 issue3410 按 #3617 双向门禁纪律翻转并补负行。两个非阻塞观察:refold 路径的携带只被间接覆盖(trio 幂等测试+merge 测试覆盖同一机制,可接受);参数化 data URL(如带 charset)会被正则判成「引用」而非内联——属过度保留方向(摘要变大、不丢句柄),且与 host 归档通道同形,是既有形状不是本 PR 引入,不建议在此 PR 动。 收工判定:代码与行为审查通过,diff 干净(14 个文件全部服务于本修复,fixture 小修独立成 commit),CI 预期在新 head |
ranxianglei
left a comment
There was a problem hiding this comment.
Review 结论:可合并(已实测,非纸面推断)
本地全量验证(head 5104bd5e,干净 worktree):
- kernel 套件 1576/1576 pass(含新文件 issue5620-ref-recoverable-media.test.ts 与翻转后的 media-payload 正反例)
- 宿主侧受影响套件(wire-contract golden + media/protected 家族)49/49 pass,golden
kernel.json差异确认只增不改(COMPRESS_PHILOSOPHY 原样) - root typecheck rc=0
根因链路复核(master @ 5dee4850,即 #5621 合入后现状)成立:
src/acp-status.ts:278的 PRESSURE NOTE 门槛仍是maxPending < nudgeGrowthTokens——媒体墙会话因为被钉住的回合永远挂着一条 viable range,该条件结构性永不满足,所以日志里看到 72% 用量却没有任何解释;- 本 PR 对
kernel/src/protected.ts谓词的拆分(有可提取句柄的非 data: URL / file id / fileUri ⇒ 可折叠,句柄机械写进摘要的MEDIA REFS段;data: 内联字节、未知形状{type:"mystery"}、document 块、videoMetadata-only 仍钉死)与日志症状一一对应;负例 pin 齐全,BLOCKED-ref 存量迁移走既有 #3805 deferred-render 通道。
两条观察(均不阻塞):
compress.tsapplySingleRange 中collectFoldMediaRefs→appendMediaRefsSection发生在validateSummaryLength之后——一次折叠若含几十条长签名 URL,存储摘要会临时超过maxSummaryLength,直到下次 T2 tier merge 重蒸馏才收敛(merge 同样携带句柄,最终有界)。行为可接受,后续可选做软上限/告警。- 残余风险(PR body 已披露、owner 已知情批准 option A):句柄是"重新取图的门牌",不是字节本身;过期签名 URL 或临时代理重启后模型侧取不到(decompress 无损恢复不受影响)。dsh 浏览器截图恰恰经常走本地代理 URL——正是 #5618 那类会话,知悉即可,不构成回滚理由。
合并顺序提示: 与 #5619(MEDIA FLOOR)共享 hasUnrecoverableMediaPayload——两者都合入后 #5619 的报告语义自动收窄为"真·无法再取的质量",互补不冲突,先后随意。分支当前基于 c29b118d、落后 master 一个 #5621(mitm-lane,文件不相交),合并前顺手 sync 一下更稳。
ranxianglei
left a comment
There was a problem hiding this comment.
Review 结论:可合并(已实测,非纸面推断)
本地全量验证(head 5104bd5e,干净 worktree):
- kernel 套件 1576/1576 pass(含新文件 issue5620-ref-recoverable-media.test.ts 与翻转后的 media-payload 正反例)
- 宿主侧受影响套件(wire-contract golden + media/protected 家族)49/49 pass,golden
kernel.json差异确认只增不改(COMPRESS_PHILOSOPHY 原样) - root typecheck rc=0
根因链路复核(master @ 5dee4850,即 #5621 合入后现状)成立:
src/acp-status.ts:278的 PRESSURE NOTE 门槛仍是maxPending < nudgeGrowthTokens——媒体墙会话因为被钉住的回合永远挂着一条 viable range,该条件结构性永不满足,所以日志里看到 72% 用量却没有任何解释;- 本 PR 对
kernel/src/protected.ts谓词的拆分(有可提取句柄的非 data: URL / file id / fileUri ⇒ 可折叠,句柄机械写进摘要的MEDIA REFS段;data: 内联字节、未知形状{type:"mystery"}、document 块、videoMetadata-only 仍钉死)与日志症状一一对应;负例 pin 齐全,BLOCKED-ref 存量迁移走既有 #3805 deferred-render 通道。
两条观察(均不阻塞):
compress.tsapplySingleRange 中collectFoldMediaRefs→appendMediaRefsSection发生在validateSummaryLength之后——一次折叠若含几十条长签名 URL,存储摘要会临时超过maxSummaryLength,直到下次 T2 tier merge 重蒸馏才收敛(merge 同样携带句柄,最终有界)。行为可接受,后续可选做软上限/告警。- 残余风险(PR body 已披露、owner 已知情批准 option A):句柄是"重新取图的门牌",不是字节本身;过期签名 URL 或临时代理重启后模型侧取不到(decompress 无损恢复不受影响)。dsh 浏览器截图恰恰经常走本地代理 URL——正是 #5618 那类会话,知悉即可,不构成回滚理由。
合并顺序提示: 与 #5619(MEDIA FLOOR)共享 hasUnrecoverableMediaPayload——两者都合入后 #5619 的报告语义自动收窄为"真·无法再取的质量",互补不冲突,先后随意。分支当前基于 c29b118d、落后 master 一个 #5621(mitm-lane,文件不相交),合并前顺手 sync 一下更稳。
…t lines #5620 added 'Media re-access handles' lines to the three default compression prompts (howToCompressRules / tier2DistillRules / tier3CondenseRules in kernel/src/compression-rules.ts). Those prompts ride every outbound turn body and feed the persisted session state, so the committed tests/golden/storage-render/*.json snapshots drift on their derived digests (per-turn sha/shaCiteNorm/bytes + state/stats/metadata/nudge). The PR regenerated tests/golden/wire-contract/kernel.json and the kernel fixtures but missed this downstream suite (golden-storage-render.test.ts + storage-engine-proofs.test.ts share these goldens), reding 8 subtests across ubuntu/windows x node 22/24. Regenerated via scripts/update-storage-render-goldens.ts. Diff is purely digest/byte-count deltas: each wire body grows a UNIFORM +359 bytes (the constant-size prompt addition), zero decreases; blockContents, fold positions (foldAt 9 / editedAt 12) and message structure are byte-identical. No behavioral regression.
(ai: vllm-qwen/qwen3.8-27b) CI 已修好并推送到 PR 分支 是什么问题:6 个红的
什么原因:本 PR(#5620)给三个默认压缩 prompt( 怎么修的:用仓库官方脚本 回归审查(重点):逐行核了重生成 diff —— 纯派生值变化,无行为回归:
是否破坏缓存:不涉及。改的是测试快照文件,不是运行时代码;前缀缓存/重试/去重等策略零改动。 是否改了设计方向 / 是否引入新配置:均无。我没有改动任何运行时代码,也未新增配置。"ref-based media 可折叠 + prompt 加行"这一行为变更是 PR 作者本就已做的,我只把下游测试快照对齐到它,没有引入任何新的行为漂移。 收工判定:本地定向验证通过(复现同签名失败 → 重生成 → diff 复查确认无害 → 两文件 17/17 绿)。CI 会在新 head 一点提醒(非阻塞):该 PR 当前 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.1028-master.23a826d # this exact build
npm install -g billion-context@master # always the newest merged build
|
kernel-change: #5618 usage-wall root fix — splits hasUnrecoverableMediaPayload's media class so ref-based media (remote URL / file id / fileUri) becomes foldable with its re-access handle mechanically recorded in the stored summary. Owner approved in-thread (「根本上解决吧」, #5618). Host side cannot solve this: the pin lives in kernel fold/advertise/ref gates (four sites, all consuming the same predicate).
What & why
In image-heavy dsh workflows every message carrying a remote-URL image or a DeepSeek Files API file ref was permanently pinned: BLOCKED ref (never citable), never advertised as compressible, carved out of every fold, and turn integrity withdrew the turns around it. Folds kept pace with growth (~50K each) but the context floor climbed to ~70% of the window and never came down (log evidence in #5618: input=566495/792576 at 72%, 19× "No compressible ranges remain").
The fix splits the class with a conservative enumeration: a media message is FOLDABLE iff it exposes at least one extractable re-access handle (non-data: URL, file id, fileUri). Folding then loses nothing recoverable — the handle rides the stored summary verbatim, and decompress restores the original message losslessly regardless.
Behavior changes (old → new)
image_urlnon-data: singular/plural,{type:"file"}parts, anthropic url-source images incl. tool_result-embedded, responsesinput_imagenon-data:, googlefileData.fileUri): OLD = permanently pinned (BLOCKED ref, never advertised, excluded from every fold with warning, partners withdrawn by turn integrity → the Context compression isn't working — it's stuck at 70% dsh. | 上下文压缩不掉 维持在 70 % dsh #5618 floor). NEW = foldable like text, numeric ref, handle appended verbatim to the block summary in a deterministicMEDIA REFSsection.tests/golden/wire-contract/kernel.jsonregenerated — justification: Context compression isn't working — it's stuck at 70% dsh. | 上下文压缩不掉 维持在 70 % dsh #5618 root fix, three fields additive, COMPRESS_PHILOSOPHY unchanged). Mechanical append does NOT rely on model compliance: on the OpenAI wire the URL string of an image part is fed to the vision encoder and may be invisible to the summarizer.Unchanged by design: control-plane structured system message pinning (#3992/#4166); unknown/opaque shapes keep pinning (conservative enumeration — no evidence their payloads are re-acquirable); archivable inline media (#2607); videoMetadata-only Google parts (no fetchable field — still pinned); document blocks in tool_results (no established ref semantics — still pinned).
Legacy sessions
No migration code: pre-fix BLOCKED refs on now-foldable media migrate to fresh numbers through the existing #3805 deferred-render path (ref minted now, carrier tag render deferred to the next fold boundary — prefix-cache safe).
Tests (targeted local; full-suite verdict = CI)
kernel/tests/issue5620-ref-recoverable-media.test.ts(8): extraction matrix per wire family + conservative-pin negative, section idempotency / stale-twin replacement, block-boundary merge carry-forward (both source handles survive), apply-path append, legacy BLOCKED→numeric migrationmedia-payload.test.ts: rows flipped to the new classification + new positive/negative rows (both directions per gate discipline Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617);carrier-advertise-validateandissue3410parity pins updated to the post-split truthResiduals (tracked in-thread, not blocking)
Part of #5618
kernel version 0.0.179 → 0.0.180