Repository navigation
Description
来源: billion-context#1095 (ranxianglei/billion-context#1095) 分析 "Feature: image pre-compression for screenshot-heavy tool results (resize/quality routing)" 时发现。owner 裁定(2026-09-21):图像路由/降采样策略与 image_full 恢复通道属于 acp-kernel 管线,内核先行;billion-context 在内核就绪后做接入。
背景
Multimodal models bill images by pixel area (Anthropic ≈ w×h/750 visual tokens; OpenAI uses a high-detail tile model). A single phone screenshot (1568×2248) costs 1600+ tokens — more than a large code block. UI-debugging sessions burn a dozen of these per problem. The only tool today is absorb (model-distill to text): one full round-trip per image, and layout/color/alignment information is lost in textification.
Target behavior (owner-approved, 与 billion-context#1097 同源同节奏):
- 截图按内容类型路由降采样(长边/quality):shrink the image itself before it enters the wire — the model still sees the UI at a fraction of the tokens;
- 非截图原图直过(non-screenshot originals pass through untouched);
- 有损不可逆兜底:本地原图缓存 + 模型显式说"看不清"时重发全分辨率 = #1097 的图像特例
image_full(retrieve over the same store)。
归内核的部分(本 issue 范围)
1. 图像路由/降采样策略(pure decision function,零原生依赖)
Input: image metadata (mediaType, dimensions via header-only parse, base64 length, billing mode) + resolved config. Output: { action: "pass" | "downsample", recipe?: { maxDimension /*longest side*/, quality, format: "webp"|"jpeg"|"png" } }.
- Content-type routing: screenshot-like → downsample; non-screenshot → pass. v1 heuristic proposal: aspect-ratio band (long/short side ≈ [1.4, 2.6]) + min-dimension floor; behind a pluggable classifier interface so a trained router (headroom-style ONNX) can slot in later without protocol changes. Heuristic parameters need maintainer confirmation (open question 2).
- Deterministic: same input + config → same recipe, always. Load-bearing: the downsampled bytes become the standing wire content after arrival-time substitution; non-determinism would invalidate the prefix cache mid-session (same invariant as #1097's placeholder determinism).
2. Billing-aware token estimation + gate
estimatedImageTokens(b64, billing): pixels mode → decode dims from header only (PNG/GIF/BMP/WebP ≤64 b64 chars, JPEG SOF scan) → pixel-tile estimate (short side scaled UP to 768, long side capped 2048, 85 + 170·ceil(w/512)·ceil(h/512)); unknown dims → 16384 fallback; bytes mode → ceil(len/4). Gate: only route images whose estimate ≥ minTokens. Today this lives in billion-context src/image-tokens.ts; propose the canonical copy moves to the kernel so host and any future consumer share one implementation (open question 3).
3. image_full 恢复通道(#1097 retrieve 的图像特例)
- Tool contract: name
image_full, args{ ref: "mNNNNN" }, restores original-resolution images for that message ref for the rest of the session (sticky). Success/failure result strings follow absorb's failure-marker convention ([image_full FAILED: ...]); byte-stable system-note fragment (e.g.[Downscaled screenshots: … call image_full with that message's ref ("mNNNNN")]). - Sticky-restore state machine: per-session set of restored refs, cleared on compaction reset — refs are re-issued after state reset, so stale entries must never survive a reset.
- Stats counters exposed on session state: images shrunk, bytes saved, tokens saved (hosts log them alongside
[acp-usage]). - Store shape: image originals live in #1097's content store as binary sidecar entries under the same
mNNNNNkey scheme + index/lifecycle (append-once, drop-on-fold). Kernel owns the entry-kind contract (kind: "image"+ mediaType); file IO stays host-side.
4. Config type + defaults + validation
ImageCompressionConfig { enabled: boolean; minTokens?: number /*512*/; maxDimension?: number /*1280*/; quality?: number /*80*/; format?: "webp"|"jpeg"|"png" } — kernel owns type/defaults/validation; hosts do the three-level (global→provider→model) sub-field-wise merge exactly like absorb. Default off. Disabled ⇒ byte-identical pass-through, zero behavior change.
明确不归内核(host 侧,billion-context 接入时做)
sharpnative dependency + actual encode execution — the kernel stays zero-native-deps (it bundles inline into every consumer); the host executes the kernel's recipe (deterministic encode: resize fit-inside withoutEnlargement → flatten → jpeg{quality, mozjpeg} / png{compressionLevel:9} / webp{quality}; keep-smaller-if-original-wins) and reports results back.- Sidecar file IO under
<stateDir>/<sessionId>/store/(follows #1097's layout). - Wire-body injection across the four protocols (anthropic
image/ openaiimage_url/ responsesinput_image/ googleinlineData) + plugin-manifest registration. - Logging (
[acp-image]per-image lines,[acp-usage]cumulative suffix), npmoptionalDependenciespackaging.
依赖与排序
- billion-context#1097 (content-addressed store +
acp_retrieve, in progress) — the image pass hooks its arrival-time substitution point; image entries share its store. - This kernel change → release acp-kernel → verify live (
npm view acp-kernel version) → billion-context bumps the exact pin + integration PR (cross-repo rule: kernel ships first).
Reference implementation (pre-redesign WIP; reusable pieces: token estimator, deterministic encode recipe, tool defs ×4, config parse): billion-context branch 2026-09-21_image-compression @ b1fda60.
Acceptance criteria(kernel 侧)
- Routing decision function: pure, deterministic, unit-tested; screenshot-like & ≥minTokens → recipe; otherwise → pass
- Pluggable classifier interface (v1 heuristic default)
- Billing-aware token estimator (pixel-tile + bytes fallback)
-
image_fullcontract: name/schema/result strings/system-note/sticky-restore state machine (reset-safe)/failure modes -
ImageCompressionConfigtype + defaults + validation; three-level merge left to hosts - Stats counters on session state (shrunk / bytes saved / tokens saved)
- Disabled ⇒ byte-identical pass-through (regression-tested)
- Determinism property test: same input+config → identical recipe N times
Open questions(for maintainer)
- Pass placement: confirm arrival-time (once, at #1097's substitution point) rather than per-forward re-evaluation — consistent with "substitution only at arrival".
- v1 screenshot heuristic: aspect band ≈[1.4, 2.6] + min-dimension floor — OK, or prefer size-only gating in v1 with the classifier deferred?
- Estimator: move from billion-context into the kernel (single canonical copy) vs. duplicate?
Activity
ranxianglei commentedon Sep 22, 2026
[bot] 🏷 Triage done — proceeding with the kernel-first implementation per your ruling. Verified facts + my answers to the three open questions:
Verified before starting:
2026-09-21_image-compression@ b1fda60 no longer exists on the remote (checked all refs) — implementing from this issue's spec instead. The estimator reference (src/image-tokens.ts) lives on billion-context master and is the porting source.acp_retrieve) is not merged in billion-context master yet, so the kernel ships contracts only (store entry-kind types, tool schemas, sticky state machine) with zero file I/O — matching "kernel first, integrate later".Open questions — my recommendations (veto at PR review if you disagree):
PR scope: new module
src/image-compress.ts(+ tests), config types/defaults/validation (3-level merge like absorb), state fields + stats counters,image_fulltool contract (3 wire shapes, absorb-style failure markers), byte-stable system-note builder. Fully additive, default OFF ⇒ byte-identical pass-through when disabled; zero change to existing behavior.Follow-up with the PR link coming.
中文摘要:已按你的裁定开始内核侧实现;三个开放问题给出明确建议(到达时一次性替换、保留 v1 启发式参数、估算器规范副本移入内核),PR 评审时可否决。
ranxianglei commentedon Sep 22, 2026
[bot] 🏷 完成 — PR: #357 (Closes #353)
Kernel-side half landed as a new pure module
src/image-compress.ts(+ barrel exports):decideImageRoute(meta, config, classifier?)— pure, deterministic decision fn (disabled → not-an-image → below-min-tokens → not-screenshot-like → downsample w/ recipe). PluggableScreenshotClassifier; v1 heuristic = aspect band [1.4, 2.6] + short-side floor 720px; trained router slots in later without pipeline changes.image-tokens.ts— header-only dim parsing (PNG/GIF/BMP/WebP VP8X|VP8L|VP8/JPEG SOF scan, budgeted base64 head) + OpenAI high-detail tile model (short side up to 768, long capped 2048,85+170·tiles); unknown dims → 16384 fallback; bytes billing →ceil(b64len/4).IMAGE_FULL_TOOL/_OPENAI/_RESPONSES, args{ref}),applyImageFull()with success/idempotent/[image_full FAILED: ...]texts, sticky per-session restored-ref set, byte-stablebuildImageFullSystemNote(count), shrink records + cumulative stats (imagesShrunk/imageBytesSaved/imageTokensSaved),resetImageFullState()(hosts call on compaction reset/fork recovery/session restart),ImageStoreEntry {kind:"image",...}contract for the pending content store (#1097).imageCompression?: {enabled:false, minTokens:512, maxDimension:1280, quality:80, format:"webp"}— defaults/validation/sub-field merge exactly likeabsorb; host-side 3-level merge unchanged.Open questions answered in the PR description: (1) arrival-time substitution (determinism + prefix-cache stability); (2) kept the heuristic band+floor per the owner-approved spec — non-screenshot originals pass through untouched; known 3:2-photo false-positive documented until a trained router lands; (3) canonical estimator moves to the kernel, hosts keep their body-scanning wrapper.
Behavior-change disclosure: none user-visible. Feature OFF by default ⇒ byte-identical pass-through (regression-tested battery under
enabled:false). Additive exports only. Fresh states now seed 3 zero stats counters + 2 empty arrays carried throughmergeCompressionState; old persisted states untouched (all readers??-guarded); one pre-existing test literal updated to match.Verification:
npm run typecheckclean ·npm test800/800 ·npm run buildclean. Host-side work (sharp encode execution, sidecar I/O, wire injection ×4 protocols, logging, packaging) stays in billion-context for the #1095/#1097 integration.中文摘要:实现了图像预压缩的路由/降采样决策与 image_full 恢复通道(纯内核模块,默认关闭、无行为变更),typecheck/800 项测试/build 全绿,可以进入 review。
feat: image pre-compression routing/downsample + image_full restore c…
Merge pull request #363 from ranxianglei/pr-357