Skip to content

Image pre-compression: routing/downsample strategy + image_full restore channel (billion-context#1095) #353

Open

Description

@ranxianglei

来源: billion-context#1095 (ranxianglei/billion-context#1095) 分析 "Feature: image pre-compression for screenshot-heavy tool results (resize/quality routing)" 时发现。owner 裁定(2026-09-21):图像路由/降采样策略与 image_full 恢复通道属于 acp-kernel 管线,内核先行;billion-context 在内核就绪后做接入。

背景

Multimodal models bill images by pixel area (Anthropic ≈ w×h/750 visual tokens; OpenAI uses a high-detail tile model). A single phone screenshot (1568×2248) costs 1600+ tokens — more than a large code block. UI-debugging sessions burn a dozen of these per problem. The only tool today is absorb (model-distill to text): one full round-trip per image, and layout/color/alignment information is lost in textification.

Target behavior (owner-approved, 与 billion-context#1097 同源同节奏):

  • 截图按内容类型路由降采样(长边/quality):shrink the image itself before it enters the wire — the model still sees the UI at a fraction of the tokens;
  • 非截图原图直过(non-screenshot originals pass through untouched);
  • 有损不可逆兜底:本地原图缓存 + 模型显式说"看不清"时重发全分辨率 = #1097 的图像特例 image_full(retrieve over the same store)。

归内核的部分(本 issue 范围)

1. 图像路由/降采样策略(pure decision function,零原生依赖)

Input: image metadata (mediaType, dimensions via header-only parse, base64 length, billing mode) + resolved config. Output: { action: "pass" | "downsample", recipe?: { maxDimension /*longest side*/, quality, format: "webp"|"jpeg"|"png" } }.

  • Content-type routing: screenshot-like → downsample; non-screenshot → pass. v1 heuristic proposal: aspect-ratio band (long/short side ≈ [1.4, 2.6]) + min-dimension floor; behind a pluggable classifier interface so a trained router (headroom-style ONNX) can slot in later without protocol changes. Heuristic parameters need maintainer confirmation (open question 2).
  • Deterministic: same input + config → same recipe, always. Load-bearing: the downsampled bytes become the standing wire content after arrival-time substitution; non-determinism would invalidate the prefix cache mid-session (same invariant as #1097's placeholder determinism).

2. Billing-aware token estimation + gate

estimatedImageTokens(b64, billing): pixels mode → decode dims from header only (PNG/GIF/BMP/WebP ≤64 b64 chars, JPEG SOF scan) → pixel-tile estimate (short side scaled UP to 768, long side capped 2048, 85 + 170·ceil(w/512)·ceil(h/512)); unknown dims → 16384 fallback; bytes mode → ceil(len/4). Gate: only route images whose estimate ≥ minTokens. Today this lives in billion-context src/image-tokens.ts; propose the canonical copy moves to the kernel so host and any future consumer share one implementation (open question 3).

3. image_full 恢复通道(#1097 retrieve 的图像特例)

  • Tool contract: name image_full, args { ref: "mNNNNN" }, restores original-resolution images for that message ref for the rest of the session (sticky). Success/failure result strings follow absorb's failure-marker convention ([image_full FAILED: ...]); byte-stable system-note fragment (e.g. [Downscaled screenshots: … call image_full with that message's ref ("mNNNNN")]).
  • Sticky-restore state machine: per-session set of restored refs, cleared on compaction reset — refs are re-issued after state reset, so stale entries must never survive a reset.
  • Stats counters exposed on session state: images shrunk, bytes saved, tokens saved (hosts log them alongside [acp-usage]).
  • Store shape: image originals live in #1097's content store as binary sidecar entries under the same mNNNNN key scheme + index/lifecycle (append-once, drop-on-fold). Kernel owns the entry-kind contract (kind: "image" + mediaType); file IO stays host-side.

4. Config type + defaults + validation

ImageCompressionConfig { enabled: boolean; minTokens?: number /*512*/; maxDimension?: number /*1280*/; quality?: number /*80*/; format?: "webp"|"jpeg"|"png" } — kernel owns type/defaults/validation; hosts do the three-level (global→provider→model) sub-field-wise merge exactly like absorb. Default off. Disabled ⇒ byte-identical pass-through, zero behavior change.

明确不归内核(host 侧,billion-context 接入时做)

  • sharp native dependency + actual encode execution — the kernel stays zero-native-deps (it bundles inline into every consumer); the host executes the kernel's recipe (deterministic encode: resize fit-inside withoutEnlargement → flatten → jpeg{quality, mozjpeg} / png{compressionLevel:9} / webp{quality}; keep-smaller-if-original-wins) and reports results back.
  • Sidecar file IO under <stateDir>/<sessionId>/store/ (follows #1097's layout).
  • Wire-body injection across the four protocols (anthropic image / openai image_url / responses input_image / google inlineData) + plugin-manifest registration.
  • Logging ([acp-image] per-image lines, [acp-usage] cumulative suffix), npm optionalDependencies packaging.

依赖与排序

  1. billion-context#1097 (content-addressed store + acp_retrieve, in progress) — the image pass hooks its arrival-time substitution point; image entries share its store.
  2. This kernel change → release acp-kernel → verify live (npm view acp-kernel version) → billion-context bumps the exact pin + integration PR (cross-repo rule: kernel ships first).

Reference implementation (pre-redesign WIP; reusable pieces: token estimator, deterministic encode recipe, tool defs ×4, config parse): billion-context branch 2026-09-21_image-compression @ b1fda60.

Acceptance criteria(kernel 侧)

  • Routing decision function: pure, deterministic, unit-tested; screenshot-like & ≥minTokens → recipe; otherwise → pass
  • Pluggable classifier interface (v1 heuristic default)
  • Billing-aware token estimator (pixel-tile + bytes fallback)
  • image_full contract: name/schema/result strings/system-note/sticky-restore state machine (reset-safe)/failure modes
  • ImageCompressionConfig type + defaults + validation; three-level merge left to hosts
  • Stats counters on session state (shrunk / bytes saved / tokens saved)
  • Disabled ⇒ byte-identical pass-through (regression-tested)
  • Determinism property test: same input+config → identical recipe N times

Open questions(for maintainer)

  1. Pass placement: confirm arrival-time (once, at #1097's substitution point) rather than per-forward re-evaluation — consistent with "substitution only at arrival".
  2. v1 screenshot heuristic: aspect band ≈[1.4, 2.6] + min-dimension floor — OK, or prefer size-only gating in v1 with the classifier deferred?
  3. Estimator: move from billion-context into the kernel (single canonical copy) vs. duplicate?

Activity

  1. ranxianglei commented on Sep 22, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 Triage done — proceeding with the kernel-first implementation per your ruling. Verified facts + my answers to the three open questions:

    Verified before starting:

    • The referenced WIP branch 2026-09-21_image-compression @ b1fda60 no longer exists on the remote (checked all refs) — implementing from this issue's spec instead. The estimator reference (src/image-tokens.ts) lives on billion-context master and is the porting source.
    • #1097 (content store / acp_retrieve) is not merged in billion-context master yet, so the kernel ships contracts only (store entry-kind types, tool schemas, sticky state machine) with zero file I/O — matching "kernel first, integrate later".

    Open questions — my recommendations (veto at PR review if you disagree):

    1. Placement → arrival-time substitution. Downsample exactly once at the #1097 substitution point; later forwards just carry the shrunk bytes. Re-evaluating per forward risks divergence from the already-cached prefix, and the original bytes are only reliably available at arrival anyway. Arrival-time gives determinism + prefix-cache stability.
    2. v1 heuristic params → keep the heuristic (aspect band long/short ∈ [1.4, 2.6], short side ≥ 720px). Size-only gating would downsample photos/charts/diagrams, contradicting the approved "non-screenshot originals pass through untouched" behavior. Known limitation: 3:2 photos (ratio 1.5) fall inside the band and will false-positive in v1 — acceptable until a trained router slots into the classifier interface. All params are exposed via config/classifier options.
    3. Estimator location → move the canonical copy into the kernel. The math (header-only dimension parse, OpenAI tile model, fallbacks) is load-bearing and host-agnostic; billion-context keeps its body-scanning wrapper and delegates per-image estimation to the kernel. Two divergent copies of the billing model would be worse than one canonical one.

    PR scope: new module src/image-compress.ts (+ tests), config types/defaults/validation (3-level merge like absorb), state fields + stats counters, image_full tool contract (3 wire shapes, absorb-style failure markers), byte-stable system-note builder. Fully additive, default OFF ⇒ byte-identical pass-through when disabled; zero change to existing behavior.

    Follow-up with the PR link coming.

    中文摘要:已按你的裁定开始内核侧实现;三个开放问题给出明确建议(到达时一次性替换、保留 v1 启发式参数、估算器规范副本移入内核),PR 评审时可否决。

  2. ranxianglei commented on Sep 22, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 完成 — PR: #357 (Closes #353)

    Kernel-side half landed as a new pure module src/image-compress.ts (+ barrel exports):

    • Routing: decideImageRoute(meta, config, classifier?) — pure, deterministic decision fn (disabled → not-an-image → below-min-tokens → not-screenshot-like → downsample w/ recipe). Pluggable ScreenshotClassifier; v1 heuristic = aspect band [1.4, 2.6] + short-side floor 720px; trained router slots in later without pipeline changes.
    • Estimator: billing-aware token estimate ported from billion-context image-tokens.ts — header-only dim parsing (PNG/GIF/BMP/WebP VP8X|VP8L|VP8/JPEG SOF scan, budgeted base64 head) + OpenAI high-detail tile model (short side up to 768, long capped 2048, 85+170·tiles); unknown dims → 16384 fallback; bytes billing → ceil(b64len/4).
    • image_full channel: tool contract ×3 wire shapes (IMAGE_FULL_TOOL/_OPENAI/_RESPONSES, args {ref}), applyImageFull() with success/idempotent/[image_full FAILED: ...] texts, sticky per-session restored-ref set, byte-stable buildImageFullSystemNote(count), shrink records + cumulative stats (imagesShrunk/imageBytesSaved/imageTokensSaved), resetImageFullState() (hosts call on compaction reset/fork recovery/session restart), ImageStoreEntry {kind:"image",...} contract for the pending content store (#1097).
    • Config: imageCompression?: {enabled:false, minTokens:512, maxDimension:1280, quality:80, format:"webp"} — defaults/validation/sub-field merge exactly like absorb; host-side 3-level merge unchanged.

    Open questions answered in the PR description: (1) arrival-time substitution (determinism + prefix-cache stability); (2) kept the heuristic band+floor per the owner-approved spec — non-screenshot originals pass through untouched; known 3:2-photo false-positive documented until a trained router lands; (3) canonical estimator moves to the kernel, hosts keep their body-scanning wrapper.

    Behavior-change disclosure: none user-visible. Feature OFF by default ⇒ byte-identical pass-through (regression-tested battery under enabled:false). Additive exports only. Fresh states now seed 3 zero stats counters + 2 empty arrays carried through mergeCompressionState; old persisted states untouched (all readers ??-guarded); one pre-existing test literal updated to match.

    Verification: npm run typecheck clean · npm test 800/800 · npm run build clean. Host-side work (sharp encode execution, sidecar I/O, wire injection ×4 protocols, logging, packaging) stays in billion-context for the #1095/#1097 integration.

    中文摘要:实现了图像预压缩的路由/降采样决策与 image_full 恢复通道(纯内核模块,默认关闭、无行为变更),typecheck/800 项测试/build 全绿,可以进入 review。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions