Skip to content

feat: content-addressed message store + acp_retrieve (built-in CCR) - #1099

Merged
ranxianglei merged 5 commits into
masterfrom
wt-1097-ework-daemon
Sep 23, 2026
Merged

ranxianglei merged 5 commits into
masterfrom
wt-1097-ework-daemon

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Sep 21, 2026 •

Copy link
Copy Markdown
Owner

What

v1 of the content-addressed message store (built-in CCR): oversized tool results become ID-referenced at arrival instead of force-distilled. The wire keeps a byte-stable placeholder (📦 [stored #m00423 · shell · 4213 tok] … → acp_retrieve("m00423")) and the original goes into a per-session content store, retrievable on demand. Lossless by default — a retrieve not made costs one cheap tool call; a detail distilled away is gone for good.

Design

  • File-backed lazy load (per the thread's efficiency refinement): the index is metadata-only and persists in the session JSON alongside blockContents; payloads live in hash-keyed sidecar files under storeDir() (<sha256(id)>/<sha1(text)>.txt), deduped by content hash, lazy-read on retrieve. Sidecars are codec-aware — AES-256-GCM parity with the session file when BILI_ENCRYPTION_KEY is set.
  • Substitution only at arrival, deterministic (no model call), byte-stable → prefix cache stays warm.
  • Tool-pair integrity: only the content inside a tool result shrinks; pairing with the assistant tool_calls is untouched (covered by tests).
  • Retrieve is ephemeral: fetched content rides the intra-request tool-result channel; it never enters fold space and consumes no message ref.
  • Config-gated compress.store.{enabled,minTokens,maxStoreBytes}, three-level merge, default off.
  • Stats: retrieve-rate + wire byte-savings surface in acp_status; each retrieve logs a [store] retrieve … line.
  • Proxy-mode-only v1 gate: plugin mode would need acp_retrieve registered in the agent's plugin manifest before a placeholder can be emitted (otherwise silent loss) — deferred.

Out of scope (tracked separately, per the issue's phasing)

Pre-flight

  • npm run typecheck — clean
  • npm test — 2168 pass / 0 fail / 2 skipped (6 new in tests/store.test.ts)
  • npm run build — tsup success
  • E2E (tests/e2e/e2e-codex.test.ts): not runnable in the agent sandbox (local upstream 127.0.0.1:8199 unreachable — preflight confirmed codex CLI 0.147.0 + dist present). The change is dormant by default (compress.store.enabled=false ⇒ no-op wire path), so the default path is byte-identical to master and covered by the unit suite; live E2E is available via manual CI dispatch if you want it before merge.

Addresses #1097 (v1).

@github-actions

github-actions Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: wt-1097-ework-daemon (1e097d0)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-1099

Each push to this PR publishes a new version under the pr-1099 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1099.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Reviewing PR #1099 (wt-1097-ework-daemon, base master): pulling the branch, re-running typecheck/test/build on the head, checking CI status, and reviewing the diff for cleanliness + both compression modes. Will report findings here shortly.

ranxianglei pushed a commit that referenced this pull request Sep 21, 2026
PR #1099 review: on the responses text/marker protocol (route compressProtocol 'marker' or ACP_COMPRESS_PROTOCOL=text) only the readonly ACP tools are declared and the text-trigger parser knows no acp_retrieve trigger; ACP_NO_INJECT_TOOL disables all injection on that wire. applyStoreView still emitted placeholders there — unretrievable by construction, i.e. the silent loss #1097 explicitly guards against everywhere else. Gate the policy stamping site (single source for all four prepare fns + plugin-tool path + acp_status) on channel availability, mirroring how absorb/rules strip themselves in text mode. Adds tests/store-marker-gate.test.ts (marker wire keeps the original verbatim; native-tool wire emits placeholder + declaration — fails on unfixed code). Also documents compress.store.{enabled,minTokens,maxStoreBytes} + BILI_STORE_DIR in CONFIGURATION/README (en+zh) and normalizes a mis-indent introduced at the stamping site.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

Review complete. Pre-flight verified locally at the PR head (typecheck clean, npm test 2168 pass / 0 fail / 2 skipped, tsup build success — matches your claims), and I walked the design through both compression modes plus the kernel id/ref contract. The core mechanics check out: client re-sends original history each turn, substitution is a pure outbound-wire view keyed by raw-id→ref, so it's deterministic/idempotent and fold-safe; payload-file-first ordering means a crash leaves an orphan file but never a dangling placeholder; pairing integrity and byte-stability are covered by tests.

One real bug found — fixed directly on the branch (commit 528c36c). The store was armed on wire paths where acp_retrieve has no channel to be executed, which is exactly the silent-loss trap #1097 guards against everywhere else:

  • Responses text/marker protocol (route config "compressProtocol": "marker" or ACP_COMPRESS_PROTOCOL=text) — operator-configurable, not dev-only. There only the readonly ACP tools are declared (server.ts:2829-2831) and the text-trigger parser knows only compress/acp_status/search_context/decompress (adapter-responses.ts:599-608) — yet applyStoreView still emitted placeholders. absorb/rules strip themselves there for precisely this reason (server.ts:2774-2775); store missed the gate.
  • ACP_NO_INJECT_TOOL=1 on the responses wire disables all injection while the store stayed armed (same trap, lower severity — dev/test escape hatch).

Fix: the policy-stamping site in handle() (single source of truth for all four prepare fns + the plugin-tool execution paths + acp_status) now disarms the store when protocol === "responses" and any of those channel-killers is active — mirroring the absorb/rules pattern. Also added tests/store-marker-gate.test.ts: marker wire keeps an oversized tool result verbatim with no placeholder/declaration; native-tool control wire emits placeholder + acp_retrieve declaration. Verified the marker case fails on the unfixed code and passes with the fix. While in there I normalized a 3-line mis-indent the original commit introduced at the stamping site (server.ts:1200-1202, cosmetic) and filled the docs gap: compress.store.{enabled,minTokens,maxStoreBytes} + BILI_STORE_DIR are now documented in CONFIGURATION.md/.zh-CN.md and README.md/.zh-CN.md (en+zh, mirroring the absorb/rules sections).

Minor findings (report-only, non-blocking):

  • maxStoreBytes sums logical per-ref bytes, so content-hash dedup isn't credited against the cap — the config doc wording "unique stored bytes" overstates it; I documented it as "summed logically per ref" in CONFIGURATION.md.
  • Orphan sidecars/files are never swept (the code comment says "swept later" but nothing sweeps yet). Per-session pool dirs accumulate until session GC — worth pairing with the existing session-GC follow-up ([Feature] Default state file cleanup strategy #1082).
  • Test coverage still missing: the encrypted-sidecar codec path (BILI_ENCRYPTION_KEY) and the maxStoreBytes cap behavior.

Post-fix pre-flight: typecheck clean, npm test 2170 pass / 0 fail / 2 skipped (+2 new), build success. CI is running on 528c36c (build-artifact + ubuntu node24 in progress, rest queued at push time). Merge stays yours.

修复了内容存储在 responses marker/文本协议及 ACP_NO_INJECT_TOOL 下仍发占位符却无 acp_retrieve 通道可执行的静默丢失问题(在策略盖章点按通道可用性解除武装,与 absorb/rules 同模式),补了回归测试(未修复代码上可复现失败)、补齐了 en/zh 文档并顺手修正了缩进;其余为报告性小问题,可以合并(等 CI 绿)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

冲突

ranxianglei pushed a commit that referenced this pull request Sep 23, 2026
PR #1099 review: on the responses text/marker protocol (route compressProtocol 'marker' or ACP_COMPRESS_PROTOCOL=text) only the readonly ACP tools are declared and the text-trigger parser knows no acp_retrieve trigger; ACP_NO_INJECT_TOOL disables all injection on that wire. applyStoreView still emitted placeholders there — unretrievable by construction, i.e. the silent loss #1097 explicitly guards against everywhere else. Gate the policy stamping site (single source for all four prepare fns + plugin-tool path + acp_status) on channel availability, mirroring how absorb/rules strip themselves in text mode. Adds tests/store-marker-gate.test.ts (marker wire keeps the original verbatim; native-tool wire emits placeholder + declaration — fails on unfixed code). Also documents compress.store.{enabled,minTokens,maxStoreBytes} + BILI_STORE_DIR in CONFIGURATION/README (en+zh) and normalizes a mis-indent introduced at the stamping site.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 冲突已解决 — 分支已 rebase 到当前 master (3c34b65a),PR 现在基于最新 master,不再落后。

What conflicted and how it was resolved (master had advanced ~88 commits since the PR's old base: session-GC feature #1082, zcode lanes #1145/#1157/#1164, acp-kernel → 0.0.84 (#1175), #1080 encryption/zstd split):

  • src/server.ts — auto-merged cleanly. My review fix (disarm the store where acp_retrieve has no tool channel: responses marker/text protocol + ACP_NO_INJECT_TOOL, now at src/server.ts:1653) survived intact next to master's new code; verified by re-running the full suite below.
  • README.md / README.zh-CN.md — both sides added paragraphs in the same spot: kept mine (the seventh-tool acp_retrieve paragraph) and master's extended protectedLatestTools paragraph (incl. its new compress.protectedTools full-history counterpart sentence). Union, no line dropped.
  • CONFIGURATION.md / CONFIGURATION.zh-CN.md — env-var table: union of BILI_SESSIONS_DIR → my new BILI_STORE_DIR row → master's four BILI_SESSION_GC* rows → master's newer BILI_ENCRYPTION_KEY text from [Feature] Use zstd compress by default #1080 (encryption & zstd as independent knobs). My side's stale BILI_ENCRYPTION_KEY wording was dropped in favor of master's.
  • All three original commits are preserved in order (feat store → fix marker-gate → feat reversible-absorb); the kernel pin is now master's exact 0.0.84.

Post-rebase pre-flight on head aa30a8c4:

  • npm run typecheck — clean
  • npm test — 2325 pass / 0 fail / 2 skipped (suite grew to 2327 with master's tests)
  • npm run build — tsup success

One process note for transparency: direct git push was failing on a TLS transport error in this environment, so I updated the branch through GitHub's Git Data API instead — each recreated commit's tree was verified byte-identical to the local one before the ref update (that's why all three commit SHAs match the local rebase exactly).

CI is currently running on the new head (zero-token e2e + version-guard already green, remaining test/build jobs in progress). Merge stays with you once checks land.

中文摘要:把 PR 分支 rebase 到了最新 master,冲突均为文档新增段落的叠加(保留双方内容、server.ts 自动合并无损),rebase 后 typecheck/2325 项测试/构建全部通过,可以等 CI 绿了合并。

ework-agent and others added 4 commits September 23, 2026 19:36
…1097)

Oversized tool results (>= minTokens, within the per-session budget) are now
ID-referenced at arrival in proxy mode: the wire keeps a byte-stable,
deterministic placeholder carrying ref/tool/size/head-preview metadata, and the
original goes into a per-session content store (hash-keyed sidecar files, index
persisted alongside blockContents), retrievable via an injected acp_retrieve
tool on all four wires. Lossless by default: a retrieve not made costs one
cheap tool call; a detail distilled away is gone for good.

- store.* config (enabled/minTokens/maxStoreBytes): three-level merge, default off
- file-backed lazy load: index = metadata only in session JSON; payloads in
  storeDir()/<sha256(id)>/<sha1(text)>, deduped by content hash, codec-aware
  (AES-256-GCM parity with session JSON when BILI_ENCRYPTION_KEY is set)
- retrieve-rate + wire byte-savings surfaced in acp_status; per-retrieve log line
- v1 gate is proxy-mode only (plugin mode needs acp_retrieve in its manifest)
- reversible-absorb (the digest half) deferred to a follow-up
PR #1099 review: on the responses text/marker protocol (route compressProtocol 'marker' or ACP_COMPRESS_PROTOCOL=text) only the readonly ACP tools are declared and the text-trigger parser knows no acp_retrieve trigger; ACP_NO_INJECT_TOOL disables all injection on that wire. applyStoreView still emitted placeholders there — unretrievable by construction, i.e. the silent loss #1097 explicitly guards against everywhere else. Gate the policy stamping site (single source for all four prepare fns + plugin-tool path + acp_status) on channel availability, mirroring how absorb/rules strip themselves in text mode. Adds tests/store-marker-gate.test.ts (marker wire keeps the original verbatim; native-tool wire emits placeholder + declaration — fails on unfixed code). Also documents compress.store.{enabled,minTokens,maxStoreBytes} + BILI_STORE_DIR in CONFIGURATION/README (en+zh) and normalizes a mis-indent introduced at the stamping site.
…l retrievable (#1097)

The v1 content-addressed store already keeps oversized tool results lossless by default. This makes the two fidelities explicit so the model can pick gist-or-full: the wire placeholder now states that summarizing is reversible (the original persists under the same ref), and a regression test proves absorb (a model-written digest) and acp_retrieve (the full original) coexist under one ref. Pure composition of the existing store + absorb — no new tool, no model-call path.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 CI 全绿 ✅ — rebase 后的 head aa30a8c4 上 8/8 checks 通过:test (ubuntu 22/24, windows 22/24)、build-artifact、version-guard、real-codex e2e (zero-token)、codex CODEX_HOME overlay。分支已就绪,等你合并。

中文摘要:冲突已解决(rebase 到最新 master,文档叠加、server.ts 自动合并无损),CI 8/8 全绿,可以合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Rework complete — kernel-native CCR (acp-kernel 0.0.84)

Branch rebased onto 2026-09-23_kernel-0.0.84 (merged as #1175); the host-side CCR implementation is replaced by the kernel-native content store + acp_retrieve:

  • processTurn now consumes/returns contentStore (adopted via adoptContentStore everywhere state is adopted; read-only count-tokens sites pass the store in but discard it by doctrine)
  • CCR armed/disarmed per session by marker protocol (host strips kernel ccr config + stamps when the wire has no tool channel) — tests/store-marker-gate.test.ts
  • acp_retrieve built-in handled through the compress-tool path (ack text kernel-owned); placeholder format [acp-stored #mNNNNN …] kernel-owned
  • Envelope persisted via kernel StateStore (clean → no write, dirty → file, reload restores byRef+byHash, rebase reset deletes)
  • Config mapping onto full kernel CcrConfig at applyCompressSettings (absent = off preserved)

Pre-flight: typecheck clean · 2295 tests (2293 pass, 2 skipped E2E gates) · build ok. CI on 41a9c67: all 8 checks green.

Known open items (documented, non-blocking):

  1. storeCoveredOriginals not wired yet — originals folded into a summary are not re-materializable via acp_retrieve; natural follow-up after kernel v2 groundwork
  2. Envelope file grows unboundedly with byHash (full originals); no eviction policy yet (kernel has none either)
  3. Marker-protocol disarm enforced host-side only; gate test pins the two Responses paths

Refs #1097. Full dev report: kernel API findings (injection role system handled by adapters — no #377 exposure; assignRefsNode skips acp_retrieved_ ids) and deviations are recorded in the devlog.

The rework onto acp-kernel 0.0.84 changed the config surface from the v1
host-side compress.store.{minTokens,maxStoreBytes} + BILI_STORE_DIR sidecars
to compress.ccr.{minToolTokens,excludeTools,toolName,maxHeadChars} with a
single .content-store.json envelope next to the session JSON, but the doc
sections still described the removed API — an operator following them would
set keys that silently keep the feature off. Rewrite both CONFIGURATION
sections and both README paragraphs to the implemented surface (kernel
defaults 4000/96, [acp-stored] placeholder format, no eviction yet), and
normalize one mis-indented comment line in server.ts.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Re-reviewed the reworked branch (head 41a9c678, now 1e097d02 after my fix-up commit).

Verdict: sound rework. The kernel-native split is the right shape — host keeps config mapping (compress.ccr.* → kernel CcrConfig at applyCompressSettings), per-session arming/disarming, the injection ride-through, and envelope persistence; kernel owns placeholders, retrieval, and stats. Spot-checked the load-bearing paths:

  • Round-1 silent-loss gate preserved. The stamping site still disarms CCR for plugin mode, ACP_NO_INJECT_TOOL, and responses marker/text-protocol wires, and every processTurn site strips ccr from the loop config when the session isn't armed — a placeholder can never appear without an execution channel. The gate tests pin both Responses paths.
  • Persistence. Payload-first ordering (envelope written before session state on save/flush), clean store writes nothing, emptied store deletes the file (rebase reset), malformed envelope degrades to fresh store (retrieve miss, not crash), same at-rest codec as the session file.
  • Regression checklist (existing behaviors touched by the diff): default-off is byte-identical everywhere (strip + test-pinned); absorb/rules loop-config semantics unchanged by the ternary→spread rewrite; persistence backward-compatible (stats backfilled ?? 0, envelope is a separate file, session JSON shape unchanged, no version bump); compact-forging rollback restores the store on all three failure paths; retrieval ids structurally excluded from ref assignment; tool-pair integrity intact (tested). All pass.

Fixed directly on the branch (commit 1e097d02):

  1. Stale docs. The rework changed the config surface but kept v1-era doc text describing the removed host-side API: key compress.store.{enabled,minTokens,maxStoreBytes} (default minTokens 500, maxStoreBytes 2 MiB), env BILI_STORE_DIR, sidecar files <sha256(sessionId)>/<sha1(text)>.txt, placeholder [stored …]. The implemented surface is compress.ccr.{enabled,minToolTokens,excludeTools,toolName,maxHeadChars} (kernel defaults 4000/96), a single .content-store.json envelope next to the session JSON sharing its at-rest codec, placeholder [acp-stored …]. An operator following the docs would have set keys that silently keep the feature off. Rewrote both CONFIGURATION sections (en/zh) and both README paragraphs (en/zh) to match, including the no-eviction-yet limitation.
  2. One mis-indented comment line in src/server.ts (cosmetic).

Pre-flight on the pushed head: typecheck clean · 2327 pass / 0 fail / 2 skipped · build ok. CI on 1e097d02: fast checks green so far (version-guard, zero-token e2e, MCP overlay, build-artifact), the four OS/node test jobs still running.

Your three open items are all reasonable deferrals — (1) storeCoveredOriginals unwired, (2) unbounded envelope growth (now also visible in the docs as "no eviction policy yet"), (3) marker disarm enforced host-side only with the two Responses paths pinned by tests. Nothing here blocks merge once CI finishes green.

重审了重构后的内核原生 CCR 实现(架构、持久化与回归检查均通过),修复了文档仍描述已删除的 v1 store 配置面——按文档配置会静默失效的问题(改写为真实的 ccr.* 键与信封存储描述,中英文同步),另修一处注释缩进,CI 转绿后可合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 CI update on 1e097d02: all 8 checks green (ubuntu/windows × node 22/24 tests, version-guard, zero-token e2e, MCP overlay, build-artifact). The branch is ready to merge whenever you are — merging itself stays with you per repo policy.

CI 已全部转绿(8/8),分支可以合并,合并操作按仓库规范由你执行。

@ranxianglei
ranxianglei merged commit 03dd9c7 into master Sep 23, 2026
8 checks passed
@ranxianglei
ranxianglei deleted the wt-1097-ework-daemon branch October 15, 2026 05:33
cskwork pushed a commit to cskwork/billion-context that referenced this pull request Oct 21, 2026
…xianglei#1097)

PR ranxianglei#1099 review: on the responses text/marker protocol (route compressProtocol 'marker' or ACP_COMPRESS_PROTOCOL=text) only the readonly ACP tools are declared and the text-trigger parser knows no acp_retrieve trigger; ACP_NO_INJECT_TOOL disables all injection on that wire. applyStoreView still emitted placeholders there — unretrievable by construction, i.e. the silent loss ranxianglei#1097 explicitly guards against everywhere else. Gate the policy stamping site (single source for all four prepare fns + plugin-tool path + acp_status) on channel availability, mirroring how absorb/rules strip themselves in text mode. Adds tests/store-marker-gate.test.ts (marker wire keeps the original verbatim; native-tool wire emits placeholder + declaration — fails on unfixed code). Also documents compress.store.{enabled,minTokens,maxStoreBytes} + BILI_STORE_DIR in CONFIGURATION/README (en+zh) and normalizes a mis-indent introduced at the stamping site.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant