Skip to content

feat: tool-result valuator + deterministic crush plugin interface (two-tier absorb gate) - #359

Merged
ranxianglei merged 2 commits into
masterfrom
2026-09-23_tool-result-crush-gate
Sep 23, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-09-23_tool-result-crush-gate

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #354

Implements the kernel side of the two-tier absorb gate per the ruling on billion-context#1094: valuation in the kernel, compression as an open plugin interface with kernel defaults, kernel first. Phases 1+2 of the issue; phase 3 (retrieve-time application) waits on billion-context#1097's store and reuses the same registry as a second call site.

What lands

Valuator (evaluateToolResult, pure + deterministic): for each absorb-candidate tool result under context pressure, decides skip (below absorb.minToolTokens or under absorb.contextThresholdPct) / crushed (deterministic compression brought it under threshold — no model round-trip) / distill (still over, or uncrushable — forwarded to the [ACP absorb] prompt path, carrying the crushed payload when crushing succeeded). Reuses defaultCountTokens + the absorb gates; adds a minReduction floor (default 0.1). Same payload + config ⇒ same bytes (prefix-cache friendly).

Plugin interface: narrow sync contract (text, meta) => { text, lossy, stats? } | null; null = pass-through. Ordered registry (registerCrushPlugin / unregisterCrushPlugin / listCrushPlugins / resetCrushPlugins), hosts can add/replace/disable; per-strategy toggles + excludeTools via config. Kernel-enforced guarantees: dispatch guards keep json/code/log payloads on their own strategies; any throw, sub-minReduction result, or invariant violation fails open to the next candidate or pass-through.

Built-in strategies:

  • json-fold (lossless): constant-field hoisting + identical row/run collapse into annotated envelopes, decodable byte-for-byte.
  • code-trim (lossy): comment/docstring elision behind a string-literal-aware scanner; template literals and assigned triple-quoted strings survive verbatim; unterminated constructs fail open.
  • log-select (lossy): level-classified line selection; warnings deduped; runtime stack frames collapsed; honest [N lines omitted] footer instead of opaque retrieval hashes.

Pipeline: new crush node between absorb-hide and absorb-prompt, so the absorb-prompt decision runs on post-crush sizes atomically in the same turn (a stale [ACP absorb] marker is stripped and re-decided). View-only per-turn recomputation — no persisted-state mutation; effects reported via NodeEffects only.

Config: new top-level config.crush block { enabled, minReduction, strategies? } resolved + validated through the existing path (defaultConfig, validateConfig). No wire-shape change beyond tool-result text itself; zero new runtime dependencies.

Behavior changes (old → new)

  1. Feature DEFAULT OFF (crush.enabled: false, and requires absorb.enabled: true). With it off, pipeline output is byte-identical to before — verified by test. No host behavior changes until explicitly opted in.
  2. New pipeline node crush between absorb-hide and absorb-prompt (inert unless both switches are on); canonical node list grows from 11 to 12 (pipeline order test updated accordingly).
  3. Deviations from the reference prototype (billion-context PR #1098 branch, still unmerged — nothing shipped depends on its names):
    • log error budget LOG_MAX_ERRORS: 10 → 20, plus a strict invariant: every distinct ERROR/FAIL line must survive verbatim in lossy log output or the result is rejected (fail-open to pass-through). The prototype could silently drop middle errors beyond its budget of 10; the kernel never will. Logs with >20 distinct errors fail open rather than drop any.
    • Annotation vocabulary: __pre_crush / [pre-crush: …] → __acp_crush / [acp-crush: …] (kernel-owned naming).
    • Config lives at top-level config.crush (block-shaped like absorb), not nested under absorb.preCrush as in the host-side prototype.

Verification

  • npm run typecheck clean; npm test 807/807 (was 766; +41 new tests in tests/crush.test.ts covering lossless round-trips, determinism, dispatch guards, fail-open paths, the error-line invariant, registry semantics, all three valuator decisions, per-strategy disable/excludeTools, stale-marker stripping, and processTurn integration incl. byte-identical-when-off); npm run build clean.
  • Touched files Prettier-formatted; format:check repo-wide failure is pre-existing on master (untouched files included) and not gated by CI here.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到 PR #359(工具结果 valuator + deterministic crush 插件接口,two-tier absorb gate)。已拉取分支,正在本地核对:diff 整洁度、typecheck/test/build、既有行为回归审查。结论稍后回复。

@ranxianglei
ranxianglei force-pushed the 2026-09-23_tool-result-crush-gate branch from 1d92a7f to b8c17c2 Compare September 22, 2026 18:27
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-23_tool-result-crush-gate (c6dd2bf)

Option A — Install from npm PR tag (recommended)

In your adapter project:

npm install acp-kernel@pr-359

Each push to this PR publishes a new version under the pr-359 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf acp-kernel-pr359.tgz
npm install ./package

This comment is automatically updated on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review complete — PR #359 (tool-result valuator + deterministic crush plugin interface)

TL;DR: Rebased onto current master (v0.0.82), fixed two blocking correctness bugs found in review plus hardcoded plugin fail-open paths, everything green. Mergeable from my side — merge is yours to do.

What I did

  1. Rebased the stale base. The PR sat on v0.0.73 (8ee1801); master had moved to v0.0.82 (b30723f) across ~9 releases (acp_rule rules feat(rules): acp_rule persistent rules — reconciled API (state, limits, hard protection) #332, checkpoint-carrier feat(ci): one-click release workflow #334/applyCompression: isSummaryMessageId 只认 acp_summary_* 前缀,宿主携带的 checkpoint 会被 plain range 当普通消息折叠 #335, terminal-escape, google wire format, protected-latest-tools feat: protectedLatestTools — protect only the latest instance of a tool #223, line-form-split-summary fix(parse): coalesce split line-form entries — refs header + summary as sibling array elements (#1001 problem 3) #330, …). GitHub reported conflicts in src/config.ts, src/index.ts, src/types.ts. All three were purely additive (both sides inserting different blocks at the same spot) — resolved as union, nothing semantically lost. Pushed b8c17c2, CI green.
  2. Full regression audit of the integration diff (table below).
  3. Fixed review findings directly on the branch → commit c6dd2bf (fast-forward; +190/−9 across src/crush.ts, tests/crush.test.ts, README.md). CI on the new head: test ×2, pr-validation, build-artifact — all success. Local pre-flight: typecheck clean, 915/915 tests (910 baseline + 5 new regression tests), build clean.

Bugs found & fixed in review (commit c6dd2bf)

B1 — json-fold silently dropped sparse-null fields (data loss). In buildConstHoist, a key absent in row 0 but null in all other rows passed the const check because canonOf(undefined) === canonOf(null); the hoisted value then became undefined, which JSON.stringify drops — the field vanished from every decoded row, violating the losslessness contract the feature advertises. Fix: skip keys not present in row 0 (if (!(k in first)) continue;).
Behavior change (disclosed): old — such fields silently lost on decode; new — never hoisted, decode stays value-lossless. Side effect: JSON with no other foldable constant may now pass through uncrushed instead of being crushed-and-corrupted (fail-safe direction).

B2 — code-trim misparsed JS regex literals as comment openers. The scanner treated any /* outside strings as a block-comment start. var re = /[/*]/; opened a "comment" that closed at the next */ — eliding real code lines (reproduced: 4 real lines elided from an 11-line function). Fix: added a regex-literal scanner (scanRegexLiteral + regexAllowedAfter/prevSigChar context checks); a / followed by anything other than * is consumed as a whole literal when context allows. Key invariant relied on: a regex pattern cannot start with * (SyntaxError), so /* is always a comment — full-line comment elision (the strategy's main win) is untouched.
Behavior change (disclosed): old — possible elision of real code after a regex literal containing /*; new — literals consumed whole; inputs whose only trimmable content lay inside such a phantom comment may now pass through.

B3/B4 — plugin contract violations could escape the fail-open design. crushText read plugin.kinds.includes(kind) outside the try (a malformed plugin threw out of the pipeline) and accepted non-string res.text (poisoning msg.text downstream). Both now guarded inside the try — bad plugins are skipped, built-ins still apply.
Behavior change (disclosed): old — TypeError / non-string payload; new — safe skip.

B5 — README overclaim. "decodable byte-for-byte" → "lossless — decodes back to an equal JSON document": json-fold is value-level lossless (key order normalizes under hoisting), not byte-level.

Every fix carries regression tests (5 new in tests/crush.test.ts), including a guard that full-line comment elision still works (protects against over-correcting B2).

Existing-behavior audit (per review spec)

Touched path Existing behavior Verdict
Pipeline shape (src/compress.ts) canonical node sequence Intentional change (disclosed): new crush node between absorb-hide and absorb-prompt; canonical list updated in tests/pipeline.test.ts (11→12). Gated off by default. This is §7.3 contract #8 territory → human-gated merge regardless.
Default run (crush.enabled: false) byte-identical output Preserved — PR's own off-by-default test + full suite green.
src/config.ts validation rules validation (from master) Preserved verbatim; crush validation purely additive.
src/index.ts exports rules exports (from master) Preserved; crush exports additive.
src/types.ts TerminalEscapeSignal / RuleRecord / RulesConfig Preserved; Config gains optional crush? (back-compat).
Absorb gates minToolTokens / contextThresholdPct semantics Untouched — crush re-evaluates in-view only, no persisted-state mutation (NodeEffects only).
Wire format / nudge invariants / protected-tool filtering — No diff in those paths.
Docs AGENTS.md module map, README pipeline section Accurate after the B5 fix. Nit: AGENTS.md §7.3(8) still says "the canonical 9-node sequence" — already stale pre-PR (actual count was 11), now 12.
Formatting format:check Pre-existing repo-wide failure (AGENTS.md/README.md fail on plain master too); PR-touched source files pass locally.

Diff cleanliness: every hunk maps to the stated scope (valuator, plugin interface, 3 built-in strategies, node wiring, config, exports, docs, tests). No generated-file churn, no mass whitespace/reformatting.

Unfixed nits (follow-up candidates, none blocking)

  • Unreachable triple-mid branch in findCommentOrTriple (the quote toggle consumes the first quote first).
  • normalizeForDedupe: the \d+→N pass runs before 0x…→ADDR, making the latter mostly dead.
  • evaluateToolResult ignores crush.enabled (direct-API footgun; doc line candidate).
  • Missing edge tests: exact minReduction boundary, unicode / empty-array round-trips, >2M cap bail-out, template-literal bail-out.
  • Duplicate-id plugin registration silently shadows built-ins — consider rejecting or warning.
  • Module-global registry is shared across cores in one process.
  • Lossy strategies normalize CRLF→LF (acceptable for log/code trimming, worth a doc note).
  • Scalar JSON / bash scripts fall into the log strategy by kind fallback.

Verdict

Two blocking correctness bugs found and fixed on the branch, base brought current, CI green on c6dd2bf. This is a feat that adds a pipeline node (§7.3 #8), so it stays human-gated either way — but from my side it is mergeable. Per repo rules I can't merge PRs; please merge yourself: #359

中文摘要: 审查了 PR #359(crush 估值器 + 插件接口):发现并直接修复了两个阻塞级正确性 bug(json-fold 常量提升会静默丢失「首行缺失但其余行为 null」的字段、code-trim 把含 /* 的 JS 正则字面量误判为注释开头导致真实代码行被裁掉),外加插件契约违规的 fail-open 加固和 README 措辞修正,并把分支 rebase 到最新 master(v0.0.82,冲突均为纯增量、按并集解决);本地 915/915 测试通过、新 head CI 全绿,功能默认关闭且关闭时输出与改动前逐字节一致——可以合并。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: tool-result valuator + deterministic compression plugin interface (two-tier absorb gate in kernel)

1 participant