Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions TECHNICAL-NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -475,3 +475,18 @@ Settled design = **stamp-by-default + empirical self-heal (probe-and-remember)**
- Modes: only the plugin lane (registered agent / dsh plugin, internal conversation header present) ever stamps; plain proxy mode stays pure passthrough in both directions, byte-for-byte unchanged.

Tests: `tests/issue2645-pck-probe.test.ts` (strict upstream self-heals turn 1, remembers turn 2, lenient pool never probes, poisoned upstream forwards the 400 and learns nothing, client-owned keys drop after ablation, model-granular pairs) + the rewritten matrix in `tests/issue2218-relay-affinity-signal.test.ts`.

## Uncovered-history audit — honest minimum verdict (#4331)

The "no compressible ranges remain — the context is already at its minimum" stop signal was structurally unverifiable: it fired whenever the PRE-FOLD OFFER was exhausted (`postCompressTail`, #2404 pinning), but nothing ever compared the union of active-block ranges against the session's ref set. A live ZCode session showed the failure at scale — 138 mid-history refs (~65K tok, ~16% of context) in no active block, largest run `m00312`–`m00411` (100 msgs), while the receipt declared the global minimum three times in a row. The kernel DOES find such spans when they are offered (verified by repro on master); what was missing was a GLOBAL coverage audit plus a verdict that only claims "minimum" when one holds.

Design (`src/uncovered.ts`, host-side only, no kernel change):

- `uncoveredHistoryRuns()`: walk the message array for entries outside every active block's coverage (the `collectVisible` predicate), group contiguous runs, then classify each run: skip if any current offer covers part of it (the model can act on it now); otherwise diff TWO `core.processTurn` passes over a `structuredClone(state)` — normal config vs relaxed `{preserveRecentMessages: 0, preserveRecentTokens: 0}` — and treat refs offered only in the relaxed pass as zone-hidden (inside the protected recent window). The empirical zone diff deliberately re-uses the kernel's own protected-zone rules instead of copying them, so it self-maintains as those rules evolve (over-approximation errs toward under-reporting). Runs below `VIABLE_RANGE_MIN_TOKENS` (200) stay silent. A caller that already ran the normal offer pass may pass it via `normalRanges` (array = trust-and-reuse, even empty; undefined = no caller data, audit runs its own pass) so acp_status pays ONE extra `processTurn`, not two. Whole function is advisory: any throw degrades to "no runs found" and never breaks receipts or status.
- **acp_status** gains an additive `UNCOVERED HISTORY` section (top-5 runs by tokens, rest counted) listing mass that sits outside every active block AND is absent from the live Compressible-ranges list above — including isolated sub-gate runs between covered blocks that no merge will ever fold (the permanent-hole class of #4331 / #847). Named for visibility, explicitly not as fold suggestions; rows drill down via `scope:"uncompressed"` unchanged.
- **Tier-2 coverage invariant (reviewed, pinned by P4)**: the audit's covered-set is `coveredMessageIds` = the union of ACTIVE blocks' `effectiveMessageIds` with NO child recursion (kernel/src/state.ts:57-64). That is safe because every block-creation path unions consumed children's `effectiveMessageIds` into the new parent (kernel/src/unit-projection.ts:178-183) and deactivates the children (kernel/src/compress.ts:1578-1581) — a T2/T3 parent therefore carries its full distilled span in the state model, and hydration preserves it (whole state JSON). DB-side walks of `blocks.payload_json` do NOT have that property and need `directBlockIds` recursion (the reason an earlier independent attempt carried one); the state model does not.
- **postCompressTail**: on clean whole-offer exhaustion with no tier distillation, the audit runs before the verdict. Runs found AND totalling ≥ `UNCOVERED_RECEIPT_FLOOR_TOKENS` (10K, receipt-only) → honest text naming them ("not at its minimum", pointer to acp_status). Below the floor — or with no runs — the verbatim #1387 contract string fires byte-stable, exactly as on master: small declined leftovers must not re-nag on every fold (pi #521 rule), while material leaks override it. The floor is deliberately NOT applied to the acp_status section: an explicit inspection surface is not a nag surface, and option 1 of #4331 specified no size gate there. The 10K total is a product value pending owner ack (matches the reviewer's independent pick).

Behavior change (disclosed): old = unconditional "already at its minimum" on offer exhaustion; new = that claim only when the global audit verifies it AND the uncovered mass clears the 10K receipt floor — below the floor the receipt is byte-identical to master (so `tests/compress-success-tail.test.ts` stays at master bytes, zero diff), and its #2404/#1387 sub-floor residue fixtures now double as the floor's pins. Sub-gate residue became visible in acp_status (previously invisible on every surface). No config/env surface added; nudge prompt text untouched (kernel-owned). Known limitation left as a separate observation: the receipt's remaining-calc removes a WHOLE offered batch when any submitted span overlaps it, so a partially-folded merged batch can under-report rather than over-report.

Tests: `tests/issue4331-uncovered-history.test.ts` (hole session → honest verdict + named span; drained session → verbatim contract string; sub-floor silence; tier precedence; **tier-2 distillation coverage pin** — three T1s distilled into one T2 parent via the real host path must not leak child spans into the audit; both acp_status directions; formatter caps) + `tests/compress-success-tail.test.ts` UNCHANGED from master (its < 10K-residue fixtures pin the floor), `tests/issue2029-status-baseline-provenance.test.ts` (every status-reader kernel call feeds the provenance baseline), `tests/issue847-compress-failure-guards.test.ts` (sub-gate residue visible in the diagnostic, still never offered).
28 changes: 28 additions & 0 deletions src/acp-status.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import { pckAblations } from "./pck-probe.js";
import { getUnrecognizedPathStats } from "./server/observability.js";
import { ccrEnabled, ccrLoopConfig, contentStoreOf } from "./store.js";
import { coveredRefSpan } from "./decompress-shared.js";
import { describeUncoveredRuns, shortTokenCount, uncoveredHistoryRuns } from "./uncovered.js";
import { preCompactionArchiveOf, statusInputBaseline, type Session } from "./session.js";
import { compressBreakerDetail, compressLastFailureCause } from "./stream.js";
import { describeAdvisory, getAdvisoryState } from "./advisory.js";
Expand Down Expand Up @@ -242,10 +243,12 @@ export function handleAcpStatus(args: Record<string, unknown>, ctx: AcpStatusCtx
const lastCause = compressLastFailureCause(ctx.session);
const staleRef = lastCause !== undefined && lastCause.startsWith("stale-ref");
const substrateDestroyed = lastCause !== undefined && lastCause.startsWith("substrate-destruction");
let offeredSpans: Array<{ startRef: string; endRef: string }> | undefined;
try {
const live = liveCompressionRanges(ctx);
if (live) {
const { nudge, ranges } = live;
offeredSpans = ranges;
extra.push("");
extra.push(nudge.shouldInject ? `Nudge: ACTIVE — ${nudge.reason}` : `Nudge: idle — ${nudge.reason}`);
// #2366: high billed pressure + exhausted compressible mass is the
Expand Down Expand Up @@ -275,6 +278,31 @@ export function handleAcpStatus(args: Record<string, unknown>, ctx: AcpStatusCtx
} catch {
// Base-only report; never fall back to a stale snapshot.
}
// #4331: uncovered history — contiguous runs outside every active block AND
// outside the live offer above (older than the protected recent window).
// They keep going upstream raw while no surface names them; additive
// section, rendered only when non-trivial. Advisory: any failure skips it.
try {
const runs = uncoveredHistoryRuns({
core: ctx.core,
messages: ctx.messages,
state: ctx.session.state,
config: ccrLoopConfig(ctx.session, ctx.config),
tokenCount: billed,
contentStore: contentStoreOf(ctx.session),
applyView: ctx.compressMessages,
actionableSpans: offeredSpans ?? [],
normalRanges: offeredSpans,
});
if (runs.length > 0) {
const totalMsgs = runs.reduce((s, r) => s + r.count, 0);
const totalTok = runs.reduce((s, r) => s + r.tokens, 0);
extra.push("");
extra.push(`UNCOVERED HISTORY — ${totalMsgs} msg(s) across ${runs.length} run(s), ${shortTokenCount(totalTok)} tok, sit outside every active block and are absent from the Compressible-ranges list above (older than the protected recent window): ${describeUncoveredRuns(runs)}. They keep going upstream raw (runs may sit below the fold-size gate — named for visibility, not as fold suggestions); inspect their rows with scope:"uncompressed".`);
}
} catch {
// Advisory section; never break the status report.
}
// #2362: dead refs — known to the session but no longer backed by any
// visible or folded message (client history rewrite / host-native
// compaction). Failure receipts point the model at acp_status, so this is
Expand Down
32 changes: 32 additions & 0 deletions src/stream.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import { attachSubagentSessions, subagentSessionNote, subagentSessionsOf, syncSu
import { METADATA_DRIFT_ESCALATED } from "./fold-reconcile.js";
import { safePrefix, safeSuffix, scrubLoneSurrogates } from "./text-safe.js";
import { applyConfiguredCompression } from "./external-summary-compress.js";
import { describeUncoveredRuns, shortTokenCount, uncoveredHistoryRuns } from "./uncovered.js";

export type RewriteCtx = {
core: CompressionCore;
Expand Down Expand Up @@ -515,6 +516,10 @@ function droppedEntriesNote(diagnostics: CompressParseDiagnostics): string {
// past the session tail and burn a round on kernel rejection. Wording is
// verbatim pi-side so both hosts speak the same contract.
const NO_RANGES_REMAIN_TEXT = "No compressible ranges remain — the context is already at its minimum; continue the task without compressing.";
// #4331 / pi #521 anti-nag rule: below this TOTAL uncovered mass the receipt
// keeps the legacy wording — small leftovers outside the offer must not re-nag
// on every fold. Material leaks (the #4331 class: tens of K tokens raw) override it.
const UNCOVERED_RECEIPT_FLOOR_TOKENS = 10_000;

// #2404: the tail's "remaining ranges" is the LAST NUDGE's offer (the pre-fold
// set the model acted on) minus the just-folded spans — NOT a fresh post-fold
Expand Down Expand Up @@ -605,6 +610,33 @@ function postCompressTail(ctx: RewriteCtx, cleanSuccess: boolean, submitted: Arr
// Partial failures stay silent too: the model still owes the errors an answer
// before any "you are done" verdict (pi #521 gate).
if (!cleanSuccess || tier !== null) return "";
// #4331: offer exhaustion is NOT proof the context is at its minimum — the
// verdict only sees this call's pre-fold offer, and mid-history runs can sit
// outside every active block AND outside the offer (disjoint folds, dead-ref
// holes, integrity withdrawals). Audit before making the global claim; the
// verbatim contract string (#1387) stays intact for genuinely-drained
// sessions. Advisory only: audit failure degrades to the old wording path.
try {
const runs = uncoveredHistoryRuns({
core: ctx.core,
messages: ctx.messages,
state: ctx.session.state,
config: ccrEnabled(ctx.session) ? ctx.config : { ...ctx.config, ccr: undefined },
tokenCount: statusInputBaseline(ctx.session),
contentStore: contentStoreOf(ctx.session),
applyView: ctx.compressMessages,
actionableSpans: remaining,
});
if (runs.length > 0) {
const totalMsgs = runs.reduce((s, r) => s + r.count, 0);
const totalTok = runs.reduce((s, r) => s + r.tokens, 0);
if (totalTok >= UNCOVERED_RECEIPT_FLOOR_TOKENS) {
return `\n\nUncovered history remains — all offered ranges are folded, but ${totalMsgs} msg(s) across ${runs.length} run(s), ${shortTokenCount(totalTok)} tok, sit outside every active block and are not in this call's offer: ${describeUncoveredRuns(runs)}. The context is not at its minimum (runs may sit below the fold-size gate — named for visibility, not as fold suggestions); inspect with acp_status(scope:"uncompressed").`;
}
}
} catch {
// Fall through to the contract wording below.
}
return `\n\n${NO_RANGES_REMAIN_TEXT}`;
}

Expand Down
187 changes: 187 additions & 0 deletions src/uncovered.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
// Uncovered-history audit (#4331). The compress stop verdict and the
// acp_status overview both derive from the per-call OFFER (what this request's
// prepare pass advertised), so mid-history spans that sit outside every active
// block AND outside the current offer are invisible to both surfaces — the
// receipt can then claim "the context is already at its minimum" while tens of
// thousands of raw tokens keep going upstream. This module answers the one
// question neither surface could ask before: which contiguous runs of messages
// are uncovered by any active block, older than the protected recent window,
// and not in the actionable offer right now?
//
// Zone exclusion is computed EMPIRICALLY as the diff between two kernel offers
// (normal config vs preserve-0 config) instead of re-implementing
// computeProtectedRefs host-side: the kernel's zone rules (#2886 pair symmetry,
// neverPreserveRecentTools, …) evolve without host notice, and an empirical
// diff self-maintains. Both passes run on a structuredClone of the session
// state — processTurn is not idempotent in its nudge cadence stamps
// (nudge-inject rides lastNudgeShownTokens down and sets it on shouldInject),
// so the audit must never touch live state. The whole audit is advisory: any
// failure degrades to "no runs" rather than breaking receipts or status.

import {
BLOCKED_REF,
VIABLE_RANGE_MIN_TOKENS,
coveredMessageIds,
countMessageTokens,
defaultCountTokens,
refForRaw,
type CompressionCore,
type CompressionState,
type Config,
type CoreMessage,
type MessageContentStore,
} from "acp-kernel";

interface UncoveredRun {
startRef: string;
endRef: string;
count: number;
tokens: number;
}

interface UncoveredAuditInput {
core: CompressionCore;
/** Full resent log (apply class) — same array class prepare runs on. */
messages: CoreMessage[];
state: CompressionState;
config: Config;
/** statusInputBaseline — the token level the live-recompute lanes use. */
tokenCount?: number;
contentStore?: MessageContentStore;
applyView?: CoreMessage[];
/** Spans the model can act on right now (post-filter offer / remaining list).
* Runs overlapping these are NOT reported — they are surfaced elsewhere. */
actionableSpans?: readonly { startRef: string; endRef: string }[];
/** A caller that already ran the normal-config offer pass may pass its
* ranges here so the audit skips re-running it (acp_status reuses its live
* Compressible-ranges pass, cutting the audit from two processTurn calls
* to one on that surface). UNDEFINED means "no caller data" — the audit
* runs its own normal pass; an array (even empty) means "trust this". */
normalRanges?: readonly { startRef: string; endRef: string }[];
}

interface RunEntry {
index: number;
ref: string;
num: number;
tokens: number;
}

function refNum(ref: string): number {
const n = Number(ref.replace(/\D/g, ""));
return Number.isFinite(n) ? n : 0;
}

type SpanWindow = readonly [number, number];

function windowsOf(spans: readonly { startRef: string; endRef: string }[]): SpanWindow[] {
const out: SpanWindow[] = [];
for (const s of spans) {
const a = refNum(s.startRef);
const b = refNum(s.endRef);
if (a > 0 && b > 0) out.push(a <= b ? [a, b] : [b, a]);
}
return out;
}

function inAnyWindow(n: number, windows: readonly SpanWindow[]): boolean {
return windows.some(([lo, hi]) => n >= lo && n <= hi);
}

export function uncoveredHistoryRuns(input: UncoveredAuditInput): UncoveredRun[] {
const { core, messages, state, config } = input;

// 1) Uncovered entries — the same predicate report.collectVisible uses:
// numbered, not folded into an active block, carrying tokens.
const covered = coveredMessageIds(state);
const entries: RunEntry[] = [];
messages.forEach((msg, index) => {
const ref = refForRaw(state.messageRefs, msg.id);
if (!ref || ref === BLOCKED_REF) return;
if (covered.has(msg.id)) return;
const tokens = countMessageTokens(msg, defaultCountTokens);
if (tokens <= 0) return;
const num = refNum(ref);
if (num <= 0) return;
entries.push({ index, ref, num, tokens });
});
if (entries.length === 0) return [];

// 2) Contiguous runs: array adjacency defines "contiguous" (what the wire
// actually carries next to itself); covered or zero-token neighbors break
// a run conservatively.
const runs: RunEntry[][] = [];
for (const e of entries) {
const last = runs[runs.length - 1];
if (last && last[last.length - 1].index === e.index - 1) last.push(e);
else runs.push([e]);
}

// 3) Empirical zone diff. Cloned state: see header note on cadence stamps.
const normalWindows = input.normalRanges ? windowsOf(input.normalRanges) : [];
let relaxedWindows: SpanWindow[] = [];
try {
const clone = structuredClone(state);
const base = {
messages,
tokenCount: input.tokenCount ?? 0,
renderTags: "none" as const,
contentStore: input.contentStore,
applyView: input.applyView,
};
if (!input.normalRanges) {
normalWindows.push(...windowsOf(
core.processTurn({ ...base, state: clone, config }).nudge?.compressibleRanges ?? [],
));
}
relaxedWindows = windowsOf(
core
.processTurn({
...base,
state: clone,
config: { ...config, preserveRecentMessages: 0, preserveRecentTokens: 0 },
})
.nudge?.compressibleRanges ?? [],
);
} catch {
return [];
}

const actionable = windowsOf(input.actionableSpans ?? []);

// 4) Per-run classification. A ref is zone-hidden when the relaxed pass
// reaches it but the normal pass does not — i.e. only the protected
// recent window keeps it off the offer. Refs in NEITHER window are
// structurally unoffered (dead-ref holes, checkpoint carriers, integrity
// withdrawals, …) and stay reported: that is exactly the mass the user
// cannot fold through ordinary ranges and needs to see. Over-approximate
// window membership errs toward under-reporting, never toward claiming a
// protected tail message is leakable mass.
const out: UncoveredRun[] = [];
for (const run of runs) {
if (run.some((e) => inAnyWindow(e.num, actionable))) continue;
const kept = run.filter((e) => !(inAnyWindow(e.num, relaxedWindows) && !inAnyWindow(e.num, normalWindows)));
if (kept.length === 0) continue;
const tokens = kept.reduce((sum, e) => sum + e.tokens, 0);
if (tokens < VIABLE_RANGE_MIN_TOKENS) continue;
out.push({
startRef: kept[0].ref,
endRef: kept[kept.length - 1].ref,
count: kept.length,
tokens,
});
}
return out;
}

export function shortTokenCount(n: number): string {
if (n < 1000) return `${n}`;
const k = n / 1000;
return `~${k >= 10 ? Math.round(k) : Math.round(k * 10) / 10}K`;
}

export function describeUncoveredRuns(runs: UncoveredRun[], maxListed = 5): string {
const listed = runs.slice(0, maxListed).map((r) => `${r.startRef}–${r.endRef} (${r.count} msgs, ${shortTokenCount(r.tokens)} tok)`);
const rest = runs.length - listed.length;
return rest > 0 ? `${listed.join(", ")}, +${rest} more run(s)` : listed.join(", ");
}
Loading
Loading