Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions docs/message-identity.md
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,43 @@ migration, applies to old and new sessions alike, storage-independent
(legacy JSON, shadow and full modes behave identically; bookkeeping rides
session metadata). The kernel ledger's content-hash contract is untouched.

## The verified mutation class: host-pruner byte-checked re-anchor (#5395)

The honesty boundary above rejects *fuzzy* content joins. One mutation
shape admits EXACT verification, and its absence was the costliest residual
class found in production: a host tool-result pruning plugin splices its
versioned sentinel between preserved head and tail of a long tool result
(`dsh-compaction-tool-result-pruner`, dsh desktop). The middle deletion
breaks every identity channel at once — norm/twin equality, wordbag/simhash
distance, positional fence — and when the host re-mints the tool-call id
across projections (#2396) the id channels starve too: the covered id goes
PERMANENTLY unmatched, re-reported every turn, feeding the #2438/#5223
escalation (reported: 614 ids × 156 turns/day, ~571K tokens/day unfolded).

The pruner-shape pass (`src/fold-reconcile.ts`; runs LAST in
`planReconciliation`, after every other claim pass) claims only on positive
evidence: the resent bytes must literally decompose as head + sentinel +
tail where head/tail are exact prefix/suffix of the message STORED in bili's
unified index, with a non-empty deleted middle (the overlap guard). Two
distinct originals cannot both decompose around the same sentinel into the
same resent bytes, so mis-attribution is structurally excluded — the
"verified class" the boundary above left open. Gates: role + content-class
agreement, strict length shrink against the anchor, k-th backbone order, one
candidate per old id; deliberately NO toolCallId equality (re-minting is
exactly the condition that makes the drift permanent). Store unavailable or
the original row GC'd → honest unmatched, never a guess. The sentinel table
is built-in code, not configuration: the strings are versioned by their host
plugin, so a new variant lands as a code change with traffic evidence. Each
verified claim lands one `host-content-mutation` conflict-ledger event
(self-limiting: repair mode re-covers the pruned ids, so the next pass sees
them present and the counter falls back to zero) — the second compressor
becomes visible on the banner/stats surfaces instead of staying log-only.
Steady state (no sentinel on the wire) pays zero storage reads.

Proof: `tests/issue5395-pruner-shape.test.ts` (positive gate, every negative
gate per #3617, k-th discipline, cross-pass convergence, warn-mode parity,
degradation, zero-read contract).

## The snapshot render engine: outbound identity moves server-side (#3756)

#3641 classified verdicts when no channel verifies. #3756 removes the
Expand Down
19 changes: 17 additions & 2 deletions src/conflict-watch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import { markDirty, type Session } from "./session.js";
import { isCodexClient } from "./codex-compact.js";
import { isDisplayOnlyConflictDetail, isSiblingConflictDetail } from "./thirdparty-scan.js";

type ConflictKind = "third-party-plugin" | "unannounced-rewrite" | "orphan-reap" | "native-compaction" | "native-compaction-inferred" | "identity-orphan" | "id-churn-adoption";
type ConflictKind = "third-party-plugin" | "unannounced-rewrite" | "orphan-reap" | "native-compaction" | "native-compaction-inferred" | "identity-orphan" | "id-churn-adoption" | "host-content-mutation";

export interface ConflictEvent {
at: number;
Expand Down Expand Up @@ -203,8 +203,14 @@ export function formatConflictSection(events: ConflictEvent[], now: number = Dat
// landing was fully rebased onto (#2372/#2432) — handled, no ongoing risk. Gated to the
// pure case so mixed ledgers keep their existing tier verbatim.
const handledNativeOnly = meaningful.length > 0 && meaningful.every((e) => e.kind === "native-compaction");
// #5395: byte-verified host-side MUTATION (a host tool-result pruner spliced
// its sentinel into folded content; bili re-anchored onto the pruned bytes).
// Own tier: it IS confirmed second-compressor evidence (the header says so),
// but unlike a native-compaction landing bili already REPAIRED it — the
// advice must name the mechanism and the switch, not command a hunt.
const hostMutationPresent = meaningful.some((e) => e.kind === "host-content-mutation");
const activeConfirmed = confirmed.some((e) => now - e.at <= CONFLICT_ACTIVE_WINDOW_MS);
lines.push(`COMPRESSION CONFLICTS — ${events.length} event(s) in this session (${active.length} active · ${historical.length} historical; active = within ${CONFLICT_ACTIVE_WINDOW_MS / 86_400_000} days). ${foreignConfirmed || nativePresent
lines.push(`COMPRESSION CONFLICTS — ${events.length} event(s) in this session (${active.length} active · ${historical.length} historical; active = within ${CONFLICT_ACTIVE_WINDOW_MS / 86_400_000} days). ${foreignConfirmed || nativePresent || hostMutationPresent
? "Two compressors on one conversation (bili + another compression plugin — third-party or bili's own sibling — or client native compaction) double-compress and corrupt message refs:"
: "Diagnostic evidence that something outside bili touched this conversation — how strong that evidence is is tiered below:"}`);
for (const e of events.slice(-10)) {
Expand Down Expand Up @@ -247,6 +253,15 @@ export function formatConflictSection(events: ConflictEvent[], now: number = Dat
advice = active.length > 0
? "The events above are client-native compaction landings bili ALREADY rebased onto (#2372/#2432): each detected checkpoint rebuilt the fold base, so there is no ongoing double-compression risk and nothing to disable. If you did not expect these sessions to auto-compact, review the client's own auto-compaction setting; otherwise clear this ledger — Web UI conflict banner / session page, or POST /__bili/conflicts/clear?session=<id>."
: "All events above are client-native compaction landings older than 7 days (historical stock) that bili already rebased onto (#2372/#2432): handled at the time, no ongoing risk. Clear this ledger — Web UI conflict banner / session page, or POST /__bili/conflicts/clear?session=<id>.";
} else if (hostMutationPresent) {
// #5395: already-repaired mutation — say so plainly (the conversation
// stayed coherent; bili re-anchored automatically) and name the switch
// that stops recurrence. Gated to ledgers carrying at least one such
// event; mixed ledgers keep their existing tier verbatim (first match
// wins, same as every branch above).
advice = active.length > 0
? "The events above are byte-verified host-side mutations: a host compression plugin (tool-result pruner) rewrote content bili had folded, and bili re-anchored its fold state onto the rewritten bytes automatically — the conversation stays coherent, but two compressors ARE touching this session. To stop the recurrence disable the host's tool-result pruning/compaction plugins (keep exactly one compressor per conversation); otherwise clear this ledger."
: "All events above are byte-verified host-side mutations older than 7 days (historical stock) that bili already re-anchored onto automatically: no ongoing corruption, but the host's tool-result pruning/compaction plugins were active alongside bili. If you did not expect them, disable them; otherwise clear this ledger.";
} else if (orphanOnly) {
// A headerless replay does not prove that the original client moved.
// Older ledgers may contain tail-only matches; neither proves state loss.
Expand Down
36 changes: 36 additions & 0 deletions src/fold-decomposition-lookup.ts
Original file line number Diff line number Diff line change
Expand Up @@ -142,3 +142,39 @@ export function resolveStoredDecompositions(
ordered.sort((a, b) => a.seq - b.seq);
return ordered.map((o) => o.m);
}

/** #5395: stored originals for caller-chosen ids (pruner-shape byte
* verification) — unlike resolveStoredDecompositions' narrow #3301 backfill
* class, the caller decides WHICH ids to pull (its cheap marker pre-filter
* already ran). Same row discipline: latest seq per id wins; unparseable rows
* are auxiliary-evidence misses, not errors. undefined = index unavailable or
* no rows → the pruner-shape channel degrades to honest unmatched (absence is
* stable for folded-out history). */
export function storedMessagesById(sessionId: string, msgIds: readonly string[]): Map<string, CoreMessage> | undefined {
let rows: StoredRow[] | null;
try {
rows = getStore().unifiedMessageRows(sessionId, msgIds);
} catch (err) {
loggerLog("warn", `[fold-reconcile] pruner-shape stored lookup failed: ${String(err)}`);
return undefined;
}
if (rows === null || rows.length === 0) return undefined;
const latestSeq = new Map<string, number>();
const out = new Map<string, CoreMessage>();
for (const row of rows) {
let parsed: unknown;
try {
parsed = JSON.parse(row.content);
} catch {
continue;
}
if (typeof parsed !== "object" || parsed === null) continue;
const m = parsed as CoreMessage;
if (typeof m.id !== "string" || typeof m.role !== "string") continue;
const prev = latestSeq.get(row.msgId);
if (prev !== undefined && prev >= row.seq) continue;
latestSeq.set(row.msgId, row.seq);
out.set(m.id, m);
}
return out.size > 0 ? out : undefined;
}
Loading
Loading