Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,15 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag

## [Unreleased]

### Changed (storage correctness wave 3, #4242)

- **State hydration can no longer drift from `mergeState`** (#4192): `COMPRESSION_STATE_FIELDS` in the new `src/state-hydration.ts` is derived mechanically from `Object.keys(mergeState(createInitialState()))` — the ingest/restore field list is generated from the same source as hydration, so a future field can never be restored-but-never-ingested (the `sweptOrphanCallIds`/`pendingOrphanSweepMinRef` loss class). Pinned by a parity test that fails the moment `createInitialState()` grows a key the list does not know.
- **CAS orphans are deleted the moment their refcount hits zero** (#3987): `releaseContent` now deletes the row on the same statement (`UPDATE … RETURNING` + delete on 0) on both the sqlite and mysql backends. Follows the FK discipline strictly — referencing rows are now always deleted *before* their content is released (five call sites reordered), so a zero-ref row can never be dropped while a surviving row still points at it.
- **Reap-shape sessions keep advancing instead of freezing** (#4216): blockContents that outlive their blocks (the #2695 reap shape) now ride the ingest plan as `sessions.orphans_json` — merged (`∪`, because LazyBlockContents may trim entries from memory) in the same transaction as everything else. The mirror skip, the async-twin skip, the importRecord skip and the #3818 writerOff JSON safety valve are all gone: the shape is representable, so `writerOff`'s "the index can represent everything I write" contract holds without an escape hatch. Cold loads after a reap now serve fresh state (the freeze-at-`requests=390` repro is pinned).
- **Raw wire sidecars join the media CAS** (#4069): `splitMediaContent` now also swaps every `data:<mime>;base64,<payload>` occurrence ≥2 KiB inside `rawOpenaiContent` / `rawOpenaiContentParts` / `rawResponsesItem` / `rawGoogleParts` for `{$media:<hash>}` markers; hydration restores the bytes byte-identically on both backends. A picture used to be stored twice (top-level `imageBase64` + the raw sidecar data URL), capping dedup at ~1.89× — the raw sidecar now references the same CAS row (33.8× on the issue's corpus shape). Pinned: no payload bytes in the stored JSON, byte-identical round-trip, shared-row refcount=2.
- **Full-mode authority over a tampered envelope is now pinned** (#4004): the four legacy-envelope integrity tests (`proxy-persist` ×3, `issue2129` ×1) pin `storage.unified=legacy` — they assert the envelope guard layer, which full mode legitimately supersedes; a new test pins the decided semantics instead (index serves while it holds the record; only when both carriers are gone/tampered does the load reject). These were flaky-by-construction under the #3818 full default.
- test: storage-render goldens regenerated once — the drift is inherited from master (#4198 headroom revert changed the stats/metadata shape inside fold-turn bodies), not from this PR; full==legacy==goldens still holds on every wire.

### dsh 归因正面修:llm/stream 会话身份随请求对象(#4082)

- dsh 的 LlmRuntime 在每次派发上盖章 `options.sessionId`(主循环/压缩/标题生成,重试重发也重新盖章),且把 llm/stream waterfall 文档化为官方拦截点。插件现在把每条流的迭代包进以该 sid 为键的 AsyncLocalStorage:适配器的 fetch 读到的会话身份**跟随请求对象,不再依赖异步链形状**——429 重试链、无 initiator 的 agentless 侧调用(compaction/标题)、已销毁的 initiator 作用域、宿主运行时缺口(#2975/#3918/#4082)全部免疫。
Expand Down
75 changes: 9 additions & 66 deletions src/persist.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,11 @@ import { VERSION } from "./version.js";
import { createStorageCodec, parseEncryptionKey } from "./encrypt.js";
import { PersistEpermAlert } from "./persist-eperm.js";
import { createInitialState, defaultCountTokens, prune, type CompressionState, type CoreMessage, type MessageContentStore } from "acp-kernel";
import { markDirty, scrubTurnSeparatorIds } from "./session.js";
import { markDirty } from "./session.js";
import { mergeState } from "./state-hydration.js";
import { LazyBlockContents, isDroppedMetadata, lazyMetadata } from "./session.js";
import type { Session, BlockContent, BlockView } from "./session.js";
import { orphanContents, resolveStorageMode, UnifiedShadow, type StorageMode } from "./storage/shadow.js";
import { resolveStorageMode, UnifiedShadow, type StorageMode } from "./storage/shadow.js";
import { resolveBackendName } from "./storage/backend.js";
import { toLegacyLike, type LegacySessionLike } from "./storage/ingest.js";
import { sha256Hex, type CheckpointRow } from "./storage/store.js";
Expand Down Expand Up @@ -215,55 +216,6 @@ interface PersistedSession {

type Logger = (level: "info" | "warn" | "error", msg: string) => void;

/** Forward-compat: merge a parsed state with a fresh one so missing fields
* (added in later versions) get sane defaults instead of `undefined`. */
function mergeState(parsed: CompressionState): CompressionState {
const fresh = createInitialState();
return {
// #2627: heal persisted state written before the turn-separator fix —
// those blocks' coverage carries outbound-only acp_turn_sep_* ids that
// no resent history can ever contain; scrubbing at hydration is what
// stops live long sessions from alarming forever after an upgrade.
blocks: (() => { const blocks = parsed.blocks ?? fresh.blocks; scrubTurnSeparatorIds(blocks); return blocks; })(),
messageRefs: parsed.messageRefs ?? fresh.messageRefs,
nudge: { ...fresh.nudge, ...(parsed.nudge ?? {}) },
stats: { ...fresh.stats, ...(parsed.stats ?? {}) },
nextBlockId: parsed.nextBlockId ?? fresh.nextBlockId,
nextRunId: parsed.nextRunId ?? fresh.nextRunId,
tokenSnapshot: parsed.tokenSnapshot ?? fresh.tokenSnapshot,
lastPassIds: parsed.lastPassIds ?? fresh.lastPassIds,
hiddenOrphanRefs: parsed.hiddenOrphanRefs ?? fresh.hiddenOrphanRefs,
// #2362: without this a restart forgets which refs are dead and the
// model re-learns each one by failing a compress call again.
deadRefs: parsed.deadRefs ?? fresh.deadRefs,
terminalStreak: parsed.terminalStreak ?? fresh.terminalStreak,
nextRuleId: parsed.nextRuleId ?? fresh.nextRuleId,
// Without this, a restart re-exposes absorbed tool outputs: state
// resurrects with absorbed=[] and hideAbsorbedMessages has nothing to hide.
absorbed: parsed.absorbed ?? fresh.absorbed,
rules: parsed.rules ?? fresh.rules,
// #3769: fold-gated orphan-sweep bookkeeping must survive hydration —
// dropping sweptOrphanCallIds resurrects already-cleaned pairs after a
// restart (mid-history rewrite = cache break); the pending marker is
// transient, losing it only defers the sweep to the next fold.
sweptOrphanCallIds: parsed.sweptOrphanCallIds ?? fresh.sweptOrphanCallIds,
pendingOrphanSweepMinRef: parsed.pendingOrphanSweepMinRef ?? fresh.pendingOrphanSweepMinRef,
// #3939 ③/#3691: the emergency-truncate latch must survive hydration —
// dropping it re-exposes clipped bytes mid-history after a restart,
// rewriting the still-cached prefix on the first post-restart turn.
emergencyTruncateLatch: parsed.emergencyTruncateLatch ?? fresh.emergencyTruncateLatch,
// #3939 ②/#3805: the render deferral must survive hydration — losing
// it stamps migration tags onto still-cached carriers right after a
// restart (a silent mid-history rewrite with no fold in sight).
renderDeferredRefs: parsed.renderDeferredRefs ?? fresh.renderDeferredRefs,
// #3939 ⑤: the view latch must survive hydration — losing it
// re-derives transform output on still-cached carriers right after a
// restart (silent mid-history rewrite, no fold in sight).
viewLatches: parsed.viewLatches ?? fresh.viewLatches,
viewLatchConfigDigest: parsed.viewLatchConfigDigest ?? fresh.viewLatchConfigDigest,
};
}

/** Extract a short host label from an upstream origin URL, safe for a
* filename. Uses the full hostname (sanitized) rather than guessing the
* registrable domain — a public-suffix-list lookup is overkill, and the full
Expand Down Expand Up @@ -2323,21 +2275,12 @@ export class SessionStore {
// Errors surface through the kernel write chain's own logging.
});
}
// #3818 writerOff safety valve: writerOff's contract is
// "the index can represent everything I write". An
// orphan-contents record (the #2695 reap shape — block
// gone from state.blocks, text kept in blockContents)
// is SKIPPED by the mirror guard, so this tick would
// otherwise persist NOTHING at all (no rows, no file).
// Land the JSON envelope as the durable copy; the record
// stays envelope-served on every later read.
if (this.shadow && !this.shadow.stats.disabled && orphanContents(toLegacyLike(session))) {
this.log("warn", `[persist] unrepresentable record for ${session.id} under writerOff — landing a JSON envelope copy (index cannot carry its blockContents)`);
const build = this.guardedBuild(session);
void this.store.writeNow(session.id, build).then(() => this.stampOwnWrite(session.id)).catch(() => {
// Errors surface through the kernel write chain's own logging.
});
}
// #3818 writerOff safety valve REMOVED (#4216/#4242):
// the orphan/reap shape is now representable — its
// contents ride sessions.orphans_json in the same
// transaction as everything else, so writerOff's contract
// ("the index can represent everything I write") holds
// without a JSON-envelope escape hatch.
} catch (err) {
// Kernel-store never-throw save discipline: an uncaughtException
// escaping this microtask is fatal in embedded/plugin hosts
Expand Down
42 changes: 6 additions & 36 deletions src/session.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import { durableMessageToolPaths, withDurableProtection } from "./durable-messag
import type { AnonymousProvenance } from "./anon-provenance.js";
import type { WireProtocol } from "./util.js";
import type { InboundSlot, SeamSlot } from "./cache-ledger.js";
import { isResponsesTurnSeparatorId } from "./turn-separator.js";

export type BlockView = { text: string; count: number };

Expand Down Expand Up @@ -1621,42 +1622,11 @@ export function foldCoverage(
return matched < coveredBefore.size ? { expected: coveredBefore.size, matched } : null;
}

/** #2627: bili-synthesized Responses turn separators (repairResponsesAssistantOrdering,
* src/server.ts) are OUTBOUND-ONLY ordering markers — no client's resent history
* ever contains them, so they can never anchor or be covered. When a
* separator-bearing view reached applyCompression (the plugin lane handed its
* processed view straight in), their ids were registered into new blocks'
* direct/effectiveMessageIds and inherited by higher folds via child merge —
* permanently "missing" on every reconcile pass: false [acp-drift]/fold-
* reconcile warns and stalled per-block ledger evidence against an unchanged
* history. They are protocol-ordering artifacts, not compressible raw history:
* exclude them everywhere coverage is stored or compared. */
export const RESPONSES_TURN_SEPARATOR_ID_PREFIX = "acp_turn_sep_";

export function isResponsesTurnSeparatorId(id: string): boolean {
return id.startsWith(RESPONSES_TURN_SEPARATOR_ID_PREFIX);
}

/** #2627: strip turn-separator ids from block coverage IN PLACE (load-time heal
* of persisted state + post-commit scrub). Returns how many ids were removed;
* 0 means the state was already clean. */
export function scrubTurnSeparatorIds(
blocks: ReadonlyArray<{ directMessageIds?: string[]; effectiveMessageIds?: string[] }>,
): number {
let removed = 0;
for (const block of blocks) {
for (const key of ["directMessageIds", "effectiveMessageIds"] as const) {
const ids = block[key];
if (!ids) continue;
const kept = ids.filter((id) => !isResponsesTurnSeparatorId(id));
if (kept.length !== ids.length) {
removed += ids.length - kept.length;
block[key] = kept;
}
}
}
return removed;
}
// #2627 turn-separator helpers moved to ./turn-separator.ts (zero-dependency leaf
// — the storage seam needs them without session.ts's built-acp-kernel import;
// the doc comments live there now). Re-exported so existing consumers keep the
// session.js import path.
export { RESPONSES_TURN_SEPARATOR_ID_PREFIX, isResponsesTurnSeparatorId, scrubTurnSeparatorIds } from "./turn-separator.js";

/** #2627: the REAL-history coverage set — active blocks' effective ids minus
* outbound-only turn separators. Single source for every drift consumer (the
Expand Down
76 changes: 76 additions & 0 deletions src/state-hydration.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
/**
* #4242 / #4192: state hydration is the single source of truth for which
* CompressionState fields survive persistence. mergeState() moved here so
* both the JSON envelope loader (persist.ts) and the unified view builder
* (storage/ingest.ts) consume ONE implementation, and the field list is
* DERIVED from it mechanically — Object.keys(mergeState(fresh)) — so a field
* added to the literal appears in kstate serialization, view rebuild and
* digest hashing automatically. The hand-kept MERGE_STATE_FIELDS list is
* gone; it had already drifted (sweptOrphanCallIds,
* pendingOrphanSweepMinRef — the #2388 explicit-field-clone disease).
*/
// Relative kernel-source import, NOT the "acp-kernel" package: this module is
// in the storage seam's load graph, which the CI mysql-contract lane runs with
// an UNBUILT kernel (install --ignore-scripts, no build step) — the package
// form resolves to kernel/dist and crashes that lane at import time (#4249).
import { createInitialState } from "../kernel/src/state.js";
import type { CompressionState } from "../kernel/src/types.js";
import { scrubTurnSeparatorIds } from "./turn-separator.js";

/** Forward-compat: merge a parsed state with a fresh one so missing fields
* (added in later versions) get sane defaults instead of `undefined`. */
export function mergeState(parsed: CompressionState): CompressionState {
const fresh = createInitialState();
return {
// #2627: heal persisted state written before the turn-separator fix —
// those blocks' coverage carries outbound-only acp_turn_sep_* ids that
// no resent history can ever contain; scrubbing at hydration is what
// stops live long sessions from alarming forever after an upgrade.
blocks: (() => { const blocks = parsed.blocks ?? fresh.blocks; scrubTurnSeparatorIds(blocks); return blocks; })(),
messageRefs: parsed.messageRefs ?? fresh.messageRefs,
nudge: { ...fresh.nudge, ...(parsed.nudge ?? {}) },
stats: { ...fresh.stats, ...(parsed.stats ?? {}) },
nextBlockId: parsed.nextBlockId ?? fresh.nextBlockId,
nextRunId: parsed.nextRunId ?? fresh.nextRunId,
tokenSnapshot: parsed.tokenSnapshot ?? fresh.tokenSnapshot,
lastPassIds: parsed.lastPassIds ?? fresh.lastPassIds,
hiddenOrphanRefs: parsed.hiddenOrphanRefs ?? fresh.hiddenOrphanRefs,
// #2362: without this a restart forgets which refs are dead and the
// model re-learns each one by failing a compress call again.
deadRefs: parsed.deadRefs ?? fresh.deadRefs,
terminalStreak: parsed.terminalStreak ?? fresh.terminalStreak,
nextRuleId: parsed.nextRuleId ?? fresh.nextRuleId,
// Without this, a restart re-exposes absorbed tool outputs: state
// resurrects with absorbed=[] and hideAbsorbedMessages has nothing to hide.
absorbed: parsed.absorbed ?? fresh.absorbed,
rules: parsed.rules ?? fresh.rules,
// #3769: fold-gated orphan-sweep bookkeeping must survive hydration —
// dropping sweptOrphanCallIds resurrects already-cleaned pairs after a
// restart (mid-history rewrite = cache break); the pending marker is
// transient, losing it only defers the sweep to the next fold.
sweptOrphanCallIds: parsed.sweptOrphanCallIds ?? fresh.sweptOrphanCallIds,
pendingOrphanSweepMinRef: parsed.pendingOrphanSweepMinRef ?? fresh.pendingOrphanSweepMinRef,
// #3939 ③/#3691: the emergency-truncate latch must survive hydration —
// dropping it re-exposes clipped bytes mid-history after a restart,
// rewriting the still-cached prefix on the first post-restart turn.
emergencyTruncateLatch: parsed.emergencyTruncateLatch ?? fresh.emergencyTruncateLatch,
// #3939 ②/#3805: the render deferral must survive hydration — losing
// it stamps migration tags onto still-cached carriers right after a
// restart (a silent mid-history rewrite with no fold in sight).
renderDeferredRefs: parsed.renderDeferredRefs ?? fresh.renderDeferredRefs,
// #3939 ⑤: the view latch must survive hydration — losing it
// re-derives transform output on still-cached carriers right after a
// restart (silent mid-history rewrite, no fold in sight).
viewLatches: parsed.viewLatches ?? fresh.viewLatches,
viewLatchConfigDigest: parsed.viewLatchConfigDigest ?? fresh.viewLatchConfigDigest,
};
}

/**
* Every CompressionState field mergeState() emits, in literal order. Derived
* at module load from the merge itself — no second list to keep in sync
* (#4192). Consumers: kstate_json serialization order (storage/ingest.ts),
* unified view rebuild, digest key order.
*/
export const COMPRESSION_STATE_FIELDS: readonly (keyof CompressionState)[] =
Object.keys(mergeState(createInitialState())) as (keyof CompressionState)[];
Loading
Loading