diff --git a/CHANGELOG.md b/CHANGELOG.md index f129729362..1860cf8d79 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,8 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag ### Changes +- **快照渲染:proxy 车道的出站前缀改由服务端正史渲染,存量会话对内核升级免疫(#3756,根治 #3752/#3751)。** 折叠覆盖此前锚定在"客户端本轮重发的字节"上——消息 id 由内容哈希现算,身份函数一变(协议切换 #3640、#3642 的一次性 re-key、dsh 图片 offload #3588、宿主自查点)覆盖名单就与历史失联:轻则整段历史因 MAX_ANCHORS=16384 封顶失锚(#3752),重则图片族 `_N` 后缀全体位移重铸、每轮 fold 断缓存(#3751)。新引擎 `src/server/snapshot-render.ts`(灰度开关 `BILI_ROLLOUT_SNAPSHOT_RENDER`,默认 0%):当会话有活跃折叠块且统一存储可读时,prepare 层的出站消息序列 = 服务端快照行前缀(覆盖区由块的 seq/signature 覆盖集自拥抑制 + `acp_summary_*` 锚原位重渲,身份跟随块而非客户端字节)+ 客户端尾巴(分歧点 join 之上逐字保留,rewind 裁剪不复活)。链回溯用 id 快路径 + 内容签名兜底(#3658 canonical 种子同构,升级后同内容仍可链上);任何环节失步整轮降级回 legacy 路径,零行为面风险。验收:身份函数原地翻盐后出站前缀逐字节不变(老病当场钉死)、协议中途切换会话不分叉、客户端任意删/checkpoint 覆盖区不复活(一轮采用滞后,模块头有注释)、稳态轮轮前缀字节稳。观测闩 `metadata.snapshotRenderTurns`;kernel 加 `__setMessageIdSaltForTests` 测试缝(kernel 0.0.134,仅测试可达,生产零调用)。 + - **Automatic upgrades now stagger across a window instead of landing within minutes; a public policy file controls the window fleet-wide (#3619).** Every bili instance that discovers a new version draws ONE random moment inside a delay window (built-in default **15 min – 48 h**), persists it, and upgrades when it arrives — restarts never re-roll. Before this, a release hit every auto-updating install within ~15–30 minutes, leaving no time to stop a bad one mid-rollout (#3401/#3600/#3394 were all shipped-too-fast incidents). The window and the emergency controls live in `auto-update-policy.json` in this repo, fetched anonymously (plain GET + ETag, zero telemetry, fail-open — an unreachable or malformed file falls back to local defaults): `hold` freezes automatic upgrades fleet-wide, `blocked` skips listed versions, `pinned` sets an upgrade ceiling, and per-version `overrides` tune the window (`{ "range": ">=0.1.200", "maxDelay": "1h" }`). Delay fields take human-readable durations — `"15m"`, `"2d"`, `"1d12h"` (units s/m/h/d/w, combinable; plain numbers still mean raw ms); malformed values are ignored with a warning, never guessed. The file steers timing/skip/pin only — it can never control what gets installed (no URLs, no commands; target versions must exist in the npm packument). Exits: manual `bili update` bypasses `hold`/stagger (still respects `blocked`/`pinned`), `update.staggerMinDelayMs: 0` + `update.staggerMaxDelayMs: 0` (or the `BILI_UPDATE_STAGGER_*_MS` twins) restore legacy immediate upgrades on one machine, `ACP_AUTO_UPDATE=0` disables auto-updates entirely. The update-driven e2e suites pin stagger 0/0 + an unroutable policy URL — they test periodic-update mechanics, not rollout timing. - **openai wire: non-array content tolerance (kernel 0.0.133, #3658 follow-up)** — the #3642 per-part rewrite iterated `messages[].content` and threw `TypeError: m.content is not iterable` on clients that emit a lone content-part object instead of an array (#2824 corpus caught it). Non-array non-string content now degrades to the legacy empty-text core (byte-compatible ids and bytes with the pre-#3642 codec) instead of throwing. Also regenerates the storage-render chat goldens for the documented one-time canonical re-key (tool-pair ids), and inverts the retired reconcile-off id-churn control into an absorption canary: with reconcile OFF, tool_call id rewrites + argument re-serialization must not disturb fold coverage (canonical ids absorb them by construction). - **protocol-stable core identity: the same logical conversation now yields byte-identical core message ids in all four dialects (#3642).** Wire codecs previously baked dialect accidents into message identity: anthropic used `"[image]"` as image text while openai deliberately kept `""` ("switching the seed would move every pre-existing shape"), tool-call ids embedded the provider's `toolCallId` bytes, raw JSON tool arguments hashed verbatim (key order = new id), and responses' empty-text reasoning cells seeded from the provider's volatile `item.id`. Switch protocols mid-session (opencode #3640: `/chat/completions` → `/v1/messages`) and ~20-68% of ids churned, tripping the fold-destroyed misdiagnosis #3641 just class-fixed. Identity now derives from a canonical form shared by all four codecs: a new `kernel/src/wire/identity.ts` (canonicalToolArgs: JSON args parsed → recursive key sort → compact stringify, verbatim for non-JSON; canonicalOpaqueSeed: provider `item.id`/`status` stripped; image marker `[image]` everywhere), the `toolCallId` slot retired from the seed (cluster suffixes disambiguate same-shape calls, deleting one sibling is local damage), and granularity normalized to one core per content part (anthropic's existing shape — its lanes take zero granularity re-keying, though tool-call/tool-result cores still re-seed since the `toolCallId` slot is retired; openai/responses/google split runs, with accumulators re-merging on rebuild so the wire stays byte-stable). The kernel first-user pin now spans the opening message's whole core run, so a multimodal opening message keeps its image bytes through folds (#781/#2607). Cross-protocol golden: text/tool/bash conversation → identical id arrays in all four dialects; canonical key-sort makes `{"file","line"}` == `{"line","file"}`. Image cells are now stamped `imagePart: true` at parse time (never text-compared — tag injection rewrites markers mid-render, which previously made image cores render as text and lose the real bytes, #767 billing fallout). Upgrade cost is one re-key of stored ids for openai/responses/google lanes; the #3641 reconcile tiers (word-bag claims, k-th image family pairing, toolCallId channel, epoch gate) absorb it without re-folding. diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 4e0bb0788b..351c9f127f 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -1877,6 +1877,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_PERSIST_EPERM_ALERT_REPEAT_MS` | Re-alert window for the persist EPERM alert, in ms. `0` (default) = alert once then stay silent; `>0` = re-alert at most every that many ms while the failures continue. | | `BILI_TUNNEL_ALLOWED_HOSTS` | `/bili/` tunnel admission for **remote clients** (#409): comma-separated `host` or `host:port` entries that unlock loopback/private destinations (e.g. a LAN relay or the machine's own sglang) for non-loopback clients. The proxy itself and link-local/metadata addresses are always denied; local (loopback) clients always pass. | | `BILI_MAX_SESSIONS` | Max sessions held in memory (default `256`; LRU eviction — disk is the source of truth). | +| `BILI_ROLLOUT_SNAPSHOT_RENDER` | Snapshot render (#3756): set `1` to render the outbound history prefix from the server-side unified-storage transcript instead of the client-resent bytes. Active only when the session has live fold blocks AND unified storage is readable; otherwise every request falls back to the legacy path unchanged. Kills the #3752/#3751 id-churn class: anchor coverage follows block seqs + content signatures, so codec/identity upgrades, client checkpoints and host compactions can no longer drop or duplicate folded history. Per-feature form of `BILI_ROLLOUT_` (default 0% rollout). | | `BILI_SESSION_WARM_MAX` | Bounded startup (#2803): max most-recently-saved sessions hydrated at boot before listening (default `0` = warm none, capped by `sessions.max`); older state reloads on demand. Set e.g. `8` to restore eager warming of the recent set. File twin: `sessions.warmMax`. | | `BILI_SESSION_WARM_MAX_BYTES` | Decoded-byte budget for the boot warm set (#2803; default `67108864` = 64 MiB). Counts DECODED size, not compressed on-disk bytes, so zstd archives cannot sneak past the RAM bound. A single file over the budget is never warmed. File twin: `sessions.warmMaxBytes`. | | `BILI_SESSION_RESIDENT_MAX_BYTES` | Soft target for estimated resident session bytes (#2803; default `536870912` = 512 MiB). Active/in-flight state stays pinned above it (the byte budget limits the idle cache, not active workload); when over target, idle state discharges back to disk as requests complete. File twin: `sessions.residentMaxBytes`. | diff --git a/kernel/package.json b/kernel/package.json index 37cc5254dc..432699877c 100644 --- a/kernel/package.json +++ b/kernel/package.json @@ -1,6 +1,6 @@ { "name": "acp-kernel", - "version": "0.0.134", + "version": "0.0.135", "description": "Framework-agnostic context-compression engine (model-driven, 3-tier LSM). Pure core: no host dependency.", "license": "MIT", "author": "ranxianglei", diff --git a/kernel/src/wire/identity.ts b/kernel/src/wire/identity.ts index ebefcf491c..1f67201be4 100644 --- a/kernel/src/wire/identity.ts +++ b/kernel/src/wire/identity.ts @@ -80,13 +80,22 @@ function canonicalize(v: unknown): unknown { * pinned empty, so plain-text messages keep their pre-#3642 ids verbatim * (zero migration churn for text-only sessions). */ +let idSalt: string | null = null; + +/** TEST ONLY (#3756): flips the id seed to simulate a codec/identity upgrade + * mid-session — every subsequent id changes although content is byte-identical. + * Production code never sets this. */ +export function __setMessageIdSaltForTests(salt: string | null): void { + idSalt = salt; +} + export function deriveCanonicalMessageId( role: string, contentType: string, text: string, options: { toolName?: string } = {}, ): string { - const seed = `${role}|${contentType}||${options.toolName ?? ""}|${text}`; + const seed = `${idSalt ? idSalt + "|" : ""}${role}|${contentType}||${options.toolName ?? ""}|${text}`; return "h_" + hashId(seed); } diff --git a/kernel/src/wire/index.ts b/kernel/src/wire/index.ts index a490eb66fe..00e8166038 100644 --- a/kernel/src/wire/index.ts +++ b/kernel/src/wire/index.ts @@ -9,3 +9,4 @@ export * from "./demoted-thinking.js"; export * from "./mirror.js"; export * from "./util.js"; export * from "./compress-detect.js"; +export * from "./identity.js"; diff --git a/package-lock.json b/package-lock.json index 7c1c70dbee..bd7e652912 100644 --- a/package-lock.json +++ b/package-lock.json @@ -48,7 +48,7 @@ }, "kernel": { "name": "acp-kernel", - "version": "0.0.134", + "version": "0.0.135", "dev": true, "license": "MIT", "devDependencies": { diff --git a/src/persist.ts b/src/persist.ts index 62f9fa9cd7..d8e9c5bc0a 100644 --- a/src/persist.ts +++ b/src/persist.ts @@ -1294,6 +1294,30 @@ export class SessionStore { } } + /** #3756 snapshot render: the logical transcript view's seq pointers + * (tail_json.tailSeqs). Null when the unified store is unavailable. */ + unifiedTailSeqs(sessionId: string): number[] | null { + if (!this.shadow || this.shadow.stats.disabled) return null; + try { + return this.shadow.tailSeqs(sessionId); + } catch { + return null; + } + } + + /** #3756 snapshot render: full-identity row view of the unified + * transcript (msgId/ref/refNum/contentHash included) in seq order. + * Null when the unified store is unavailable — the render lane must + * degrade to the legacy path, never guess. */ + unifiedTailViewRows(sessionId: string): import("./storage/store.js").MessageRow[] | null { + if (!this.shadow || this.shadow.stats.disabled) return null; + try { + return this.shadow.tailViewRows(sessionId); + } catch { + return null; // sidecar degraded mid-call: treat as unavailable + } + } + /** #2960 R2.1: summary rows for list surfaces (CLI export, web exposure * tick). Full mode only — in shadow the JSON tree is the truth base and * rows may lag files never yet recorded, so list surfaces must walk. diff --git a/src/rollout.ts b/src/rollout.ts index fdc67a0d02..fe727bc590 100644 --- a/src/rollout.ts +++ b/src/rollout.ts @@ -61,6 +61,14 @@ export const ROLLOUT_FEATURES: Record = { // `rollout.features.anthropicThreadsReforge: true` or // `BILI_ROLLOUT_ANTHROPIC_THREADS_REFORGE=1`. anthropicThreadsReforge: { percent: 0 }, + // #3756 snapshot render: assemble the outbound prefix from the server-side + // canonical transcript (unified storage) instead of re-deriving message + // identity from client-resent bytes, making folded sessions immune to + // codec/identity upgrades (#3752) and to client-side head + // deletion/checkpointing (#3723). Degrades to the legacy path whenever the + // rollout is off or the unified transcript cannot resolve. Default 0% — + // force on with `BILI_ROLLOUT_SNAPSHOT_RENDER=1`. + snapshotRender: { percent: 0 }, }; export const BP_MAX = 10_000; diff --git a/src/server/prepare-anthropic.ts b/src/server/prepare-anthropic.ts index 67d84035b9..ea5423e251 100644 --- a/src/server/prepare-anthropic.ts +++ b/src/server/prepare-anthropic.ts @@ -10,6 +10,7 @@ import { applyReasoningDrop, diagNudge, diagTagSummary, deriveTitle, effectiveTo import { buildDecisionPrompt, buildDirectiveText, consumeFallback, ladderMode, resolveDecisionRange, stripDecisionReasoningFields, type DecideConfig } from "../nudge-decide.js"; import { excludeDeadRefRanges } from "../acp-status.js"; import { reconcileFoldCoverage, noteSystemPromptFingerprint, resolveFoldReconcileMode } from "../fold-reconcile.js"; +import { maybeSnapshotView } from "./snapshot-render.js"; import { nudgeSuppressed } from "../session-self-heal.js"; import { compressBreakerArmed } from "../stream.js"; import { applyCompactionArchive, coveredRealHistoryIds, detectUnannouncedHistoryRewrite, foldCoverage, markCompactionBoundary, markDirty, markNativeCompactionBoundary, reconcileNativeCompactionBoundary, REWRITE_MIN_INCOMING_TOTAL, snapshotMessages, type PendingRetrieval, type Session } from "../session.js"; @@ -196,6 +197,11 @@ export async function prepareAnthropic( try { const { msgs, cacheControls } = anthropicToCore(parsed); originalMessages = msgs; + // #3756 snapshot render: outbound prefix from the unified transcript + // (fold-time ids, immune to identity re-mints); live tail keeps the + // client's bytes. No-op when the lane is inactive/degraded; the + // reconcile call below still observes the CLIENT view. + const renderMsgs = maybeSnapshotView(session, msgs, { sessionId, log }); // #1320: signature-only thinking blocks bill restored thinking tokens upstream // but project as empty text locally — attribute the provider-vs-local residual // to them so every meter sees the billed context (metering-only, wire untouched). @@ -224,7 +230,7 @@ export async function prepareAnthropic( // zero-baseline forks replay their FULL raw history with no measurement, // so feeding 0 blinds the nudge (usage 0%, growth ref 0) and no // compression trigger fires until overflow. See effectiveTokenCount. - const { tokens: tokenCount, source: tokenCountSource } = effectiveTokenCount(session, msgs, inboundImageTokens); + const { tokens: tokenCount, source: tokenCountSource } = effectiveTokenCount(session, renderMsgs, inboundImageTokens); const activeBefore = new Set(session.state.blocks.filter((b) => b.active).map((b) => b.blockId)); // Absorb markers are injected by the kernel's processTurn from // config.absorb. With no channel to call the tool (injection off), @@ -287,8 +293,8 @@ export async function prepareAnthropic( // #3222: BEFORE processTurn — its hide-compress-calls node removes // inactive call+result pairs inside, which would strip the anchors // this pass needs (the leak applies to the bare name too). - reasoningPlan = planReasoningDrop(msgs, reasoning, isStrictReasoningEcho(session, upstreamOrigin, modelIdOf(parsed))); - const turn = core.processTurn({ messages: msgs, state: session.state, config: loopConfig, tokenCount, renderTags: renderStrategy, contentStore: contentStoreOf(session) }); + reasoningPlan = planReasoningDrop(renderMsgs, reasoning, isStrictReasoningEcho(session, upstreamOrigin, modelIdOf(parsed))); + const turn = core.processTurn({ messages: renderMsgs, state: session.state, config: loopConfig, tokenCount, renderTags: renderStrategy, contentStore: contentStoreOf(session) }); session.state = turn.state; adoptContentStore(session, turn.contentStore); // The fold from last turn's compress has now materialized in state — diff --git a/src/server/prepare-openai.ts b/src/server/prepare-openai.ts index 2fb81e6499..13cebd11ad 100644 --- a/src/server/prepare-openai.ts +++ b/src/server/prepare-openai.ts @@ -10,6 +10,7 @@ import { applyReasoningDrop, diagNudge, diagTagSummary, deriveTitle, effectiveTo import { buildDecisionPrompt, buildDirectiveText, consumeFallback, ladderMode, resolveDecisionRange, stripDecisionReasoningFields, type DecideConfig } from "../nudge-decide.js"; import { excludeDeadRefRanges } from "../acp-status.js"; import { reconcileFoldCoverage, noteSystemPromptFingerprint, resolveFoldReconcileMode } from "../fold-reconcile.js"; +import { maybeSnapshotView } from "./snapshot-render.js"; import { nudgeSuppressed } from "../session-self-heal.js"; import { compressBreakerArmed } from "../stream.js"; import { applyCompactionArchive, coveredRealHistoryIds, detectUnannouncedHistoryRewrite, foldCoverage, markCompactionBoundary, markDirty, markNativeCompactionBoundary, reconcileNativeCompactionBoundary, REWRITE_MIN_INCOMING_TOTAL, snapshotMessages, type PendingRetrieval, type Session } from "../session.js"; @@ -112,13 +113,19 @@ export async function prepareOpenai( openaiSystemText = outcome.outbound; } originalMessages = msgs; + // #3756 snapshot render: the outbound prefix renders from the unified + // transcript (server-authoritative, fold-time ids — immune to identity + // re-mints #3752); only the live tail keeps the client's bytes. + // No-op returns `msgs` when the lane is inactive/degraded. + // reconcileFoldCoverage below still observes the CLIENT view. + const renderMsgs = maybeSnapshotView(session, msgs, { sessionId, log }); // #1001: pre-turn snapshot — processTurn below assigns fresh refs to every // previously-unknown id, which would make rewrite detection read 1.0. const knownRefsBefore = new Set(Object.keys(session.state.messageRefs.byRaw)); // tokenCount = upstream's real input_tokens from the previous turn // tokenCount = upstream's real input_tokens from the previous turn // (see anthropic branch comment + its #553-follow-up exception). - const { tokens: tokenCount, source: tokenCountSource } = effectiveTokenCount(session, msgs, imageReserveFor(session, "openai", parsed, opts, billingUpstream ?? upstreamOrigin)); + const { tokens: tokenCount, source: tokenCountSource } = effectiveTokenCount(session, renderMsgs, imageReserveFor(session, "openai", parsed, opts, billingUpstream ?? upstreamOrigin)); const activeBefore = new Set(session.state.blocks.filter((b) => b.active).map((b) => b.blockId)); // Absorb markers ride in the kernel's processTurn output (gated by @@ -177,8 +184,8 @@ export async function prepareOpenai( // #3222: BEFORE processTurn — its hide-compress-calls node removes // inactive call+result pairs inside, which would strip the anchors // this pass needs (the leak applies to the bare name too). - reasoningPlan = planReasoningDrop(msgs, reasoning, isStrictReasoningEcho(session, upstreamOrigin, modelIdOf(parsed))); - const turn = core.processTurn({ messages: msgs, state: session.state, config: loopConfig, tokenCount, renderTags: renderStrategy, contentStore: contentStoreOf(session) }); + reasoningPlan = planReasoningDrop(renderMsgs, reasoning, isStrictReasoningEcho(session, upstreamOrigin, modelIdOf(parsed))); + const turn = core.processTurn({ messages: renderMsgs, state: session.state, config: loopConfig, tokenCount, renderTags: renderStrategy, contentStore: contentStoreOf(session) }); session.state = turn.state; adoptContentStore(session, turn.contentStore); // The fold from last turn's compress has now materialized in state — diff --git a/src/server/prepare-responses.ts b/src/server/prepare-responses.ts index b8ec042f3e..b5de5b5bdb 100644 --- a/src/server/prepare-responses.ts +++ b/src/server/prepare-responses.ts @@ -12,6 +12,7 @@ import { excludeDeadRefRanges } from "../acp-status.js"; import { mergeAdjacentConfigurationUpdates, patchResponsesInputWithToolImages as patchResponsesInput, responsesToCoreWithToolImages as responsesToCore } from "../responses-tool-output.js"; import { dropWhitespaceResponsesMessages, normalizeResponsesMessageItems, sanitizeResponsesInputIds } from "../loop/adapter-responses.js"; import { METADATA_LAST_RESPONSES_MODE, noteCanonicalSystemFingerprint, reconcileFoldCoverage, noteSystemPromptFingerprint, resolveFoldReconcileMode } from "../fold-reconcile.js"; +import { maybeSnapshotView } from "./snapshot-render.js"; import { nudgeSuppressed, pluginLaneDegraded } from "../session-self-heal.js"; import { compressBreakerArmed } from "../stream.js"; import { coveredRealHistoryIds, foldCoverage, markDirty, markNativeCompactionBoundary, reconcileNativeCompactionBoundary, REWRITE_MIN_INCOMING_TOTAL, snapshotMessages, type Session } from "../session.js"; @@ -208,10 +209,15 @@ export async function prepareResponses( } const { msgs } = projection; originalMessages = msgs; + // #3756 snapshot render: outbound prefix from the unified transcript + // (fold-time ids, immune to identity re-mints); live tail keeps the + // client's bytes. No-op when the lane is inactive/degraded; the + // reconcile call below still observes the CLIENT view. + const renderMsgs = maybeSnapshotView(session, msgs, { sessionId, log }); if (opts.debug) { log("info", `[${sessionId}] input items: ${Array.isArray(parsed.input) ? parsed.input.map((i: ResponseInputItem) => i.type).join(",") : "(string)"}`); } - const { tokens: tokenCount, source: tokenCountSource } = effectiveTokenCount(session, msgs, imageReserveFor(session, "responses", parsed, opts, billingUpstream ?? upstreamOrigin)); + const { tokens: tokenCount, source: tokenCountSource } = effectiveTokenCount(session, renderMsgs, imageReserveFor(session, "responses", parsed, opts, billingUpstream ?? upstreamOrigin)); // Absorb markers ride in the kernel's processTurn output (gated by // config.absorb). The marker/text protocol has no native tool channel, // so strip absorb from the loop config there (both modes). @@ -282,8 +288,8 @@ export async function prepareResponses( // #3222: BEFORE processTurn — its hide-compress-calls node removes // inactive call+result pairs inside, which would strip the anchors // this pass needs (the leak applies to the bare name too). - reasoningPlan = planReasoningDrop(msgs, reasoning, isStrictReasoningEcho(session, upstreamOrigin, modelIdOf(parsed))); - const turn = core.processTurn({ messages: msgs, state: session.state, config: loopConfig, tokenCount, renderTags, contentStore: contentStoreOf(session) }); + reasoningPlan = planReasoningDrop(renderMsgs, reasoning, isStrictReasoningEcho(session, upstreamOrigin, modelIdOf(parsed))); + const turn = core.processTurn({ messages: renderMsgs, state: session.state, config: loopConfig, tokenCount, renderTags, contentStore: contentStoreOf(session) }); session.state = turn.state; adoptContentStore(session, turn.contentStore); // The fold from last turn's compress has now materialized in state — diff --git a/src/server/snapshot-render.ts b/src/server/snapshot-render.ts new file mode 100644 index 0000000000..a7933cc6fa --- /dev/null +++ b/src/server/snapshot-render.ts @@ -0,0 +1,235 @@ +import type { BiliMessage } from "acp-kernel/wire"; +import { getStore } from "../persist.js"; +import { featureEnabled } from "../rollout.js"; +import type { Session } from "../session.js"; + +/** + * #3756 snapshot render: assemble the outbound message view for a proxy + * request from the server-side canonical transcript instead of re-deriving + * message identity from the client-resent bytes. + * + * Doctrine (issue #3756, m-tier design): the (sessionId, unified snapshot) + * pair IS the conversation; the client's resent history is an alignment + * observation. Concretely: + * + * - prefix authority: the unified transcript rows (core-form, fold-time + * message ids). Once a row exists, the outbound byte for it never + * changes — codec/identity upgrades re-mint client ids every turn but + * rows keep the ids the kernel assigned at fold time (#3752 becomes a + * non-event: refs/tags stay stable, the upstream cache prefix holds). + * - tail authority: the client's live messages — everything the client + * sent that the transcript has not yet recorded (this turn's fresh + * user message, the previous reply before it is re-observed) ride the + * client's own bytes. + * - alignment: a contiguous backward chain from the client's end against + * the transcript view (id fast path, then the #3658 canonical identity + * signature — role|contentType|toolName|text — so an upgrade turn where + * every id is re-minted still chains by content). + * - client head edits (checkpoints, deletions, compactions of the head) + * never resurrect or duplicate rows: the prefix renders what the server + * has, the chain's first unmatched client message starts the tail, and + * anything below the chain is dropped (server-authoritative head). + * - rewinds truncate the transcript view at the chain join: rows the + * client no longer vouches for (beyond the last chained row) do not + * render — the tail is client-authoritative past the join point. + * + * The merged view keeps kernel semantics untouched downstream: covered rows + * ride fold-time ids so prune strips them by effectiveMessageIds and renders + * the real summaries; a synthetic anchor (acp_summary_) is inserted + * past each block's covered region so collectSummaryAnchors resolves it at + * a deterministic position even when every covered row is outside the view. + * + * Inactive (rollout off / no active blocks / store unavailable / empty or + * unchainable transcript / any parse failure) ⇒ returns the client view + * unchanged — the legacy path, never a guess. + */ + +/** Client-tail walk budget. Full-resend histories are bounded by the wire; + * beyond this the lane degrades rather than scanning unbounded tails. */ +const MAX_CHAIN_WALK = 8192; + +/** #3658 canonical identity signature minus the hash: the tuple the kernel + * derives message ids from. Equal signature ⇒ same logical message even + * when the id bytes were re-minted by an upgrade. Image cores share the + * "[image]" text, so the sidecar length rides the signature (same image ⇒ + * same bytes ⇒ same length; different images practically always differ). */ +function identitySig(m: BiliMessage): string { + const text = typeof m.text === "string" ? m.text : ""; + const img = m.imagePart === true ? `|img:${(m.imageBase64 ?? "").length}` : ""; + return `${m.role}|${m.contentType}|${m.toolName ?? ""}|${text}${img}`; +} + +interface ViewRow { + seq: number; + msg: BiliMessage; +} + +interface SnapshotRenderDeps { + sessionId: string; + log?: (level: string, msg: string) => void; +} + +/** One-time + per-turn observability: the latch counts active renders; the + * activated flag logs exactly once per session. */ +function noteRender(session: Session, deps: SnapshotRenderDeps): void { + const prior = typeof session.metadata["snapshotRenderTurns"] === "number" ? (session.metadata["snapshotRenderTurns"] as number) : 0; + session.metadata["snapshotRenderTurns"] = prior + 1; + if (prior === 0) deps.log?.("info", `[snapshot-render] active: outbound prefix now rendered from the unified transcript (issue #3756)`); +} + +function parseRow(content: string, seq: number): BiliMessage | null { + try { + const parsed = JSON.parse(content) as { id?: unknown; role?: unknown }; + if (typeof parsed.id !== "string" || typeof parsed.role !== "string") return null; + return parsed as BiliMessage; + } catch { + return null; + } +} + +/** + * Returns the merged view, or `clientMsgs` unchanged when the lane is + * inactive/degraded. Pure with respect to the transcript: the only mutation + * is the observability latch on `session.metadata`. + */ +export function maybeSnapshotView(session: Session, clientMsgs: BiliMessage[], deps: SnapshotRenderDeps): BiliMessage[] { + if (!featureEnabled("snapshotRender")) return clientMsgs; + const activeBlocks = session.state.blocks.filter((b) => b.active); + if (activeBlocks.length === 0) return clientMsgs; + const store = getStore(); + if (!store) return clientMsgs; + let rows: ViewRow[] = []; + try { + const rawRows = store.unifiedTailViewRows(session.id); + const seqs = store.unifiedTailSeqs(session.id); + if (!rawRows || rawRows.length === 0 || !seqs || seqs.length === 0) return clientMsgs; + const bySeq = new Map(); + for (const row of rawRows) { + const msg = parseRow(row.content, row.seq); + if (!msg) return clientMsgs; // untrustworthy transcript: degrade, never guess + bySeq.set(row.seq, { seq: row.seq, msg }); + } + for (const seq of seqs) { + const v = bySeq.get(seq); + if (!v) return clientMsgs; // view pointer to a missing row: degrade + rows.push(v); + } + } catch { + return clientMsgs; + } + if (rows.length === 0) return clientMsgs; + + // Backward chain: from the client's end, match view rows (id fast path, + // then identity signature). The chain must be contiguous; the first + // unmatched client message bounds the client tail. + let joinRow = -1; // view index of the TOPMOST matched row — the join point + let clientJoin = clientMsgs.length; // first client index of the tail + { + const byId = new Map(); // msgId -> view index (last occurrence) + for (let i = 0; i < rows.length; i++) byId.set(rows[i].msg.id, i); + const bySig = new Map(); // sig -> descending view indices + for (let i = rows.length - 1; i >= 0; i--) { + const sig = identitySig(rows[i].msg); + const list = bySig.get(sig); + if (list) list.push(i); + else bySig.set(sig, [i]); + } + let cursor = rows.length; // exclusive upper bound: matched rows must sit below this + let started = false; + for (let k = clientMsgs.length - 1; k >= 0 && clientMsgs.length - 1 - k < MAX_CHAIN_WALK; k--) { + const m = clientMsgs[k]; + const sig = identitySig(m); + let hit = -1; + const byIdHit = byId.get(m.id); + if (byIdHit !== undefined && byIdHit < cursor) hit = byIdHit; + else { + const cands = bySig.get(sig); + if (cands) { + for (const idx of cands) { + if (idx < cursor) { + hit = idx; + break; // cands are descending: first below cursor is the nearest + } + } + } + } + if (hit < 0) { + // Before the chain starts, unmatched messages are the live + // tail (this turn's fresh user message, last turn's reply + // not yet re-observed) — skip them. Once the chain has + // started, an unmatched message ENDS it (contiguity). + if (started) break; + continue; + } + if (!started) { + started = true; + clientJoin = k + 1; // tail = everything above the topmost matched message + joinRow = hit; // the join point: the prefix renders every row up to and including it + } + cursor = hit; + } + if (!started) return clientMsgs; // nothing chains: epoch-like rewrite, legacy + } + const d = joinRow + 1; // prefix = rows[0..d) — everything up to the join; rows beyond (rewound-away tails) never render + + // Anchor synthesis: one anchor per active block, positioned past the + // block's covered region in the transcript (first view row whose seq + // exceeds this block's deepest covered row; end-of-prefix when none). + // Per-block covered seqs seed from effectiveMessageIds; the identity + // signature propagates them to upgrade-twin rows (id re-minted, same + // content). Covered rows themselves ride the prefix so prune strips them + // by effectiveMessageIds and renders the real summary — the anchor is + // the deterministic fallback position, not the stripper. + const anchors: { at: number; msg: BiliMessage }[] = []; + const coveredSeqUnion = new Set(); + for (const block of activeBlocks) { + const blockSeqs = new Set(); + const blockSigs = new Set(); + for (const id of block.effectiveMessageIds) { + for (let i = 0; i < rows.length; i++) { + if (rows[i].msg.id === id) { + blockSeqs.add(rows[i].seq); + blockSigs.add(identitySig(rows[i].msg)); + } + } + } + if (blockSeqs.size > 0) { + for (let i = 0; i < rows.length; i++) { + if (!blockSeqs.has(rows[i].seq) && blockSigs.has(identitySig(rows[i].msg))) blockSeqs.add(rows[i].seq); + } + } + for (const seq of blockSeqs) coveredSeqUnion.add(seq); + let maxCover = -1; + for (const seq of blockSeqs) maxCover = Math.max(maxCover, seq); + let at = d; + for (let i = 0; i < d; i++) { + if (rows[i].seq > maxCover) { + at = i; + break; + } + } + anchors.push({ + at, + msg: { id: `acp_summary_${block.blockId}`, role: "system", contentType: "text", text: "" } as BiliMessage, + }); + } + anchors.sort((a, b) => a.at - b.at); + + // The ENGINE owns coverage on this lane: covered rows never render — + // prune's effectiveMessageIds strip cannot be trusted here because + // reconcile may have re-keyed the block to the client's re-minted ids + // (#3752 upgrade turns), while the transcript rows keep fold-time ids. + // The synthesized anchor above carries each block's summary instead. + const merged: BiliMessage[] = []; + let ai = 0; + for (let i = 0; i < d; i++) { + while (ai < anchors.length && anchors[ai].at === i) merged.push(anchors[ai++].msg); + if (coveredSeqUnion.has(rows[i].seq)) continue; + merged.push(rows[i].msg); + } + while (ai < anchors.length) merged.push(anchors[ai++].msg); + for (let k = clientJoin; k < clientMsgs.length; k++) merged.push(clientMsgs[k]); + + noteRender(session, deps); + return merged; +} diff --git a/src/storage/shadow.ts b/src/storage/shadow.ts index 749d6aa598..e8f7cd39e9 100644 --- a/src/storage/shadow.ts +++ b/src/storage/shadow.ts @@ -28,6 +28,7 @@ import { UnifiedStore, type CheckpointRow, type SessionSummaryRow, type WebSumma import { canonicalize, sha256 } from "./canonical.ts"; import { recordSidecarInstance, recordSidecarOpenFailure } from "./runtime-status.ts"; import type { SqliteEngineName, SqliteEngineSource } from "./driver.ts"; +import type { MessageRow } from "./store.js"; import type { CoreMessage, MessageContentStore } from "acp-kernel"; export const MAX_CONSECUTIVE_FAILURES = 3; @@ -383,6 +384,24 @@ export class UnifiedShadow { return this.store.getMessages(sessionId).map((m) => ({ seq: m.seq, content: m.content })); } + /** #3756 snapshot render: the logical view's seq pointer list + * (tail_json.tailSeqs). Null when absent/malformed. */ + tailSeqs(sessionId: string): number[] | null { + this.assertLive(); + return this.store.tailSeqs(sessionId); + } + + /** #3756 snapshot render: full row view (id/ref/hash included) of the + * unified transcript in seq order — the render source for the outbound + * prefix. Same rows as rawMessages, plus the identity columns the + * coverage matcher needs (msgId for fast path, ref for the + * upgrade-tolerant interval path, contentHash for the exact-content + * dedup path). */ + tailViewRows(sessionId: string): MessageRow[] { + this.assertLive(); + return this.store.getMessages(sessionId); + } + sessionSummaries(): SessionSummaryRow[] { this.assertLive(); return this.store.sessionSummaries(); diff --git a/src/storage/store.ts b/src/storage/store.ts index 65e660fd36..fd4feecb2c 100644 --- a/src/storage/store.ts +++ b/src/storage/store.ts @@ -737,6 +737,24 @@ export class UnifiedStore { return { seq, tailSeqs: seqs }; } + /** #3756 snapshot render: the sessions row's tail_json view — the seq + * pointer list of the logical transcript view (lastMessages order at + * save time). Unlike ingestWatermark this does NOT require a trusted + * watermark; it just reads the pointer list. Null when absent/malformed + * — the render lane degrades to legacy. */ + tailSeqs(sessionId: string): number[] | null { + const row = this.db + .prepare("SELECT tail_json FROM sessions WHERE session_id = ?") + .get(sessionId) as { tail_json: string } | undefined; + if (!row) return null; + try { + const tail = JSON.parse(row.tail_json) as { tailSeqs?: number[] | null }; + return Array.isArray(tail.tailSeqs) ? tail.tailSeqs : null; + } catch { + return null; + } + } + /** #3166 review (heal half of the sampled-verify backstop): drop the * durable watermark so the next save takes the exact full path and * re-points every tail slot. Called when the read-back digest proves diff --git a/tests/snapshot-render.test.ts b/tests/snapshot-render.test.ts new file mode 100644 index 0000000000..cc8611d78f --- /dev/null +++ b/tests/snapshot-render.test.ts @@ -0,0 +1,403 @@ +// #3756: snapshot render — proxy-lane outbound rendered from the server-side +// canonical history instead of the client's resent bytes. +// +// Contract staked BEFORE the implementation (打桩): +// +// 1. steady state: the outbound prefix (covered region rendered as summary +// anchors + uncovered snapshot rows) is byte-stable turn over turn; +// 2. seed upgrade (a codec/identity change re-mints every message id on +// identical content — the #3752/#3658 regression class): the outbound +// prefix stays byte-identical, the block survives, no destroy verdict; +// 3. client-side deletion/checkpointing of the covered region is ignored +// (server-authoritative prefix), the checkpoint itself rides the tail; +// 4. rewind: dropped tail messages do not come back; +// 5. protocol switch mid-session: same session, anchors render, stable; +// 6. rollout off (default): the legacy path keeps today's behavior — the +// upgrade scenario demonstrably churns (the bug this lane fixes); +// 7. storage unavailable: D quietly falls back to the legacy path. +import assert from "node:assert/strict"; +import http from "node:http"; +import { once } from "node:events"; +import test from "node:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; + +process.env.NODE_ENV = "test"; +process.env.BILI_PERSIST_ZSTD = "0"; +process.env.BILI_CONFIG_FILE = path.join(mkdtempSync(path.join(tmpdir(), "bili-snap-hermetic-")), "no-config.json"); + +import { defaultConfig } from "acp-kernel"; +import { __setMessageIdSaltForTests } from "acp-kernel/wire"; +import { startServer } from "../src/server.ts"; +import type { ProxyOptions } from "../src/config.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { _setForTest as setRegistryForTest } from "../src/registry.ts"; +import { listSessions, _resetSessionsForTest } from "../src/session.ts"; +import { rmrf } from "./tmp-rm.ts"; + +const SUMMARY = "MAIN-SUMMARY-SETUP-CONTEXT-FOLDED-BY-COMPRESSION-LONG-ENOUGH-FOR-KERNEL-MIN-LENGTH-CHECK"; + +function sse(obj: unknown): string { + return `data: ${JSON.stringify(obj)}\n\n`; +} + +// 23 messages × ~1.5k chars (~375 tokens each): total ~9k tokens stays under +// the 15k window (no preflight); the default protection zone (preserveRecent +// 3 + 5000-token walk ≈ 14 messages) leaves the first ~9 compressible, so the +// m00001..m00008 range lands intact. The transcript ENDS ON A USER message — +// every turn's streamed reply then appends exactly one assistant message and +// the alternation never breaks (cache-friendly client discipline). +function conversation(tail: { role: string; content: string }[] = []): { role: string; content: string }[] { + const out: { role: string; content: string }[] = []; + for (let i = 0; i < 23; i++) { + out.push({ role: i % 2 === 0 ? "user" : "assistant", content: `Message ${i} of the working session. ` + `WORK_${i}_content_`.repeat(88) }); + } + return [...out, ...tail]; +} + +type Harness = { + proxy: http.Server; + upstream: http.Server; + bodies: string[]; + openaiUrl: string; + anthropicUrl: string; + sessionOf: (label: string) => { meta: { label: string }; state: { blocks: { id: string; active: boolean }[] }; metadata: Record } | undefined; +}; + +async function withHarness(fn: (h: Harness) => Promise, storeEnabled: boolean): Promise { + const bodies: string[] = []; + const upstream = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (c: Buffer) => chunks.push(c)); + req.on("end", () => { + bodies.push(Buffer.concat(chunks).toString("utf8")); + const n = bodies.length; + res.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" }); + if (n === 1) { + // turn 1 request: the model folds the oldest range and stops. + const args = JSON.stringify({ content: [{ startId: "m00001", endId: "m00008", topic: "setup", summary: SUMMARY }] }); + res.write(sse({ id: "c1", object: "chat.completion.chunk", choices: [{ index: 0, delta: { role: "assistant", content: null, tool_calls: [{ index: 0, id: "call_p", type: "function", function: { name: "compress", arguments: args } }] } }] })); + res.write(sse({ id: "c1", object: "chat.completion.chunk", choices: [{ index: 0, delta: {}, finish_reason: "tool_calls" }] })); + } else { + // turn 1 re-request (folded view) and later client turns: plain text. + const text = n === 2 ? "done compressing" : `ok ${n}`; + res.write(sse({ id: "c1", object: "chat.completion.chunk", choices: [{ index: 0, delta: { role: "assistant", content: text } }] })); + res.write(sse({ id: "c1", object: "chat.completion.chunk", choices: [{ index: 0, delta: {}, finish_reason: "stop" }] })); + } + res.end(); + }); + }); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const upstreamPort = (upstream.address() as { port: number }).port; + + const dir = mkdtempSync(path.join(tmpdir(), "bili-snap-store-")); + const store = storeEnabled ? new SessionStore({ enabled: true, dir, debounceMs: 0 }) : new SessionStore({ enabled: false }); + _setStoreForTest(store); + _resetSessionsForTest(); + setRegistryForTest({}); + const proxy = await startServer({ + port: 0, + host: "127.0.0.1", + upstream: "http://127.0.0.1", + routes: { [`http://127.0.0.1:${upstreamPort}`]: { models: { "gpt-test": { context: 15_000 } } } }, + modelContextLimit: 15_000, + kernelConfig: defaultConfig(15_000), + compress: { injectTool: true, injectNudge: true }, + promptCache: { routing: "auto" }, + sessionHeader: "x-acp-session", + log: Boolean(process.env.SNAP_DEBUG), + debug: Boolean(process.env.SNAP_DEBUG), + passthrough: false, + passthroughSource: null, + autoUpdate: false, + autoRestartOnUpdate: false, + updateTag: "latest", + advisoryCheck: true, + releaseNotesCheck: true, + compat: { roles: {} }, + streamErrorShape: "protocol", + mitm: { enabled: false, domains: [] }, + } as unknown as ProxyOptions); + await once(proxy, "listening"); + const port = (proxy.address() as { port: number }).port; + const h: Harness = { + proxy, + upstream, + bodies, + openaiUrl: `http://127.0.0.1:${port}/bili/http://127.0.0.1:${upstreamPort}/v1/chat/completions`, + anthropicUrl: `http://127.0.0.1:${port}/bili/http://127.0.0.1:${upstreamPort}/v1/messages`, + sessionOf: (label) => listSessions().find((x) => x.meta.label === label) as never, + }; + try { + await fn(h); + } finally { + proxy.close(); + proxy.closeIdleConnections?.(); + await once(proxy, "close"); + upstream.close(); + upstream.closeIdleConnections?.(); + await once(upstream, "close"); + __setMessageIdSaltForTests(null); + delete process.env.BILI_ROLLOUT_SNAPSHOT_RENDER; + store.closeQuietly(); + void rmrf(dir); + } +} + +type Msg = { role: string; content: string }; + +async function turn(h: Harness, url: string, label: string, messages: unknown[], history: Msg[] | null = null, extraHeaders: Record = {}): Promise<{ status: number; body: string; idx: number }> { + const r = await fetch(url, { + method: "POST", + headers: { "content-type": "application/json", "x-acp-session": label, ...extraHeaders }, + body: JSON.stringify({ model: "gpt-test", messages, stream: true }), + }); + const body = await r.text(); + await new Promise((res) => setTimeout(res, 60)); // let debounced=0 writes land + void history; // (history appended below after body parse) + // realistic client: fold the assistant's visible reply into the transcript + if (history) { + for (const m of extractAssistantText(body)) history.push(m); + } + return { status: r.status ?? 0, body, idx: h.bodies.length - 1 }; +} + +/** One assistant transcript entry per response: all plain-text content deltas + * concatenated (cache-friendly-proxy client discipline — the stateless client + * sees the model text plus any streamed 📦 receipt line as one reply). */ +function extractAssistantText(body: string): Msg[] { + let text = ""; + for (const block of body.trim().split("\n\n")) { + if (!block.startsWith("data: ")) continue; + const data = block.slice(6).trim(); + if (data === "[DONE]") continue; + try { + const delta = (JSON.parse(data) as { choices?: { delta?: { content?: unknown } }[] }).choices?.[0]?.delta; + const c = delta?.content; + if (typeof c === "string") text += c; + } catch { /* keep scanning */ } + } + return text ? [{ role: "assistant", content: text }] : []; +} + +/** outbound openai messages from the i-th upstream hit */ +function sent(bodies: string[], i: number): { role: string; content: unknown }[] { + return JSON.parse(bodies[i]).messages as { role: string; content: unknown }[]; +} + +function textOf(m: { role: string; content: unknown }): string { + return typeof m.content === "string" ? m.content : JSON.stringify(m.content); +} + +/** Count anchor-shaped renders (the receipt echo also contains the summary + * head; only the section marker counts as an anchor copy). */ +function anchorCount(s: string): number { + return s.split("[Compressed conversation section]").length - 1; +} + +/** The stable-prefix comparator: the serialized message stream of `a` up to + * its tail must equal `b` element-wise. tailLen = how many trailing messages + * of `a` are new (the live tail). */ +function assertPrefixStable(a: { role: string; content: unknown }[], b: { role: string; content: unknown }[], tailLenA: number, note: string): void { + const pa = a.slice(0, a.length - tailLenA); + assert.ok(b.length >= pa.length, `${note}: other view at least as long (${b.length} < ${pa.length})`); + for (let i = 0; i < pa.length; i++) { + assert.equal(JSON.stringify(pa[i]), JSON.stringify(b[i]), `${note}: prefix diverges at outbound message ${i}`); + } +} + +async function armBlock(h: Harness, label: string): Promise { + const r = await turn(h, h.openaiUrl, label, conversation()); + assert.equal(r.status, 200, r.body); + const s = h.sessionOf(label); + assert.ok(s, "session exists"); + assert.ok(s!.state.blocks.some((b) => b.active), "setup created an active block"); +} + +test("D on: steady state — anchor rendered once, covered rows absent, prefix byte-stable", async () => { + process.env.BILI_ROLLOUT_SNAPSHOT_RENDER = "100"; + await withHarness(async (h) => { + const history = conversation(); + const r1 = await turn(h, h.openaiUrl, "s-steady", history, history); + assert.equal(r1.status, 200, r1.body); + assert.ok(h.sessionOf("s-steady")!.state.blocks.some((b) => b.active), "block armed"); + // turn 2: transcript + a live tail of 2 + const tail = [ + { role: "user", content: "Message 24 tail question. " + "TAIL_A_".repeat(80) }, + ]; + const full2 = [...history, ...tail]; + const r2 = await turn(h, h.openaiUrl, "s-steady", full2, history); + assert.equal(r2.status, 200, r2.body); + const out2 = sent(h.bodies, r2.idx); + const joined2 = out2.map(textOf).join("\n"); + assert.ok(joined2.includes(SUMMARY), "summary anchor rendered"); + assert.equal(anchorCount(joined2), 1, "exactly one anchor copy"); + assert.ok(!joined2.includes("WORK_1_content_"), "covered row 1 suppressed"); + assert.ok(!joined2.includes("WORK_7_content_"), "covered row 7 suppressed"); + assert.ok(joined2.includes("TAIL_A_"), "live tail present"); + // turn 3: identical transcript + one more tail message → whole prefix + // (incl. everything before the new message) byte-identical to turn 2. + const r3 = await turn(h, h.openaiUrl, "s-steady", [...history, { role: "user", content: "Message 26 follow-up. " + "TAIL_C_".repeat(80) }], history); + assert.equal(r3.status, 200, r3.body); + const out3 = sent(h.bodies, r3.idx); + // tail = the new user message + the assistant run that absorbed the + // turn-2 reply (#3658 openai assistant run-merge discipline). + assertPrefixStable(out3, out2, 2, "steady-state"); + const s = h.sessionOf("s-steady")!; + assert.ok((s.metadata["snapshotRenderTurns"] as number) >= 2, "D observability latch counted turns"); + }, true); +}); + +test("D on: identity-seed upgrade — outbound prefix byte-identical, block alive, no destroy verdict", async () => { + process.env.BILI_ROLLOUT_SNAPSHOT_RENDER = "100"; + await withHarness(async (h) => { + const history = conversation(); + const r1 = await turn(h, h.openaiUrl, "s-upg", history, history); + assert.equal(r1.status, 200, r1.body); + assert.ok(h.sessionOf("s-upg")!.state.blocks.some((b) => b.active), "block armed"); + const tail = [{ role: "user", content: "Post-upgrade question. " + "UPG_A_".repeat(80) }]; + const r2 = await turn(h, h.openaiUrl, "s-upg", [...history, ...tail], history); + assert.equal(r2.status, 200, r2.body); + const out2 = sent(h.bodies, r2.idx); + // simulate the #3752/#3658 regression class: every id re-mints on the + // SAME byte-identical content. The client resends the same history. + __setMessageIdSaltForTests("upgrade-v2"); + const r3 = await turn(h, h.openaiUrl, "s-upg", [...history, ...tail], null); + assert.equal(r3.status, 200, r3.body); + const out3 = sent(h.bodies, r3.idx); + const joined3 = out3.map(textOf).join("\n"); + assert.ok(joined3.includes(SUMMARY), "anchor still rendered after id re-mint"); + assert.ok(!joined3.includes("WORK_1_content_"), "covered rows still suppressed (no duplicate render)"); + assert.ok(joined3.includes("UPG_A_"), "tail present"); + assert.equal(anchorCount(joined3), 1, "exactly one anchor copy"); + // prefix (everything except the live tail) byte-identical to pre-upgrade + assertPrefixStable(out3, out2, 1, "seed-upgrade"); + const s = h.sessionOf("s-upg")!; + assert.ok(s.state.blocks.some((b) => b.active), "block survived the upgrade"); + assert.equal(s.metadata["foldDriftEscalated"], undefined, "no destroy verdict"); + }, true); +}); + +test("D on: client drops/checkpoints the covered region — server-authoritative prefix, client checkpoint ignored", async () => { + process.env.BILI_ROLLOUT_SNAPSHOT_RENDER = "100"; + await withHarness(async (h) => { + const history = conversation(); + const r1 = await turn(h, h.openaiUrl, "s-drop", history, history); + assert.equal(r1.status, 200, r1.body); + assert.ok(h.sessionOf("s-drop")!.state.blocks.some((b) => b.active), "block armed"); + // dsh-style checkpoint: first 10 messages replaced by one summary message + const dropped = [ + { role: "user", content: "Earlier context was summarized by the client checkpoint: setup work happened." }, + ...history.slice(10), + { role: "user", content: "Continuing after checkpoint. " + "KEEP_A_".repeat(80) }, + ]; + const r2 = await turn(h, h.openaiUrl, "s-drop", dropped, null); + const out2 = sent(h.bodies, r2.idx); + const joined2 = out2.map(textOf).join("\n"); + assert.ok(joined2.includes(SUMMARY), "anchor rendered from server state"); + assert.ok(!joined2.includes("WORK_1_content_"), "client's idea of the covered region does not resurrect rows"); + assert.ok(joined2.includes("WORK_10_content_"), "uncovered snapshot rows render"); + // Server-authoritative head (#3756 doctrine): the client's checkpoint + // replacement sits BELOW the chain join, so it never renders — the + // covered region belongs to the server's own summary (anchor), and + // the uncovered head renders from the transcript rows it replaced. + assert.ok(!joined2.includes("client checkpoint"), "client-side checkpoint of the head is ignored (server-authoritative)"); + assert.equal(anchorCount(joined2), 1, "exactly one anchor copy"); + // next turn: the transcript view has adopted the client's checkpoint + // shape (one-turn adoption lag, documented in the engine header); + // the anchor still renders from block state, covered rows stay gone. + const r3 = await turn(h, h.openaiUrl, "s-drop", [...dropped, { role: "user", content: "And one more. " + "KEEP_B_".repeat(80) }], null); + assert.equal(r3.status, 200, r3.body); + const out3 = sent(h.bodies, r3.idx); + const joined3 = out3.map(textOf).join("\n"); + assert.ok(joined3.includes(SUMMARY), "anchor still rendered after checkpoint adoption"); + assert.ok(!joined3.includes("WORK_1_content_"), "covered rows stay suppressed after adoption"); + // steady after the edit: the next turn is byte-stable on the prefix + const r4 = await turn(h, h.openaiUrl, "s-drop", [...dropped, { role: "user", content: "And one more. " + "KEEP_B_".repeat(80) }, { role: "user", content: "Final follow. " + "KEEP_C_".repeat(80) }], null); + assert.equal(r4.status, 200, r4.body); + const out4 = sent(h.bodies, r4.idx); + assertPrefixStable(out4, out3, 1, "client-checkpoint"); + }, true); +}); + +test("D on: rewind — dropped tail messages do not come back", async () => { + process.env.BILI_ROLLOUT_SNAPSHOT_RENDER = "100"; + await withHarness(async (h) => { + const history = conversation(); + const r1 = await turn(h, h.openaiUrl, "s-rw", history, history); + assert.equal(r1.status, 200, r1.body); + const r2 = await turn(h, h.openaiUrl, "s-rw", [...history, { role: "user", content: "Rewind target question. " + "RW_A_".repeat(80) }], history); + assert.equal(r2.status, 200, r2.body); + // client rewinds the last exchange and asks something else + const rewound = [...conversation(), { role: "user", content: "Different question instead. " + "RW_B_".repeat(80) }]; + const r3 = await turn(h, h.openaiUrl, "s-rw", rewound); + assert.equal(r3.status, 200, r3.body); + const joined3 = sent(h.bodies, r3.idx).map(textOf).join("\n"); + assert.ok(!joined3.includes("RW_A_"), "rewound message not rendered"); + assert.ok(joined3.includes("RW_B_"), "new question present"); + assert.ok(joined3.includes(SUMMARY), "anchor still rendered"); + }, true); +}); + +test("D on: protocol switch mid-session — anchors render on the new wire, session not forked", async () => { + process.env.BILI_ROLLOUT_SNAPSHOT_RENDER = "100"; + await withHarness(async (h) => { + const history = conversation(); + const r1 = await turn(h, h.openaiUrl, "s-switch", history, history); + assert.equal(r1.status, 200, r1.body); + assert.ok(h.sessionOf("s-switch")!.state.blocks.some((b) => b.active), "block armed"); + // next turn arrives on the anthropic wire, same session header + const msgs = [...history, { role: "user", content: "Switched protocol. " + "SW_A_".repeat(80) }].map((m) => ({ + role: m.role, + content: [{ type: "text", text: m.content }], + })); + const r2 = await turn(h, h.anthropicUrl, "s-switch", msgs, null, { "anthropic-version": "2023-06-01" }); + assert.equal(r2.status, 200, r2.body); + const joined2 = sent(h.bodies, r2.idx).map(textOf).join("\n"); + assert.ok(joined2.includes(SUMMARY), "anchor renders on the anthropic wire"); + assert.ok(!joined2.includes("WORK_1_content_"), "covered rows suppressed after the switch"); + assert.ok(joined2.includes("SW_A_"), "tail present"); + const s = h.sessionOf("s-switch"); + assert.ok(s, "session survived the protocol switch (not forked away)"); + }, true); +}); + +test("D off (default): the seed-upgrade scenario churns — documents the bug this lane fixes", async () => { + delete process.env.BILI_ROLLOUT_SNAPSHOT_RENDER; + await withHarness(async (h) => { + const history = conversation(); + const r1 = await turn(h, h.openaiUrl, "s-legacy", history, history); + assert.equal(r1.status, 200, r1.body); + assert.ok(h.sessionOf("s-legacy")!.state.blocks.some((b) => b.active), "block armed"); + const tail = [{ role: "user", content: "Post-upgrade question. " + "UPG_A_".repeat(80) }]; + const r2 = await turn(h, h.openaiUrl, "s-legacy", [...history, ...tail], history); + assert.equal(r2.status, 200, r2.body); + __setMessageIdSaltForTests("upgrade-v2"); + const r3 = await turn(h, h.openaiUrl, "s-legacy", [...history, ...tail], null); + assert.equal(r3.status, 200, r3.body); + const joined3 = sent(h.bodies, r3.idx).map(textOf).join("\n"); + // legacy: coverage matching by id fails wholesale → raw rows re-render + // (or the block deactivates). Either way the covered content leaks back. + const leaked = joined3.includes("WORK_0_content_") || !joined3.includes(SUMMARY); + assert.ok(leaked, "legacy path churns on identity re-mint (the #3752 regression class)"); + assert.equal(h.sessionOf("s-legacy")!.metadata["snapshotRenderTurns"], undefined, "no D latch"); + }, true); +}); + +test("D on but storage unavailable: legacy fallback, request still served", async () => { + process.env.BILI_ROLLOUT_SNAPSHOT_RENDER = "100"; + await withHarness(async (h) => { + const history = conversation(); + const r1 = await turn(h, h.openaiUrl, "s-nostore", history, history); + assert.equal(r1.status, 200, r1.body); + const s = h.sessionOf("s-nostore"); + assert.ok(s!.state.blocks.some((b) => b.active), "block armed in memory"); + const r2 = await turn(h, h.openaiUrl, "s-nostore", [...history, { role: "user", content: "Still works. " + "NS_A_".repeat(80) }], history); + assert.equal(r2.status, 200, r2.body); + const joined2 = sent(h.bodies, r2.idx).map(textOf).join("\n"); + assert.ok(joined2.includes("NS_A_"), "served"); + assert.equal(h.sessionOf("s-nostore")!.metadata["snapshotRenderTurns"], undefined, "D inactive without rows"); + }, false); +});