diff --git a/PLUGIN.md b/PLUGIN.md index 990d684595..c00351fa19 100644 --- a/PLUGIN.md +++ b/PLUGIN.md @@ -175,7 +175,7 @@ Discover `capabilities.fork` from the manifest; do not infer support from the pa } ``` -`messages` is in exactly the same order as `orderedMessages`, describes the raw `pluginSnapshot` rather than a folded view, and includes optional `text`, `toolName`, and `toolCallId`. Compare original text and role to establish the longest matching prefix; tool identities/content type must match too. Copy the corresponding ordered identities verbatim; `identityHash` and revisions are opaque and clients do not need a content-hash algorithm. A message the kernel hard-excludes from compression (`compress.protectedTools`) carries `"ref": null` in both arrays — it participates in ordering and identity but has no addressable ref; copy it verbatim like any other prefix entry. Only `mNNNNN` strings or `null` are legal ref values. Multimodal/opaque content that cannot be verified by this text projection returns `409`, `status: unavailable`, `code: SNAPSHOT_UNAVAILABLE`, not a guessed text match. Missing raw snapshots or inconsistent raw/ref/CCR mappings likewise fail closed. Raw snapshot retention is capped (`BILI_PUBLIC_SNAPSHOT_CAP_BYTES`, default 16 MiB, `0` disables): a session whose serialized raw snapshot grows past the cap stops being forkable — snapshot/fork answer `409 SNAPSHOT_UNAVAILABLE` — instead of retaining an unbounded raw copy forever. +`messages` is in exactly the same order as `orderedMessages`, describes the raw source view (the retained `pluginSnapshot`, or the session's most recent raw request view when no retained copy exists — never a folded view), and includes optional `text`, `toolName`, and `toolCallId`. Compare original text and role to establish the longest matching prefix; tool identities/content type must match too. Copy the corresponding ordered identities verbatim; `identityHash` and revisions are opaque and clients do not need a content-hash algorithm. A message the kernel hard-excludes from compression (`compress.protectedTools`) carries `"ref": null` in both arrays — it participates in ordering and identity but has no addressable ref; copy it verbatim like any other prefix entry. Only `mNNNNN` strings or `null` are legal ref values. Multimodal/opaque content that cannot be verified by this text projection returns `409`, `status: unavailable`, `code: SNAPSHOT_UNAVAILABLE`, not a guessed text match. When neither a retained copy nor a recent raw view exists, or the raw/ref/CCR mappings are inconsistent, the endpoint likewise fails closed with `409 SNAPSHOT_UNAVAILABLE`. That code is RETRYABLE: the condition clears on the parent's next model request (and the recent-raw-view source makes it rare after any traffic at all), so clients must retry on later requests instead of giving up inheritance permanently (#3011). Raw snapshot retention is capped (`BILI_PUBLIC_SNAPSHOT_CAP_BYTES`, default 16 MiB, `0` disables): a session whose serialized raw snapshot grows past the cap stops being forkable — snapshot/fork answer `409 SNAPSHOT_UNAVAILABLE` — instead of retaining an unbounded raw copy forever. `POST /__bili/plugin/fork` body: @@ -194,7 +194,7 @@ Discover `capabilities.fork` from the manifest; do not infer support from the pa The branch is an ordered prefix, including an empty prefix. Compute its existing order hash as lowercase SHA-256 of UTF-8 `JSON.stringify(orderedMessages)` with each identity object's keys in order `rawId`, `ref`, `identityHash`; for a full prefix reuse snapshot `orderHash`. Do not hash descriptors or folded text. Extra child messages, including a final model reply that has not left the proxy yet, may be appended on the child's first model request after copying the matched prefix. -New success is `201`; exact payload replay is `200` with `replayed: true`, even after restart or later parent changes. Both return `status`, `parentRevision`, `childRevision`, `sessionId`, `branchPoint`, `inheritedBlocks`, and `expandedBlocks`. Reusing the child id with another parent, key or payload returns `409 CHILD_CONFLICT`. Invalid shapes/version/identifiers return `400 INVALID_REQUEST`; absent parent returns `404 PARENT_NOT_FOUND`; stale parent returns `409 PARENT_REVISION_CONFLICT`; altered order/raw/ref/identity returns `409 BRANCH_POINT_CONFLICT`. Missing nested originals return `409 PARENT_STATE_INCOMPLETE` with `status: unavailable`. Publication/persistence failure returns `503 FORK_FAILED` with `status: unavailable` and exposes no child; the same request may be retried. +New success is `201`; exact payload replay is `200` with `replayed: true`, even after restart or later parent changes. A child whose first stamped request landed before adoption may also complete late: the proxy registers the conversation during that request, so if the existing child has never been forked (no receipt, zero self-blocks) and its current raw view is still strictly equal to the requested prefix — same length, per-message identity, per-ref numbering — the proxy materializes the inheritance onto the existing session in place and answers `201` with `lateAdopted: true` (#3025). Both return `status`, `parentRevision`, `childRevision`, `sessionId`, `branchPoint`, `inheritedBlocks`, and `expandedBlocks`. Reusing the child id with another parent, key or payload — or a child that has since grown its own turns, folded its own history, or diverged in ref numbering — returns `409 CHILD_CONFLICT`. Invalid shapes/version/identifiers return `400 INVALID_REQUEST`; absent parent returns `404 PARENT_NOT_FOUND`; stale parent returns `409 PARENT_REVISION_CONFLICT`; altered order/raw/ref/identity returns `409 BRANCH_POINT_CONFLICT`. Missing nested originals return `409 PARENT_STATE_INCOMPLETE` with `status: unavailable`. Publication/persistence failure returns `503 FORK_FAILED` with `status: unavailable` and exposes no child; the same request may be retried. Boundary handling is `exact` when complete blocks fit inside the prefix, `expanded` when a block crosses it (crossing summaries are omitted, affected nested descendants are expanded, matched originals are retained), and `unavailable` when safe reconstruction is impossible. Blocks, nested caches, raw messages, issued ref namespace, token snapshots, rules and prefix CCR payloads are independent copies. Note that the child inherits the parent's ENTIRE issued-ref namespace (dead refs included): refs are never reused, so forks keep allocating fresh refs without colliding with ids a sibling may still hold. The child never reads its parent's later state as a fallback. Registration and first-request resume heuristics do not replace its copied state. With existing persistence enabled, raw snapshots, receipts, nested originals and the child's CCR payloads survive restart; with persistence disabled, durability is not promised. Atomic child publication/idempotency covers concurrent requests to one proxy process; concurrent writers sharing one state directory across processes are not supported by this protocol. diff --git a/src/agent/fork-adopt.ts b/src/agent/fork-adopt.ts index ee82a7c595..eb31faabc2 100644 --- a/src/agent/fork-adopt.ts +++ b/src/agent/fork-adopt.ts @@ -56,7 +56,11 @@ type SnapshotResponse = { type ForkAdoptionResult = | { outcome: "adopted"; branchPoint: number; replayed: boolean } - | { outcome: "degraded"; reason: string }; + // #3011: retryable carries the server's structured verdict (its error + // code / status class), NOT a re-parse of the reason string — the old + // regex /^(snapshot|fork) (error|http 5\d\d)/ missed the proxy's own + // documented-retryable 409 SNAPSHOT_UNAVAILABLE and gave up forever. + | { outcome: "degraded"; reason: string; retryable: boolean }; function forkStableJson(value: unknown): string { if (Array.isArray(value)) return `[${value.map(forkStableJson).join(",")}]`; @@ -387,7 +391,12 @@ async function readBoundedErrorBody(res: Response): Promise<{ code: string; erro * failure path returns { outcome: "degraded" } — the child then simply starts * a fresh conversation, exactly today's behavior; adoption is best effort. * A 409 PARENT_REVISION_CONFLICT (parent mutated between snapshot and fork) - * is retried once with a fresh snapshot before degrading. + * is retried once with a fresh snapshot before degrading. Degraded results + * carry `retryable` (#3011): whether a LATER attempt can succeed — network / + * 5xx / 409 SNAPSHOT_UNAVAILABLE are states the parent can outlive (a fresh + * parent request re-establishes its raw view); deterministic 4xx verdicts + * (unknown parent, diverged prefix, conflicting child) cannot change in this + * process. Consumers classify by the field, never by the reason text. */ export async function tryForkAdoption(opts: { base: string; @@ -404,10 +413,10 @@ export async function tryForkAdoption(opts: { // projected above with pinned parity (#2469 stage 2). const wire = detectWireFormat(opts.body); if (wire === "google") { - return { outcome: "degraded", reason: `wire unsupported (${wire})` }; + return { outcome: "degraded", reason: `wire unsupported (${wire})`, retryable: false }; } const core = chatBodyToCore(opts.body); - if (core === null || core.length === 0) return { outcome: "degraded", reason: "body unmappable" }; + if (core === null || core.length === 0) return { outcome: "degraded", reason: "body unmappable", retryable: true }; const snapshotUrl = `${opts.base}/__bili/plugin/snapshot?conversationId=${encodeURIComponent(opts.parentConversationId)}`; for (let attempt = 0; attempt < 2; attempt += 1) { let snapshot: Awaited>; @@ -418,15 +427,21 @@ export async function tryForkAdoption(opts: { // SNAPSHOT_UNAVAILABLE + why) instead of hiding them behind a // bare status code. const detail = await readBoundedErrorBody(res); - return { outcome: "degraded", reason: `snapshot http ${res.status}${detail.code ? ` ${detail.code}` : ""}${detail.error ? ` (${detail.error})` : ""}` }; + // #3011: classify by the proxy's structured verdict, not by + // re-parsing the reason string. 5xx and 409 + // SNAPSHOT_UNAVAILABLE are conditions the parent can outlive + // (a fresh parent request re-establishes its raw view); every + // other 4xx names a state this process cannot fix. + const retryable = res.status >= 500 || (res.status === 409 && detail.code === "SNAPSHOT_UNAVAILABLE"); + return { outcome: "degraded", reason: `snapshot http ${res.status}${detail.code ? ` ${detail.code}` : ""}${detail.error ? ` (${detail.error})` : ""}`, retryable }; } snapshot = snapshotOf(await res.json()); } catch (err) { - return { outcome: "degraded", reason: `snapshot error ${err instanceof Error ? err.message : String(err)}` }; + return { outcome: "degraded", reason: `snapshot error ${err instanceof Error ? err.message : String(err)}`, retryable: true }; } - if (snapshot === null) return { outcome: "degraded", reason: "snapshot malformed" }; + if (snapshot === null) return { outcome: "degraded", reason: "snapshot malformed", retryable: false }; const branchPoint = matchForkPrefix(core, snapshot.orderedMessages); - if (branchPoint === 0) return { outcome: "degraded", reason: "no prefix match" }; + if (branchPoint === 0) return { outcome: "degraded", reason: "no prefix match", retryable: false }; const prefix = snapshot.orderedMessages.slice(0, branchPoint); const orderHash = branchPoint === snapshot.orderedMessages.length ? snapshot.orderHash : forkOrderHashOf(prefix); const payload = { @@ -447,7 +462,7 @@ export async function tryForkAdoption(opts: { signal: AbortSignal.timeout(FORK_TIMEOUT_MS), }); } catch (err) { - return { outcome: "degraded", reason: `fork error ${err instanceof Error ? err.message : String(err)}` }; + return { outcome: "degraded", reason: `fork error ${err instanceof Error ? err.message : String(err)}`, retryable: true }; } if (res.status === 201 || res.status === 200) { let replayed = false; @@ -465,9 +480,12 @@ export async function tryForkAdoption(opts: { opts.log?.(`fork for ${opts.childConversationId} hit parent revision conflict, retrying once with a fresh snapshot (#2399)`); continue; } - return { outcome: "degraded", reason: `fork http ${res.status}${code ? ` ${code}` : ""}` }; + // CHILD_CONFLICT / PARENT_STATE_INCOMPLETE / BRANCH_POINT_CONFLICT and + // friends are deterministic verdicts about THIS child's state — only + // 5xx can still change under a later attempt (#3011). + return { outcome: "degraded", reason: `fork http ${res.status}${code ? ` ${code}` : ""}`, retryable: res.status >= 500 }; } - return { outcome: "degraded", reason: "parent revision conflict retry exhausted" }; + return { outcome: "degraded", reason: "parent revision conflict retry exhausted", retryable: false }; } const forkCapabilityCache = new Map(); @@ -562,15 +580,21 @@ export function createForkAdopter(log: (line: string) => void, opts?: { maxAttem done.add(input.child); return; } - log(`fork adoption for ${input.child} degraded (${result.reason}) — the session continues without inherited compression state (#2399/#2469)`); - // Only transient failures (network / 5xx / unmappable body) leave the - // window open so a later request retries up to maxAttempts; every other - // outcome is terminal. Once this child's stamped request has landed, a - // retry can only ever meet CHILD_CONFLICT (the server registers the - // child conversation before any later fork POST), which latches here as - // a non-transient outcome (#2403 review). - const transient = /^(snapshot|fork) (error|http 5\d\d)/.test(result.reason) || result.reason === "body unmappable"; - if (!transient || (attempts.get(input.child) ?? 0) >= maxAttempts) done.add(input.child); + // #3011: classify by the server's structured verdict on the result + // (retryable), never by re-parsing the reason string — the old regex + // missed 409 SNAPSHOT_UNAVAILABLE and gave up after one attempt. + // Transient outcomes leave the window open for later requests up to + // maxAttempts; every other outcome is terminal. Once this child's + // stamped request has landed, a retry can only ever meet + // CHILD_CONFLICT (the server registers the child conversation before + // any later fork POST), which latches here as non-retryable (#2403). + const exhausted = !result.retryable || (attempts.get(input.child) ?? 0) >= maxAttempts; + if (exhausted) { + log(`fork adoption for ${input.child} degraded (${result.reason}) — permanently disabled, no further retries in this process; the session continues without inherited compression state (#3011/#2399/#2469)`); + done.add(input.child); + } else { + log(`fork adoption for ${input.child} degraded (${result.reason}) — transient, will retry on a later request (attempt ${attempts.get(input.child)}/${maxAttempts}); the session continues without inherited compression state until adopted (#3011/#2399/#2469)`); + } }; return { async maybeAdopt(input): Promise { diff --git a/src/plugin.ts b/src/plugin.ts index 74e6c7038b..7fac323b04 100644 --- a/src/plugin.ts +++ b/src/plugin.ts @@ -998,13 +998,44 @@ function sessionRevisionOf(session: Session): string { return revision; } +/** #3011: pick the raw view forkSnapshot serves. pluginSnapshot is the + * canonical guarded copy (#2077), but it is only PERSISTED after a fork + * receipt or explicit retention — any residency gap (LRU evict / host + * restart) loses it until the parent's next model request, which is exactly + * when a fork child needs it most (the child's adoption window precedes its + * first forwarded request). session.lastMessages — written by EVERY prepare* + * with the raw incoming view and persisted unconditionally — covers that + * gap. It also wins when its head diverges from a stale snapshot: after a + * full-cover native compaction the zero-overlap guard freezes pluginSnapshot + * pre-compaction while the state rebase re-issues the refs, so serving the + * stale copy fails closed on "raw/ref mapping inconsistent" forever; the + * head-divergent tail is the freshest raw view the client will replay next. + * A folded tail (lastMessagesFolded) is never a raw source. An EMPTY retained + * copy is still a valid source: a zero-branchpoint fork child legitimately + * holds pluginSnapshot === [] and its own forkSnapshot() (childRevision) must + * serve that emptiness, not report "unavailable". */ +function forkSourceMessages(session: Session): { messages: CoreMessage[]; fromTail: boolean } | null { + const primary = Array.isArray(session.pluginSnapshot) ? session.pluginSnapshot : undefined; + const tail = session.lastMessages && session.lastMessagesFolded !== true && session.lastMessages.length > 0 ? session.lastMessages : undefined; + if (!primary) return tail ? { messages: tail, fromTail: true } : null; + if (tail && primary.length > 0 && forkMessageIdentityHash(tail[0]!) !== forkMessageIdentityHash(primary[0]!)) return { messages: tail, fromTail: true }; + return { messages: primary, fromTail: false }; +} + function forkSnapshot(session: Session) { - const messages = session.pluginSnapshot; - if (!messages) throw new Error(session.metadata.publicSnapshotCapped === true ? "raw snapshot exceeded the retention cap (BILI_PUBLIC_SNAPSHOT_CAP_BYTES); fork is refused rather than retaining an unbounded raw copy" : "raw snapshot unavailable; send a fresh plugin model request"); + // The retention cap stays a hard refusal even when a recent raw view is + // available: serving around it would make the cap unenforceable (#2433). + if (session.metadata.publicSnapshotCapped === true) throw new Error("raw snapshot exceeded the retention cap (BILI_PUBLIC_SNAPSHOT_CAP_BYTES); fork is refused rather than retaining an unbounded raw copy"); + const source = forkSourceMessages(session); + const messages = source?.messages; + if (!messages) throw new Error("raw snapshot unavailable (no retained raw copy and no recent raw view); send a fresh plugin model request"); if (session.metadata.publicSnapshotTextComparable === false || messages.some((m) => (m.thinkingTokens ?? 0) > 0)) throw new Error("multimodal or opaque content cannot be compared by text"); const store = contentStoreOf(session); const expectedStoredRefs = session.metadata.publicSnapshotStoredRefs; - if (Array.isArray(expectedStoredRefs) && expectedStoredRefs.some((ref) => typeof ref !== "string" || !store.byRef[ref])) throw new Error("CCR original index unavailable"); + // The stored-ref list is bookkeeping for the RETAINED copy; on the tail + // path it describes a different (older or reset) view, so only the + // per-message CCR checks below apply to what is actually served (#3011). + if (!source!.fromTail && Array.isArray(expectedStoredRefs) && expectedStoredRefs.some((ref) => typeof ref !== "string" || !store.byRef[ref])) throw new Error("CCR original index unavailable"); const indexedHashes = new Set(Object.values(store.byRef).map((entry) => entry.hash)); if (Object.entries(store.byHash).some(([hash, text]) => !indexedHashes.has(hash) || typeof text !== "string" || createHash("sha256").update(text, "utf8").digest("hex") !== hash)) throw new Error("CCR original payload/index inconsistent"); if (Object.entries(store.byRef).some(([ref, entry]) => typeof store.byHash[entry.hash] !== "string" || session.state.messageRefs.byRaw[entry.rawId] !== ref || session.state.messageRefs.byRef[ref] !== entry.rawId)) throw new Error("CCR original alias inconsistent"); @@ -1162,6 +1193,134 @@ function forkChildHeldBySameParent(child: string, parent: string): boolean { return registeredIds.get(child)?.parentConversationId === parent; } +/** #3025: materialize fork inheritance onto `child` — shared by the fresh-fork + * path (`late=false`, target just minted) and the late-completion path + * (`late=true`, an existing UNFORKED child whose stamped request landed + * before adoption and latched it behind CHILD_CONFLICT). A late child already + * ran a real request: its own rules/nudge/stats/metadata/transient + * bookkeeping stay untouched (overwriting would regress measured values) — + * what it receives is exactly the inheritance: blocks, refs, counters, raw + * baseline, content store, receipt. Returns false after replying 409 when the + * parent state cannot support the inheritance; replies 201 on success. */ +function completeForkTarget(res: ServerResponse, child: Session, parent: Session, snapshot: ReturnType, request: ForkRequest, requestHash: string, late: boolean): boolean { + const prefix = snapshot.orderedMessages.slice(0, request.branchPoint.messageCount); + const rawIds = new Set(prefix.map((m) => m.rawId)); + const crossing = parent.state.blocks.filter((b) => b.effectiveMessageIds.some((id) => rawIds.has(id)) && !b.effectiveMessageIds.every((id) => rawIds.has(id))); + const blocks = parent.state.blocks.filter((b) => b.effectiveMessageIds.length > 0 && b.effectiveMessageIds.every((id) => rawIds.has(id))); + const blockIds = new Set(blocks.map((b) => b.blockId)); + const expandedChildren = new Set(crossing.flatMap((b) => b.directBlockIds)); + for (const b of [...blocks, ...crossing]) { + const content = parent.blockContents.get(b.blockId); + if (!content || !content.full || typeof content.full.text !== "string" || !content.full.text || b.directBlockIds.some((id) => !parent.state.blocks.some((childBlock) => childBlock.blockId === id))) { forkReply(res, 409, { ok: false, status: "unavailable", code: "PARENT_STATE_INCOMPLETE", error: "nested block/original content unavailable" }); return false; } + } + for (const b of blocks) { + if (b.directBlockIds.some((id) => !blockIds.has(id))) { forkReply(res, 409, { ok: false, status: "unavailable", code: "PARENT_STATE_INCOMPLETE", error: "nested block outside matched prefix" }); return false; } + child.state.blocks.push({ ...structuredClone(b), ...(expandedChildren.has(b.blockId) ? { active: false, expanded: true } : {}) }); + child.blockContents.set(b.blockId, structuredClone(parent.blockContents.get(b.blockId)!)); + } + // Keep the issued ref namespace reserved, including dead refs, to prevent reuse. + child.state.messageRefs = structuredClone(parent.state.messageRefs); + for (const { rawId, ref } of prefix) { + // #2620: ref null = intentionally unaddressable; keep the raw id reserved under + // the kernel sentinel exactly as the parent holds it. byRef/tokenSnapshot are + // numeric-ref spaces and never see the sentinel. + child.state.messageRefs.byRaw[rawId] = ref ?? BLOCKED_REF; + if (ref !== null) { + child.state.messageRefs.byRef[ref] = rawId; + if (parent.state.tokenSnapshot[ref] !== undefined) child.state.tokenSnapshot[ref] = parent.state.tokenSnapshot[ref]; + } + } + child.state.nextBlockId = parent.state.nextBlockId; + child.state.nextRunId = parent.state.nextRunId; + child.state.lastPassIds = prefix.map((m) => m.rawId); + if (!late) child.state.rules = structuredClone(parent.state.rules); + if (!late) child.state.nextRuleId = parent.state.nextRuleId; + child.state.hiddenOrphanRefs = parent.state.hiddenOrphanRefs?.filter((ref) => prefix.some((m) => m.ref === ref)); + // #2362: inverse of the orphan filter — a dead ref whose message IS + // in the forked prefix is visible again in the child; refs outside + // it stay dead (their messages are absent from the child too). + child.state.deadRefs = parent.state.deadRefs?.filter((ref) => !prefix.some((m) => m.ref === ref)); + child.state.stats.tokensCompressed = child.state.blocks.filter((b) => b.active).reduce((sum, b) => sum + b.compressedTokens, 0); + child.state.stats.compressionCount = blocks.length; + if (!late && prefix.length === snapshot.orderedMessages.length) child.state.nudge = structuredClone(parent.state.nudge); + // #3011: the child's raw baseline must come from the SAME source + // view the snapshot was served from — the retained copy may be + // absent (residency gap) while the recent raw view is not. + const parentSource = forkSourceMessages(parent); + if (!parentSource) { forkReply(res, 409, { ok: false, status: "unavailable", code: "SNAPSHOT_UNAVAILABLE", error: "raw snapshot unavailable; send a fresh plugin model request" }); return false; } + child.pluginSnapshot = structuredClone(parentSource.messages.slice(0, request.branchPoint.messageCount)); + const forkView = prune(child.pluginSnapshot, child.state); + // Hiding the original pair is safe only if its summary survives the fork. + child.state.absorbed = structuredClone(parent.state.absorbed?.filter((a) => + rawIds.has(a.callMessageId) && rawIds.has(a.resultMessageId) && ( + (a.absorbCallId !== undefined && forkView.some((m) => m.contentType === "tool-call" && m.toolCallId === a.absorbCallId)) || + child.state.blocks.some((b) => b.active && !b.expanded && a.summary.length > 0 && b.summary.includes(a.summary)) + )) ?? []); + child.lastMessages = structuredClone(child.pluginSnapshot); + child.lastMessagesFolded = false; + const parentStore = contentStoreOf(parent); + for (const { ref, rawId } of prefix) { + if (ref === null) continue; + const entry = parentStore.byRef[ref]; + if (entry && (entry.rawId !== rawId || typeof parentStore.byHash[entry.hash] !== "string")) { forkReply(res, 409, { ok: false, status: "unavailable", code: "PARENT_STATE_INCOMPLETE", error: "CCR original unavailable or alias inconsistent" }); return false; } + } + child.contentStore = cloneStoreForRefs(parentStore, new Set(prefix.filter((m) => m.ref !== null).map((m) => m.ref!))) ?? undefined; + child.metadata.publicSnapshotTextComparable = true; + child.metadata.publicSnapshotStoredRefs = Object.keys(child.contentStore?.byRef ?? {}); + child.metadata.parentConversationId = request.parentConversationId; + child.metadata.parentRevision = request.parentRevision; + if (!late) { + // Fresh children start cold: clone the parent's resolution context and + // seed the nudge sizing fallback (#728 lane). The receipt starts the + // child on a FRESH stats block: no usage-grade baseline (billing + // isolation is deliberate) and no per-turn localInputEstimate yet, so + // effectiveTokenCount feeds the kernel tokenCount 0 and every nudge + // trigger goes dark on the child's first turn — exactly when the + // inherited context is at its fullest. The receipt's own estimate + // describes the child's exact next outbound view: seeding it keeps + // the nudge live from turn 1. It errs EARLY only (char-count upper + // bound), a real usage report takes precedence immediately (#728 + // invariant), and every prepare* overwrites it with the measured + // outbound each turn. Late children skip all of this: they already + // ran a real request and own valid measured values. + for (const key of ["pluginAgent", "lastModel", "effectiveConfig", "effectiveCcr", "effectiveContextLimit", "lastWindowSource", "systemPromptTokens"]) if (parent.metadata[key] !== undefined) child.metadata[key] = structuredClone(parent.metadata[key]); + child.stats.contextTokens = prune(child.pluginSnapshot, child.state).reduce((sum, m) => sum + countMessageTokens(m), 0) + (typeof child.metadata.systemPromptTokens === "number" ? child.metadata.systemPromptTokens : 0); + child.stats.contextTokensSource = "estimate"; + recordContextObservation(child, child.stats.contextTokens, "estimate"); + child.stats.localInputEstimate = child.stats.contextTokens; + } + const response = { ok: true, status: crossing.length > 0 ? "expanded" : "exact", protocolVersion: 2, parentConversationId: request.parentConversationId, childConversationId: request.childConversationId, sessionId: child.id, parentRevision: request.parentRevision, childRevision: forkSnapshot(child).parentRevision, branchPoint: request.branchPoint, expandedBlocks: crossing.map((b) => b.blockId), inheritedBlocks: child.state.blocks.map((b) => ({ id: b.blockId, tier: b.tier, active: b.active, expanded: b.expanded === true })), replayed: false, ...(late ? { lateAdopted: true } : {}) }; + child.metadata.publicForkReceipt = { requestHash, response }; + if (late) { + // Late children are ALREADY RESIDENT (they carried traffic): the + // fresh-session guard in publishForkSession would reject them. Flush + // eagerly instead so the receipt and the materialized inheritance + // land in one write (#2077 continuity — a crash between the two would + // leave blocks without a receipt and re-latch the 409). + if (!getStore().flushSync(child)) loggerLog("warn", `fork ${request.childConversationId}: late-adopted child flush failed (${child.id}); the debounced save will retry`); + } else { + publishForkSession(child); + } + recordPluginSession(request.childConversationId, child.id); + // [#2399 stage 2] the host's session_start identity register claimed + // this id with the same parent (see forkChildHeldBySameParent) — the + // fork-created conversation supersedes it; a lingering entry would + // re-bind plugin mode on the child's next request, which the map + // binding already does without dragging the derived-parent path in. + registeredIds.delete(request.childConversationId); + if (!late) (child.transient ??= {}).pluginMessages = { processed: structuredClone(child.pluginSnapshot), original: structuredClone(child.pluginSnapshot) }; + // #2077: the parent snapshot is now an external contract — this child + // was cut from it, and later forks/replays depend on its continuity. + // The receipt lives on the CHILD only, so without this sticky flag a + // lazily-persisted parent would drop its snapshot on disk and the next + // restart would 409. Flush eagerly (publishForkSession precedent) so + // the flag and the snapshot land in one write. + parent.metadata.publicSnapshotRetained = true; + if (!getStore().flushSync(parent)) loggerLog("warn", `fork ${request.childConversationId}: parent snapshot flush failed (${parent.id}); the debounced save will retry`); + forkReply(res, 201, response); + return true; +} + export async function handlePluginFork(payload: string, res: ServerResponse): Promise { let request: ForkRequest; try { request = parseForkRequest(payload); } @@ -1171,19 +1330,33 @@ export async function handlePluginFork(payload: string, res: ServerResponse): Pr const existing = resolveForkConversation(request.childConversationId); if (!existing) return false; const receipt = existing.metadata.publicForkReceipt as { requestHash?: unknown; response?: unknown } | undefined; - if (receipt?.requestHash === requestHash) forkReply(res, 200, { ...(receipt.response as Record), replayed: true }); - else forkReply(res, 409, { ok: false, code: "CHILD_CONFLICT", error: "child conversation already exists or idempotency payload differs" }); - return true; + if (receipt?.requestHash === requestHash) { forkReply(res, 200, { ...(receipt.response as Record), replayed: true }); return true; } + // #3025: an existing child WITHOUT a matching receipt is not settled + // here — it may be a stamped-first race victim still eligible for + // in-place late completion. The locked section decides under both + // locks; genuine conflicts still end in CHILD_CONFLICT. + return false; }; if (replay()) return; - if (conversations.has(request.childConversationId) || pendingRegisters.some((r) => r.conversationId === request.childConversationId) || (registeredIds.has(request.childConversationId) && !forkChildHeldBySameParent(request.childConversationId, request.parentConversationId))) return forkReply(res, 409, { ok: false, code: "CHILD_CONFLICT", error: "child conversation already registered" }); + // #3025: recordPluginSession runs during the child's FIRST stamped request + // (before forwarding), so a race-lost adoption latches the child behind + // the registration gates below forever. An UNFORKED child (no receipt, + // never materialized, zero self-blocks) may bypass them — the strict + // eligibility re-check happens under both locks. Eligibility only ever + // degrades across the window, so the cheap pre-lock peek is sound. + const lateCandidate = (): boolean => { + const candidate = resolveForkConversation(request.childConversationId); + return !!candidate && candidate.metadata.publicForkReceipt === undefined && candidate.metadata.parentConversationId === undefined && candidate.state.blocks.length === 0; + }; + const registrationConflict = (): boolean => conversations.has(request.childConversationId) || pendingRegisters.some((r) => r.conversationId === request.childConversationId) || (registeredIds.has(request.childConversationId) && !forkChildHeldBySameParent(request.childConversationId, request.parentConversationId)); + if (!lateCandidate() && registrationConflict()) return forkReply(res, 409, { ok: false, code: "CHILD_CONFLICT", error: "child conversation already registered" }); const parent = resolveForkConversation(request.parentConversationId); if (!parent) return forkReply(res, 404, { ok: false, code: "PARENT_NOT_FOUND", error: "parent conversation not found" }); acquireInFlight(parent); try { - await withSessionLock(parent, () => { + await withSessionLock(parent, async () => { if (replay()) return; - if (conversations.has(request.childConversationId) || pendingRegisters.some((r) => r.conversationId === request.childConversationId) || (registeredIds.has(request.childConversationId) && !forkChildHeldBySameParent(request.childConversationId, request.parentConversationId))) return forkReply(res, 409, { ok: false, code: "CHILD_CONFLICT", error: "child conversation already registered" }); + if (!lateCandidate() && registrationConflict()) return forkReply(res, 409, { ok: false, code: "CHILD_CONFLICT", error: "child conversation already registered" }); let snapshot: ReturnType; try { snapshot = forkSnapshot(parent); } catch (err) { return forkReply(res, 409, { ok: false, status: "unavailable", code: "SNAPSHOT_UNAVAILABLE", error: String(err) }); } @@ -1197,107 +1370,37 @@ export async function handlePluginFork(payload: string, res: ServerResponse): Pr } const prefix = snapshot.orderedMessages.slice(0, request.branchPoint.messageCount); if (prefix.length !== request.branchPoint.messageCount || forkOrderHash(request.orderedMessages) !== request.branchPoint.orderHash || stableJson(prefix) !== stableJson(request.orderedMessages)) return forkReply(res, 409, { ok: false, code: "BRANCH_POINT_CONFLICT", error: "ordered prefix/hash does not match parent" }); - const rawIds = new Set(prefix.map((m) => m.rawId)); - const child = createSession(request.childConversationId, { ...parent.meta, label: request.childConversationId }); - const crossing = parent.state.blocks.filter((b) => b.effectiveMessageIds.some((id) => rawIds.has(id)) && !b.effectiveMessageIds.every((id) => rawIds.has(id))); - const blocks = parent.state.blocks.filter((b) => b.effectiveMessageIds.length > 0 && b.effectiveMessageIds.every((id) => rawIds.has(id))); - const blockIds = new Set(blocks.map((b) => b.blockId)); - const expandedChildren = new Set(crossing.flatMap((b) => b.directBlockIds)); - for (const b of [...blocks, ...crossing]) { - const content = parent.blockContents.get(b.blockId); - if (!content || !content.full || typeof content.full.text !== "string" || !content.full.text || b.directBlockIds.some((id) => !parent.state.blocks.some((childBlock) => childBlock.blockId === id))) return forkReply(res, 409, { ok: false, status: "unavailable", code: "PARENT_STATE_INCOMPLETE", error: "nested block/original content unavailable" }); - } - for (const b of blocks) { - if (b.directBlockIds.some((id) => !blockIds.has(id))) return forkReply(res, 409, { ok: false, status: "unavailable", code: "PARENT_STATE_INCOMPLETE", error: "nested block outside matched prefix" }); - child.state.blocks.push({ ...structuredClone(b), ...(expandedChildren.has(b.blockId) ? { active: false, expanded: true } : {}) }); - child.blockContents.set(b.blockId, structuredClone(parent.blockContents.get(b.blockId)!)); - } - // Keep the issued ref namespace reserved, including dead refs, to prevent reuse. - child.state.messageRefs = structuredClone(parent.state.messageRefs); - for (const { rawId, ref } of prefix) { - // #2620: ref null = intentionally unaddressable; keep the raw id reserved under - // the kernel sentinel exactly as the parent holds it. byRef/tokenSnapshot are - // numeric-ref spaces and never see the sentinel. - child.state.messageRefs.byRaw[rawId] = ref ?? BLOCKED_REF; - if (ref !== null) { - child.state.messageRefs.byRef[ref] = rawId; - if (parent.state.tokenSnapshot[ref] !== undefined) child.state.tokenSnapshot[ref] = parent.state.tokenSnapshot[ref]; + const lateChild = resolveForkConversation(request.childConversationId); + if (lateChild) { + // #3025 late completion: the child's stamped request landed before + // adoption. Complete the inheritance IN PLACE only while the child + // is provably still the pure prefix — strict full-length identity + // (any own turn or self-fold disqualifies) plus a per-ref + // consistency assertion: numbering divergence means refuse, never + // corrupt. Re-checked after taking the child lock: state may have + // moved meanwhile. Parent → child is the only two-lock order in + // the proxy, so this cannot deadlock. + const eligible = (): boolean => lateChild.metadata.publicForkReceipt === undefined + && lateChild.metadata.parentConversationId === undefined + && lateChild.state.blocks.length === 0 + && Array.isArray(lateChild.lastMessages) + && lateChild.lastMessages.length === request.orderedMessages.length + && lateChild.lastMessages.every((m, i) => forkMessageIdentityHash(m) === request.orderedMessages[i]!.identityHash) + && request.orderedMessages.every(({ rawId, ref }) => lateChild.state.messageRefs.byRaw[rawId] === (ref ?? BLOCKED_REF)); + if (!eligible()) return forkReply(res, 409, { ok: false, code: "CHILD_CONFLICT", error: "child conversation already exists or idempotency payload differs" }); + acquireInFlight(lateChild); + try { + await withSessionLock(lateChild, () => { + if (!eligible()) return forkReply(res, 409, { ok: false, code: "CHILD_CONFLICT", error: "child conversation already exists or idempotency payload differs" }); + completeForkTarget(res, lateChild, parent, snapshot, request, requestHash, true); + }); + } finally { + releaseInFlight(lateChild); } + return; } - child.state.nextBlockId = parent.state.nextBlockId; - child.state.nextRunId = parent.state.nextRunId; - child.state.lastPassIds = prefix.map((m) => m.rawId); - child.state.rules = structuredClone(parent.state.rules); - child.state.nextRuleId = parent.state.nextRuleId; - child.state.hiddenOrphanRefs = parent.state.hiddenOrphanRefs?.filter((ref) => prefix.some((m) => m.ref === ref)); - // #2362: inverse of the orphan filter — a dead ref whose message IS - // in the forked prefix is visible again in the child; refs outside - // it stay dead (their messages are absent from the child too). - child.state.deadRefs = parent.state.deadRefs?.filter((ref) => !prefix.some((m) => m.ref === ref)); - child.state.stats.tokensCompressed = child.state.blocks.filter((b) => b.active).reduce((sum, b) => sum + b.compressedTokens, 0); - child.state.stats.compressionCount = blocks.length; - if (prefix.length === snapshot.orderedMessages.length) child.state.nudge = structuredClone(parent.state.nudge); - child.pluginSnapshot = structuredClone(parent.pluginSnapshot!.slice(0, request.branchPoint.messageCount)); - const forkView = prune(child.pluginSnapshot, child.state); - // Hiding the original pair is safe only if its summary survives the fork. - child.state.absorbed = structuredClone(parent.state.absorbed?.filter((a) => - rawIds.has(a.callMessageId) && rawIds.has(a.resultMessageId) && ( - (a.absorbCallId !== undefined && forkView.some((m) => m.contentType === "tool-call" && m.toolCallId === a.absorbCallId)) || - child.state.blocks.some((b) => b.active && !b.expanded && a.summary.length > 0 && b.summary.includes(a.summary)) - )) ?? []); - child.lastMessages = structuredClone(child.pluginSnapshot); - child.lastMessagesFolded = false; - const parentStore = contentStoreOf(parent); - for (const { ref, rawId } of prefix) { - if (ref === null) continue; - const entry = parentStore.byRef[ref]; - if (entry && (entry.rawId !== rawId || typeof parentStore.byHash[entry.hash] !== "string")) return forkReply(res, 409, { ok: false, status: "unavailable", code: "PARENT_STATE_INCOMPLETE", error: "CCR original unavailable or alias inconsistent" }); - } - child.contentStore = cloneStoreForRefs(parentStore, new Set(prefix.filter((m) => m.ref !== null).map((m) => m.ref!))) ?? undefined; - child.metadata.publicSnapshotTextComparable = true; - child.metadata.publicSnapshotStoredRefs = Object.keys(child.contentStore?.byRef ?? {}); - for (const key of ["pluginAgent", "lastModel", "effectiveConfig", "effectiveCcr", "effectiveContextLimit", "lastWindowSource", "systemPromptTokens"]) if (parent.metadata[key] !== undefined) child.metadata[key] = structuredClone(parent.metadata[key]); - child.metadata.parentConversationId = request.parentConversationId; - child.metadata.parentRevision = request.parentRevision; - child.stats.contextTokens = prune(child.pluginSnapshot, child.state).reduce((sum, m) => sum + countMessageTokens(m), 0) + (typeof child.metadata.systemPromptTokens === "number" ? child.metadata.systemPromptTokens : 0); - child.stats.contextTokensSource = "estimate"; - recordContextObservation(child, child.stats.contextTokens, "estimate"); - // Seed the nudge sizing fallback (#728 lane). The receipt starts - // the child on a FRESH stats block: no usage-grade baseline (billing - // isolation is deliberate) and no per-turn localInputEstimate yet, - // so effectiveTokenCount feeds the kernel tokenCount 0 and every - // nudge trigger goes dark on the child's first turn — exactly when - // the inherited context is at its fullest. A mid-history model - // switch onto a smaller window then falls straight into the - // preflight summarization marathon on every request (field repro: - // dsh fork at branch point 93, 671 msgs, "nudge idle: usage=0% - // (0/936000)" with 929710 pending, 19-round preflight per request). - // The receipt's own estimate describes the child's exact next - // outbound view: seeding it keeps the nudge live from turn 1. It - // errs EARLY only (char-count upper bound), a real usage report - // takes precedence immediately (#728 invariant), and every prepare* - // overwrites it with the measured outbound each turn. - child.stats.localInputEstimate = child.stats.contextTokens; - const response = { ok: true, status: crossing.length > 0 ? "expanded" : "exact", protocolVersion: 2, parentConversationId: request.parentConversationId, childConversationId: request.childConversationId, sessionId: child.id, parentRevision: request.parentRevision, childRevision: forkSnapshot(child).parentRevision, branchPoint: request.branchPoint, expandedBlocks: crossing.map((b) => b.blockId), inheritedBlocks: child.state.blocks.map((b) => ({ id: b.blockId, tier: b.tier, active: b.active, expanded: b.expanded === true })), replayed: false }; - child.metadata.publicForkReceipt = { requestHash, response }; - publishForkSession(child); - recordPluginSession(request.childConversationId, child.id); - // [#2399 stage 2] the host's session_start identity register claimed - // this id with the same parent (see forkChildHeldBySameParent) — the - // fork-created conversation supersedes it; a lingering entry would - // re-bind plugin mode on the child's next request, which the map - // binding already does without dragging the derived-parent path in. - registeredIds.delete(request.childConversationId); - (child.transient ??= {}).pluginMessages = { processed: structuredClone(child.pluginSnapshot), original: structuredClone(child.pluginSnapshot) }; - // #2077: the parent snapshot is now an external contract — this child - // was cut from it, and later forks/replays depend on its continuity. - // The receipt lives on the CHILD only, so without this sticky flag a - // lazily-persisted parent would drop its snapshot on disk and the next - // restart would 409. Flush eagerly (publishForkSession precedent) so - // the flag and the snapshot land in one write. - parent.metadata.publicSnapshotRetained = true; - if (!getStore().flushSync(parent)) loggerLog("warn", `fork ${request.childConversationId}: parent snapshot flush failed (${parent.id}); the debounced save will retry`); - forkReply(res, 201, response); + const child = createSession(request.childConversationId, { ...parent.meta, label: request.childConversationId }); + completeForkTarget(res, child, parent, snapshot, request, requestHash, false); }); } catch (err) { forkReply(res, 503, { ok: false, status: "unavailable", code: "FORK_FAILED", error: String(err) }); diff --git a/src/session.ts b/src/session.ts index dc64f7c0cf..5304d01c6c 100644 --- a/src/session.ts +++ b/src/session.ts @@ -996,6 +996,14 @@ export function resetSessionCompression(session: Session): void { // deletes the on-disk envelope on the next save. session.contentStore = undefined; session.contentStoreDirty = true; + // #3011: the rebase re-issues refs and wipes the CCR store — the retained + // snapshot's bookkeeping (stored-ref list, wire comparability verdict, + // tracked byte count) describes content whose mapping no longer exists; + // each field re-evaluates on the parent's next request. + delete session.metadata.publicSnapshotStoredRefs; + delete session.metadata.publicSnapshotWireHash; + delete session.metadata.publicSnapshotBytes; + delete session.metadata.publicSnapshotTextComparable; // #1343: rebase wipes both the queued injections and their acks (context // rebuilt), so clear the durable ledger too — otherwise a later // reconcileReloadedRetrievals would misattribute this expected loss as a diff --git a/tests/dsh-fork-adopt.test.ts b/tests/dsh-fork-adopt.test.ts index ad0eed8d39..62867e8e9d 100644 --- a/tests/dsh-fork-adopt.test.ts +++ b/tests/dsh-fork-adopt.test.ts @@ -209,16 +209,16 @@ test("tryForkAdoption never posts a fork when the prefix does not match", async const calls: Recorded[] = []; const fetchImpl = routerFetch({ snapshot: () => jsonRes(snapshotFixture(seedCore)) }, calls); const result = await tryForkAdoption({ base: "http://proxy", parentConversationId: "parent", childConversationId: "child", body: bodyFor([{ role: "user", content: "unrelated history" }]), fetchImpl }); - assert.deepEqual(result, { outcome: "degraded", reason: "no prefix match" }); + assert.deepEqual(result, { outcome: "degraded", reason: "no prefix match", retryable: false }); assert.equal(calls.some((c) => c.url.includes("/__bili/plugin/fork")), false); }); test("tryForkAdoption degrades on snapshot failures without posting", async () => { const calls: Recorded[] = []; const notFound = routerFetch({ snapshot: () => new Response("{}", { status: 404 }) }, calls); - assert.deepEqual(await tryForkAdoption({ base: "http://proxy", parentConversationId: "parent", childConversationId: "child", body: parentBody, fetchImpl: notFound }), { outcome: "degraded", reason: "snapshot http 404" }); + assert.deepEqual(await tryForkAdoption({ base: "http://proxy", parentConversationId: "parent", childConversationId: "child", body: parentBody, fetchImpl: notFound }), { outcome: "degraded", reason: "snapshot http 404", retryable: false }); const malformed = routerFetch({ snapshot: () => jsonRes({ ok: true, orderedMessages: "nope" }) }, calls); - assert.deepEqual(await tryForkAdoption({ base: "http://proxy", parentConversationId: "parent", childConversationId: "child", body: parentBody, fetchImpl: malformed }), { outcome: "degraded", reason: "snapshot malformed" }); + assert.deepEqual(await tryForkAdoption({ base: "http://proxy", parentConversationId: "parent", childConversationId: "child", body: parentBody, fetchImpl: malformed }), { outcome: "degraded", reason: "snapshot malformed", retryable: false }); const throwing = (async () => { throw new Error("boom"); }) as unknown as typeof fetch; const degraded = await tryForkAdoption({ base: "http://proxy", parentConversationId: "parent", childConversationId: "child", body: parentBody, fetchImpl: throwing }); assert.equal(degraded.outcome, "degraded"); @@ -230,7 +230,11 @@ test("tryForkAdoption surfaces bounded snapshot diagnostics instead of a bare st const calls: Recorded[] = []; const unavailable = routerFetch({ snapshot: () => new Response(JSON.stringify({ ok: false, status: "unavailable", code: "SNAPSHOT_UNAVAILABLE", error: "Error: multimodal or opaque content cannot be compared by text" }), { status: 409, headers: { "content-type": "application/json" } }) }, calls); const result = await tryForkAdoption({ base: "http://proxy", parentConversationId: "parent", childConversationId: "child", body: parentBody, fetchImpl: unavailable }); - assert.deepEqual(result, { outcome: "degraded", reason: "snapshot http 409 SNAPSHOT_UNAVAILABLE (Error: multimodal or opaque content cannot be compared by text)" }); + // Field-based classification (#3011): the client cannot tell a residency-gap + // 409 from a multimodal 409 by code alone — over-retrying burns at most + // maxAttempts cheap GETs before the explicit permanent-disable log, while + // under-retrying resurrects the original bug. + assert.deepEqual(result, { outcome: "degraded", reason: "snapshot http 409 SNAPSHOT_UNAVAILABLE (Error: multimodal or opaque content cannot be compared by text)", retryable: true }); assert.equal(calls.length, 1, "one snapshot attempt, no fork POST"); assert.equal(calls.every((c) => !c.url.includes("/__bili/plugin/fork")), true); }); @@ -250,11 +254,11 @@ test("tryForkAdoption sanitizes control characters out of snapshot diagnostics ( test("tryForkAdoption degrades safely on non-JSON or oversized snapshot error bodies (#2469)", async () => { const nonJson = routerFetch({ snapshot: () => new Response("gateway exploded", { status: 409 }) }, []); - assert.deepEqual(await tryForkAdoption({ base: "http://proxy", parentConversationId: "parent", childConversationId: "child", body: parentBody, fetchImpl: nonJson }), { outcome: "degraded", reason: "snapshot http 409" }); + assert.deepEqual(await tryForkAdoption({ base: "http://proxy", parentConversationId: "parent", childConversationId: "child", body: parentBody, fetchImpl: nonJson }), { outcome: "degraded", reason: "snapshot http 409", retryable: false }); // A body past the 1KB read cap is truncated mid-JSON: parsing must fail // closed to the status-only reason, never hang or buffer unboundedly. const oversized = routerFetch({ snapshot: () => new Response(JSON.stringify({ ok: false, code: "BIG", error: "x".repeat(5000) }), { status: 409 }) }, []); - assert.deepEqual(await tryForkAdoption({ base: "http://proxy", parentConversationId: "parent", childConversationId: "child", body: parentBody, fetchImpl: oversized }), { outcome: "degraded", reason: "snapshot http 409" }); + assert.deepEqual(await tryForkAdoption({ base: "http://proxy", parentConversationId: "parent", childConversationId: "child", body: parentBody, fetchImpl: oversized }), { outcome: "degraded", reason: "snapshot http 409", retryable: false }); }); test("tryForkAdoption ends google children before any network request; responses go through the protocol (#2469)", async () => { @@ -263,20 +267,20 @@ test("tryForkAdoption ends google children before any network request; responses const responsesCalls: Recorded[] = []; const responsesFetch = routerFetch({ snapshot: () => jsonRes(snapshotFixture(seedCore)), fork: () => jsonRes({ ok: true }, 201) }, responsesCalls); const responses = await tryForkAdoption({ base: "http://proxy", parentConversationId: "parent", childConversationId: "child", body: { model: "gpt", input: [{ type: "message", role: "user", content: "hi" }] }, fetchImpl: responsesFetch }); - assert.deepEqual(responses, { outcome: "degraded", reason: "no prefix match" }); + assert.deepEqual(responses, { outcome: "degraded", reason: "no prefix match", retryable: false }); assert.equal(responsesCalls.filter((c) => c.url.includes("snapshot")).length, 1, "the responses child reaches the snapshot"); assert.equal(responsesCalls.filter((c) => c.url.endsWith("/__bili/plugin/fork")).length, 0, "...but posts no fork receipt for a mismatch"); const googleCalls: Recorded[] = []; const googleFetch = routerFetch({ snapshot: () => jsonRes(snapshotFixture(seedCore)) }, googleCalls); const google = await tryForkAdoption({ base: "http://proxy", parentConversationId: "parent", childConversationId: "child", body: { contents: [{ role: "user", parts: [{ text: "hi" }] }] }, fetchImpl: googleFetch }); - assert.deepEqual(google, { outcome: "degraded", reason: "wire unsupported (google)" }); + assert.deepEqual(google, { outcome: "degraded", reason: "wire unsupported (google)", retryable: false }); assert.equal(googleCalls.length, 0); }); test("tryForkAdoption surfaces fork rejections and replays", async () => { const conflict = routerFetch({ snapshot: () => jsonRes(snapshotFixture(seedCore)), fork: () => jsonRes({ ok: false, code: "CHILD_CONFLICT" }, 409) }, []); const rejected = await tryForkAdoption({ base: "http://proxy", parentConversationId: "parent", childConversationId: "child", body: parentBody, fetchImpl: conflict }); - assert.deepEqual(rejected, { outcome: "degraded", reason: "fork http 409 CHILD_CONFLICT" }); + assert.deepEqual(rejected, { outcome: "degraded", reason: "fork http 409 CHILD_CONFLICT", retryable: false }); const replay = routerFetch({ snapshot: () => jsonRes(snapshotFixture(seedCore)), fork: () => jsonRes({ ok: true, replayed: true }, 200) }, []); const replayed = await tryForkAdoption({ base: "http://proxy", parentConversationId: "parent", childConversationId: "child", body: parentBody, fetchImpl: replay }); assert.deepEqual(replayed, { outcome: "adopted", branchPoint: 3, replayed: true }); diff --git a/tests/pi-omp-fork-adopt.test.ts b/tests/pi-omp-fork-adopt.test.ts index f7a54ab570..e747037bd2 100644 --- a/tests/pi-omp-fork-adopt.test.ts +++ b/tests/pi-omp-fork-adopt.test.ts @@ -255,6 +255,68 @@ test("createForkAdopter retries transient failures until the cap, then closes (# assert.equal(calls.filter((c) => c.url.endsWith("/__bili/plugin/fork")).length, 3, "past the cap the window is closed"); }); +test("tryForkAdoption classifies snapshot failures by the server's structured verdict (#3011)", async () => { + const input = adoptInput(); + const run = (respond: () => Response | Promise) => tryForkAdoption({ base: input.base, parentConversationId: input.parent, childConversationId: input.child, body: input.body, fetchImpl: fakeFetchRouting([manifestRoute, { match: snapshotRoute.match, respond }]).fetchImpl }); + const unavailable = (code: string, status = 409) => new Response(JSON.stringify({ ok: false, status: "unavailable", code, error: "raw snapshot unavailable; send a fresh plugin model request" }), { status }); + // The incident: the proxy's own documented-retryable 409 was classified + // terminal by the old regex and gave up after one attempt. + const asDegraded = async (p: Promise>>) => { const r = await p; if (r.outcome !== "degraded") throw new Error(`expected degraded, got ${JSON.stringify(r)}`); return r; }; + assert.deepEqual(await run(() => unavailable("SNAPSHOT_UNAVAILABLE")), { outcome: "degraded", reason: "snapshot http 409 SNAPSHOT_UNAVAILABLE (raw snapshot unavailable; send a fresh plugin model request)", retryable: true }); + assert.equal((await asDegraded(run(() => unavailable("PARENT_NOT_FOUND", 404)))).retryable, false, "an unknown parent cannot appear mid-process"); + assert.equal((await asDegraded(run(() => unavailable("BRANCH_POINT_CONFLICT")))).retryable, false, "other 409 codes are deterministic verdicts"); + const s503 = await asDegraded(run(() => new Response("upstream melted", { status: 503 }))); + assert.equal(s503.retryable, true, "5xx stays retryable"); + const boom = await asDegraded(run(async () => { throw new Error("network exploded"); })); + assert.equal(boom.retryable, true, "network errors stay retryable"); +}); + +test("tryForkAdoption classifies fork POST failures by status class (#3011)", async () => { + const input = adoptInput(); + const core = openaiBodyToCore(input.body as Record)!; + const matchSnapshot = { match: snapshotRoute.match, respond: () => Response.json({ ok: true, protocolVersion: 1, status: "exact", parentRevision: hash("rev1"), orderHash: hash("full"), orderedMessages: [{ rawId: "a", ref: "m00001", identityHash: forkIdentityHashOf(core[0]!) }] }) }; + const run = (respond: () => Response) => tryForkAdoption({ base: input.base, parentConversationId: input.parent, childConversationId: input.child, body: input.body, fetchImpl: fakeFetchRouting([manifestRoute, matchSnapshot, { match: (c: Recorded) => c.url.endsWith("/__bili/plugin/fork"), respond }]).fetchImpl }); + const conflict = await run(() => new Response(JSON.stringify({ ok: false, code: "CHILD_CONFLICT", error: "child conversation already registered" }), { status: 409 })); + if (conflict.outcome !== "degraded") throw new Error(`expected degraded, got ${JSON.stringify(conflict)}`); + assert.equal(conflict.retryable, false, "CHILD_CONFLICT latches — this child keeps its id, so the verdict cannot change"); + assert.match(conflict.reason, /fork http 409 CHILD_CONFLICT/); + const fork503 = await run(() => new Response("boom", { status: 503 })); + if (fork503.outcome !== "degraded") throw new Error(`expected degraded, got ${JSON.stringify(fork503)}`); + assert.equal(fork503.retryable, true, "fork 5xx stays retryable"); +}); + +test("createForkAdopter retries 409 SNAPSHOT_UNAVAILABLE until the parent recovers (#3011)", async () => { + resetForkCapabilityCacheForTest(); + const core = openaiBodyToCore(adoptInput().body as Record)!; + const recovered = { match: snapshotRoute.match, respond: () => Response.json({ ok: true, protocolVersion: 1, status: "exact", parentRevision: hash("rev1"), orderHash: hash("full"), orderedMessages: [{ rawId: "a", ref: "m00001", identityHash: forkIdentityHashOf(core[0]!) }] }) }; + let snapshots = 0; + const flaky = { match: snapshotRoute.match, respond: () => { snapshots += 1; return snapshots <= 2 ? new Response(JSON.stringify({ ok: false, status: "unavailable", code: "SNAPSHOT_UNAVAILABLE", error: "raw snapshot unavailable; send a fresh plugin model request" }), { status: 409 }) : recovered.respond(); } }; + const { fetchImpl, calls } = fakeFetchRouting([manifestRoute, flaky, forkRoute]); + const lines: string[] = []; + const adopter = createForkAdopter((l) => lines.push(l), { fetchImpl }); + await adopter.maybeAdopt(adoptInput()); + await adopter.maybeAdopt(adoptInput()); + assert.equal(lines.filter((l) => l.includes("transient, will retry")).length, 2, "each residency-gap 409 leaves the window open"); + await adopter.maybeAdopt(adoptInput()); + assert.equal(calls.filter((c) => c.url.endsWith("/__bili/plugin/fork")).length, 1, "the recovered parent adopts on the third request"); + assert.ok(!lines.some((l) => l.includes("permanently disabled"))); +}); + +test("createForkAdopter closes permanently after the cap and says so explicitly (#3011)", async () => { + resetForkCapabilityCacheForTest(); + const unavailable = () => new Response(JSON.stringify({ ok: false, status: "unavailable", code: "SNAPSHOT_UNAVAILABLE", error: "raw snapshot unavailable; send a fresh plugin model request" }), { status: 409 }); + const { fetchImpl, calls } = fakeFetchRouting([manifestRoute, { match: snapshotRoute.match, respond: unavailable }, forkRoute]); + const lines: string[] = []; + const adopter = createForkAdopter((l) => lines.push(l), { fetchImpl }); + for (let i = 0; i < 4; i += 1) await adopter.maybeAdopt(adoptInput()); + assert.equal(calls.filter((c) => c.url.includes("snapshot")).length, 3, "maxAttempts=3 bounds the retries"); + assert.equal(calls.filter((c) => c.url.endsWith("/__bili/plugin/fork")).length, 0); + assert.ok(lines.some((l) => l.includes("permanently disabled")), "the give-up is explicit, not silent"); + const after = lines.length; + await adopter.maybeAdopt(adoptInput()); + assert.equal(lines.length, after, "no further attempts or log lines after the latch"); +}); + test("createForkAdopter skips side-shaped bodies without consuming the attempt budget (#2399 gate ④)", async () => { resetForkCapabilityCacheForTest(); const core = openaiBodyToCore(adoptInput().body as Record)!; @@ -568,7 +630,7 @@ test("pi lane: an exploding getBranch fails toward capability and still adopts ( test("tryForkAdoption rejects a responses child whose prefix matches nothing (#2469 stage 2)", async () => { const { fetchImpl, calls } = fakeFetchRouting([snapshotRoute]); const result = await tryForkAdoption({ base: "http://px", parentConversationId: "p1", childConversationId: "c1", body: { model: "gpt", input: [{ type: "message", role: "user", content: "completely unrelated" }] }, fetchImpl }); - assert.deepEqual(result, { outcome: "degraded", reason: "no prefix match" }); + assert.deepEqual(result, { outcome: "degraded", reason: "no prefix match", retryable: false }); assert.equal(calls.filter((c) => c.url.endsWith("/__bili/plugin/fork")).length, 0, "no fork receipt is posted for a mismatch"); }); @@ -648,14 +710,14 @@ test("fork endpoint: identity-registered child yields only for the same parent ( { role: "system", content: "system prompt" }, { role: "user", content: "first original ".repeat(250) }, ] } }); - assert.deepEqual(conflicted, { outcome: "degraded", reason: "fork http 409 CHILD_CONFLICT" }, "different-parent claim stays a conflict"); + assert.deepEqual(conflicted, { outcome: "degraded", reason: "fork http 409 CHILD_CONFLICT", retryable: false }, "different-parent claim stays a conflict"); assert.equal((await register("child-bare")).status, 200); const bare = await tryForkAdoption({ base: h.origin, parentConversationId: "parent-sid", childConversationId: "child-bare", body: { model: "claude-test", max_tokens: 1024, stream: false, messages: [ { role: "system", content: "system prompt" }, { role: "user", content: "first original ".repeat(250) }, ] } }); - assert.deepEqual(bare, { outcome: "degraded", reason: "fork http 409 CHILD_CONFLICT" }, "parentless claim stays a conflict"); + assert.deepEqual(bare, { outcome: "degraded", reason: "fork http 409 CHILD_CONFLICT", retryable: false }, "parentless claim stays a conflict"); } finally { await h.close(); } diff --git a/tests/plugin-fork.test.ts b/tests/plugin-fork.test.ts index 297cf6d1bb..ddf5b02c3f 100644 --- a/tests/plugin-fork.test.ts +++ b/tests/plugin-fork.test.ts @@ -13,7 +13,7 @@ import { startServer } from "../src/server.ts"; import type { ProxyOptions } from "../src/config.ts"; import { SessionStore, _setStoreForTest } from "../src/persist.ts"; import { _resetPluginStateForTest, forkParentRevisionLegacy, rememberPluginMessages, resolveConversation } from "../src/plugin.ts"; -import { _resetSessionsForTest, getSession, markDirty } from "../src/session.ts"; +import { _resetSessionsForTest, getSession, markDirty, withSessionLock } from "../src/session.ts"; import { _setForTest } from "../src/registry.ts"; import { recordToolWitness, resetToolRingForTest } from "../src/tool-ring.ts"; import { contentStoreOf } from "../src/store.ts"; @@ -41,6 +41,7 @@ interface ForkResponse { inheritedBlocks: { id: string }[]; expandedBlocks: string[]; replayed: boolean; + lateAdopted?: boolean; } interface StatusResponse { @@ -1278,3 +1279,173 @@ test("fork protocol v2: revision is stable across state churn that does not move assert.notEqual(moved.body.parentRevision, before.body.parentRevision, "a snapshot change must flip the v2 revision"); } finally { await h.close(); } }); + +test("HTTP snapshot serves the recent raw view when no retained copy exists (#3011)", async () => { + const h = await harness(); + try { + const session = getSession("parent"); + assert.ok(session.pluginSnapshot, "the seeded parent retains a raw snapshot"); + // Simulate the residency gap: LRU evict / host restart loses the + // retained copy (persisted only after a fork receipt or explicit + // retention) while lastMessages — written by every request and + // persisted unconditionally — survives. Stale stored-ref bookkeeping + // from the lost view must not veto the tail path. + session.pluginSnapshot = undefined; + session.metadata.publicSnapshotStoredRefs = ["m99998"]; + const snapshot = await h.request("/__bili/plugin/snapshot?conversationId=parent"); + assert.equal(snapshot.status, 200, JSON.stringify(snapshot.body)); + assert.equal(snapshot.body.orderedMessages.length, 3); + assert.deepEqual(snapshot.body.messages.map((m) => m.text), h.messages.map((m) => m.content)); + // A full-prefix fork materializes from the same source view (the old + // code crashed here on parent.pluginSnapshot!.slice). + const fork = await h.request("/__bili/plugin/fork", forkRequest(snapshot.body, "child", 3)); + assert.equal(fork.status, 201, JSON.stringify(fork.body)); + const childSnap = (await h.request("/__bili/plugin/snapshot?conversationId=child")).body; + assert.deepEqual(childSnap.orderedMessages, snapshot.body.orderedMessages); + } finally { await h.close(); } +}); + +test("HTTP snapshot stays unavailable with neither retained copy nor recent raw view (#3011)", async () => { + const h = await harness(); + try { + const session = getSession("parent"); + session.pluginSnapshot = undefined; + session.lastMessages = undefined; + const snapshot = await h.request("/__bili/plugin/snapshot?conversationId=parent"); + assert.equal(snapshot.status, 409); + assert.equal(snapshot.body.code, "SNAPSHOT_UNAVAILABLE"); + } finally { await h.close(); } +}); + +test("HTTP snapshot never serves a folded tail view (#3011)", async () => { + const h = await harness(); + try { + const session = getSession("parent"); + session.pluginSnapshot = undefined; + session.lastMessagesFolded = true; + const snapshot = await h.request("/__bili/plugin/snapshot?conversationId=parent"); + assert.equal(snapshot.status, 409); + assert.equal(snapshot.body.code, "SNAPSHOT_UNAVAILABLE"); + } finally { await h.close(); } +}); + +test("HTTP snapshot prefers the fresh raw view over a stale frozen snapshot (#3011)", async () => { + const h = await harness(); + try { + const session = getSession("parent"); + assert.ok(session.lastMessages && session.lastMessages.length > 0); + // A full-cover native compaction froze pluginSnapshot pre-compaction + // (zero-overlap guard refuses to shrink it) while the rebase re-issued + // the ref space; the next raw request rewrote lastMessages. Serving + // the stale head would fail closed on its dead mapping forever — the + // head-divergent tail is the freshest raw view the client replays next. + const freshHead = { id: hash("fresh-head"), role: "user" as const, contentType: "text" as const, text: "post-compaction opener" }; + session.state.messageRefs.byRaw[freshHead.id] = "m00099"; + session.state.messageRefs.byRef["m00099"] = freshHead.id; + session.lastMessages = [freshHead, ...session.lastMessages.slice(1)]; + session.lastMessagesFolded = false; + const snapshot = await h.request("/__bili/plugin/snapshot?conversationId=parent"); + assert.equal(snapshot.status, 200, JSON.stringify(snapshot.body)); + assert.equal(snapshot.body.messages[0].text, freshHead.text); + assert.equal(snapshot.body.orderedMessages[0].ref, "m00099"); + } finally { await h.close(); } +}); + +// #3025: a child whose first stamped request landed before adoption gets +// registered server-side during that request — pre-fix every later fork POST +// then met CHILD_CONFLICT forever. These tests drive that exact shape: the +// child sends its inherited history as a real model request FIRST, and the +// fork POST arrives afterwards. +async function latchChild(h: Awaited>, conversationId: string, messages: unknown[]) { + const r = await fetch(`${h.origin}/bili/${h.upstreamUrl}/v1/messages`, { method: "POST", headers: { "content-type": "application/json", "x-bili-plugin": "test", "x-bili-plugin-conversation": conversationId }, body: JSON.stringify({ model: "claude-test", max_tokens: 1024, stream: false, messages }) }); + assert.equal(r.status, 200); + await r.text(); +} + +test("HTTP fork completes late for a latched unforked child instead of latching CHILD_CONFLICT (#3025)", async () => { + const h = await harness(); + try { + await compress(h); + await latchChild(h, "child", h.messages); + const snapshot = await h.request("/__bili/plugin/snapshot?conversationId=parent"); + assert.equal(snapshot.status, 200, JSON.stringify(snapshot.body)); + // Pre-fix this POST hit the registration gate and latched as a + // permanent 409 CHILD_CONFLICT. + const fork = await h.request("/__bili/plugin/fork", forkRequest(snapshot.body, "child", 3)); + assert.equal(fork.status, 201, JSON.stringify(fork.body)); + assert.equal(fork.body.lateAdopted, true); + assert.equal(fork.body.inheritedBlocks.length, 1); + const childSession = resolveConversation("child").session!; + assert.equal(childSession.metadata.parentConversationId, "parent"); + assert.notEqual(childSession.metadata.publicForkReceipt, undefined); + // Idempotency survives late completion: identical payload replays the receipt. + const again = await h.request("/__bili/plugin/fork", forkRequest(snapshot.body, "child", 3)); + assert.equal(again.status, 200); + assert.equal(again.body.replayed, true); + // The adopted child serves the inherited prefix... + const childSnap = await h.request("/__bili/plugin/snapshot?conversationId=child"); + assert.equal(childSnap.status, 200, JSON.stringify(childSnap.body)); + assert.deepEqual(childSnap.body.orderedMessages, snapshot.body.orderedMessages); + // ...and the inheritance is live: the next request folds through it. + const next = await fetch(`${h.origin}/bili/${h.upstreamUrl}/v1/messages`, { method: "POST", headers: { "content-type": "application/json", "x-bili-plugin": "test", "x-bili-plugin-conversation": "child" }, body: JSON.stringify({ model: "claude-test", max_tokens: 1024, stream: false, messages: [...h.messages.slice(0, 2), { role: "user", content: "continue child" }] }) }); + assert.equal(next.status, 200, await next.text()); + const forwarded = JSON.stringify(h.forwarded.at(-1)?.messages); + assert(forwarded.includes(summary), forwarded); + assert(!forwarded.includes("second original"), forwarded); + assert.equal(childSession.state.blocks[0].active, true); + } finally { await h.close(); } +}); + +test("HTTP fork late completion refuses a child that grew its own turns (#3025)", async () => { + const h = await harness(); + try { + await latchChild(h, "child-grown", h.messages); + await latchChild(h, "child-grown", [...h.messages, { role: "assistant", content: "answer" }, { role: "user", content: "fourth original" }]); + const snapshot = await h.request("/__bili/plugin/snapshot?conversationId=parent"); + const fork = await h.request("/__bili/plugin/fork", forkRequest(snapshot.body, "child-grown", 3)); + assert.equal(fork.status, 409); + assert.equal(fork.body.code, "CHILD_CONFLICT"); + assert.equal(resolveConversation("child-grown").session!.metadata.publicForkReceipt, undefined); + } finally { await h.close(); } +}); + +test("HTTP fork late completion refuses a child that folded its own history (#3025)", async () => { + const h = await harness(); + try { + await latchChild(h, "child-folded", h.messages); + await compress(h, "child-folded", "m00001", "m00002"); + const snapshot = await h.request("/__bili/plugin/snapshot?conversationId=parent"); + const fork = await h.request("/__bili/plugin/fork", forkRequest(snapshot.body, "child-folded", 3)); + assert.equal(fork.status, 409); + assert.equal(fork.body.code, "CHILD_CONFLICT"); + assert.equal(resolveConversation("child-folded").session!.metadata.publicForkReceipt, undefined); + } finally { await h.close(); } +}); + +test("HTTP fork late completion refuses ref-numbering divergence instead of corrupting (#3025)", async () => { + const h = await harness(); + try { + await latchChild(h, "child-drift", h.messages); + const snapshot = await h.request("/__bili/plugin/snapshot?conversationId=parent"); + resolveConversation("child-drift").session!.state.messageRefs.byRaw[snapshot.body.orderedMessages[0].rawId] = "m99999"; + const fork = await h.request("/__bili/plugin/fork", forkRequest(snapshot.body, "child-drift", 3)); + assert.equal(fork.status, 409); + assert.equal(fork.body.code, "CHILD_CONFLICT"); + assert.equal(resolveConversation("child-drift").session!.metadata.publicForkReceipt, undefined); + } finally { await h.close(); } +}); + +test("HTTP fork late completion serializes behind an in-flight child request without deadlocking (#3025)", async () => { + const h = await harness(); + try { + await compress(h); + await latchChild(h, "child-race", h.messages); + const snapshot = await h.request("/__bili/plugin/snapshot?conversationId=parent"); + const childSession = resolveConversation("child-race").session!; + const held = withSessionLock(childSession, () => new Promise((resolve) => setTimeout(resolve, 400))); + const forkPromise = h.request("/__bili/plugin/fork", forkRequest(snapshot.body, "child-race", 3)); + const [fork] = await Promise.all([forkPromise, held]); + assert.equal(fork.status, 201, JSON.stringify(fork.body)); + assert.equal(fork.body.lateAdopted, true); + } finally { await h.close(); } +});