From 6bb67c4f066c8a4e28b73cb05a88a901875f05ff Mon Sep 17 00:00:00 2001 From: ci-check Date: Tue, 27 Oct 2026 01:09:09 +0800 Subject: [PATCH 1/2] =?UTF-8?q?fix:=20incremental=20lane=20=E2=80=94=20war?= =?UTF-8?q?n=20on=20dual=20hosts=20sharing=20a=20conversation;=20prune=20d?= =?UTF-8?q?ead-branch=20cache=20marks=20on=20resync?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #4868, two P3 improvements in the incremental lane (both verified against the code paths; no correctness impact either way): 1. Dual hosts sharing one conversationId mutually kill each other with zero visibility: each resync invalidates the other's acks, both burn 3 consecutive 409s into the sticky REJECTION_CIRCUIT=3 full-resend fallback until process restart. Every envelope now carries an optional per-process instanceId fingerprint (randomUUID, one value per host process, stamped in buildBody so all four native adapters get it for free); the server records the last successful resync source per conversation and warns once per 5min cooldown when two DISTINCT sources replace the same canonical within 60s. Missing or malformed fingerprints degrade to anonymous (no false alarm); old servers ignore the field, old clients omit it. - src/agent/incremental-client.ts: INSTANCE_ID + __incrementalInstanceIdForTests, buildBody stamps it - src/incremental-lane.ts: noteResyncSource (bounded map, budget pattern like droppedParamWarned), called on successful resync turns; reset seam clears it 2. inc:cache-marks grew unbounded across branches: resync replaced the canonical but entry.marks kept every historical id forever (map + ext key rewritten whole each turn, O(turns across branches)). applyTurn now prunes marks to the live canonical set on every resync before merging the turn's own marks. Ids are unique per session, so a surviving mark can only attach to a live message; under pluginLaneServerHistory the marks were already orphaned at assembly (derived-id keys vs pinned ids) — the prune makes storage match assembly reality. Behavior changes (additive): new optional envelope field instanceId; new warn log line for suspected dual writers; marks lifecycle OLD = all historical marks kept forever → NEW = pruned to the live set on each resync. Both rollout features (pluginIncrementalTurns, pluginLaneServerHistory) default to 0% — opt-in installs only. Tests: dual-writer alarm positive + same-source negative + missing/malformed fingerprint negative; marks pruning (dead branch pruned, live/new survive); client envelope stamp stability. PLUGIN.md documents the new field. --- PLUGIN.md | 2 +- src/agent/incremental-client.ts | 18 +++- src/incremental-lane.ts | 53 +++++++++- tests/incremental-client-recovery.test.ts | 19 +++- tests/incremental-lane.test.ts | 112 +++++++++++++++++++++- 5 files changed, 194 insertions(+), 10 deletions(-) diff --git a/PLUGIN.md b/PLUGIN.md index d12ef260a1..eef9c2bef2 100644 --- a/PLUGIN.md +++ b/PLUGIN.md @@ -202,7 +202,7 @@ Boundary handling is `exact` when complete blocks fit inside the prefix, `expand Lineage-only declaration (#2408): a host that cannot drive the full fork protocol (no snapshot matching, e.g. claude `--fork-session`) may still declare the parent by sending a plain identity register — `POST /__bili/plugin/register` with `conversationId` (the fresh child id), `identity: true` and `parentConversationId` — from its SessionStart hook. The declaration scopes the proxy's content-match resume inheritance (#1486) to that parent chain, so a byte-exact replay of the parent's wire history seeds refs/blocks on the child's first request, and a non-matching replay falls back to the read-only parent link (#1333). A later register without `parentConversationId` never erases the declared parent. This is strictly weaker than §8 (no dead-ref namespace, no nudge inheritance, no receipt verification) but needs no snapshot access. -Incremental turns (#3248; not yet advertised in the plugin manifest — detect by trying the endpoint and handling the 503): a host that keeps its own full local history may stop resending it. `POST /__bili/plugin/turn` (same `/__bili/` gate; headers `x-bili-plugin` + optional `x-bili-plugin-model`, `x-bili-plugin-context-window`, `x-bili-plugin-max-output`) with `{ conversationId, sinceSeq, resync?, system?, path?, request, messages }`. The first turn (or any recovery) sends `sinceSeq: 0` + the full `messages[]` (`resync: true` is implied on seed — a wholesale replacement never 409s and works whether the canonical is empty or already live); every later turn sends ONLY what the client added since the last ack — including the previous assistant reply (client-owned). `request` carries anthropic params (model, max_tokens, temperature, top_p, top_k, stop_sequences, stream, tools, tool_choice, metadata, thinking, service_tier, parallel_tool_calls); `system` is sticky (omit to keep the last one); `path` defaults to `/v1/messages` (bare relative rides the lane's default upstream; the `/bili//…` form is explicit). The response is the upstream body streamed through, with `x-bili-applied-seq` = the canonical length after applying the delta — echo that value as the next turn's `sinceSeq`. `409 { error: "seq-mismatch" | "turn-in-flight", serverSeq }` means the client MUST recover with a resync turn (`sinceSeq: serverSeq` if it still has the history, else `resync: true` full seed). `503` means the lane runs without `storage.unified` (the canonical log lives in the unified messages table). The lane owns no session identity of its own: the assembled request carries the client's own `x-bili-plugin-conversation`, so incremental and normal full-resend traffic for one conversation share one session, one fold ledger and one canonical — switching modes needs no ceremony, and a conversation may interleave them (e.g. fall back to full resend while a resync is pending). Folding, nudges, tools and the plugin tool endpoints all work unchanged on the assembled body; the model never sees the difference. v1 caveats: the canonical log is the session's messages table — a turn is acked after the assembled body is persisted (best-effort immediate flush; a crash in between surfaces as 409 → resync), and the pipeline still walks the assembled prefix (O(full) CPU, cheap and zero-hit — the server-authored fast path is tracked on #3248). A `resync: true` that replaces a diverged canonical records a base-seq epoch (`session_ext` key `inc:base-seq`): the stale branch's rows stay in the append-only log for audit, but cold rebuilds slice past them — one resync heals, restarts included. On legacy-only storage (no ext surface) the marker degrades and cold rebuilds serve the full row union (409 → resync once, as before). +Incremental turns (#3248; not yet advertised in the plugin manifest — detect by trying the endpoint and handling the 503): a host that keeps its own full local history may stop resending it. `POST /__bili/plugin/turn` (same `/__bili/` gate; headers `x-bili-plugin` + optional `x-bili-plugin-model`, `x-bili-plugin-context-window`, `x-bili-plugin-max-output`) with `{ conversationId, sinceSeq, resync?, system?, path?, request, messages, instanceId? }`. `instanceId` is an OPTIONAL per-process client fingerprint (printable tokens, ≤128 chars): the lane uses it ONLY to warn when two DISTINCT sources replace the same canonical within a short window — two hosts sharing one conversation id flip each other's history into a 409 storm until both degrade to sticky full resend, and the warn is the only log evidence of that race (#4868); it is never persisted nor forwarded upstream, and older clients omitting it simply cannot be fingerprinted (no false alarm). The first turn (or any recovery) sends `sinceSeq: 0` + the full `messages[]` (`resync: true` is implied on seed — a wholesale replacement never 409s and works whether the canonical is empty or already live); every later turn sends ONLY what the client added since the last ack — including the previous assistant reply (client-owned). `request` carries anthropic params (model, max_tokens, temperature, top_p, top_k, stop_sequences, stream, tools, tool_choice, metadata, thinking, service_tier, parallel_tool_calls); `system` is sticky (omit to keep the last one); `path` defaults to `/v1/messages` (bare relative rides the lane's default upstream; the `/bili//…` form is explicit). The response is the upstream body streamed through, with `x-bili-applied-seq` = the canonical length after applying the delta — echo that value as the next turn's `sinceSeq`. `409 { error: "seq-mismatch" | "turn-in-flight", serverSeq }` means the client MUST recover with a resync turn (`sinceSeq: serverSeq` if it still has the history, else `resync: true` full seed). `503` means the lane runs without `storage.unified` (the canonical log lives in the unified messages table). The lane owns no session identity of its own: the assembled request carries the client's own `x-bili-plugin-conversation`, so incremental and normal full-resend traffic for one conversation share one session, one fold ledger and one canonical — switching modes needs no ceremony, and a conversation may interleave them (e.g. fall back to full resend while a resync is pending). Folding, nudges, tools and the plugin tool endpoints all work unchanged on the assembled body; the model never sees the difference. v1 caveats: the canonical log is the session's messages table — a turn is acked after the assembled body is persisted (best-effort immediate flush; a crash in between surfaces as 409 → resync), and the pipeline still walks the assembled prefix (O(full) CPU, cheap and zero-hit — the server-authored fast path is tracked on #3248). A `resync: true` that replaces a diverged canonical records a base-seq epoch (`session_ext` key `inc:base-seq`): the stale branch's rows stay in the append-only log for audit, but cold rebuilds slice past them — one resync heals, restarts included. On legacy-only storage (no ext surface) the marker degrades and cold rebuilds serve the full row union (409 → resync once, as before). Marker divert (#3248 phase 2 — native extensions): instead of switching the provider URL to the turn endpoint, a host may send the incremental body on its ORDINARY model request. Stamp `x-bili-plugin-turn: 1` on the request (the lane requires `x-bili-plugin` too, as for every plugin signal); the body shape — top-level `conversationId` + `sinceSeq` + `messages`, no top-level `model` — is the final word: a marker'd body that is NOT incremental-shaped rides the normal pipeline byte-identically, so a host may stamp the marker whenever the feature is enabled and only convert turns it wants. Incremental-SHAPED turns are lane traffic and carry the same trust class as the direct turn endpoint (#3343): non-loopback / tunnel / untrusted-origin callers get 403 on them (their capability probe already 403s, so a well-behaved host never sends deltas from afar), and management prefixes (`/__bili/`, `/__acp/`) never ride the divert. The lane forwards the assembled request to the ORIGINAL request path (bare or `/bili/…` — no provider reconfiguration). Capability probe: `GET /__bili/plugin/turn` → `{ ok, protocol: 1, feature: "pluginIncrementalTurns", enabled, storage }` (loopback-gated like every `/__bili/` surface; a 404 = older server, stay full-resend). Gating: rollout feature `pluginIncrementalTurns` (default 0% = every install byte-identical to before) — enable with `rollout.features.pluginIncrementalTurns: true` (or a percent) in the bili config, or `BILI_ROLLOUT_PLUGIN_INCREMENTAL_TURNS=1`; the feature additionally requires a live unified sidecar (shadow or full). When disabled mid-flight the lane answers 503 on incremental-shaped turns — fall back to a plain full-resend request. Reference host wiring: pi rides its `before_provider_headers` (marker) + `before_provider_request` (payload replacement) + `after_provider_response` (ack) extension events; dsh-native rides the fetch patch's `beforeSend` (body conversion) + `headersFor` (marker) + `onResponse` (ack). pi-native inherits the pi wiring unchanged (same extension factory — only proxy discovery differs). opencode rides its `http.request` hook (an `incremental` adapter callback appended after routing/stamping swaps the request body via `replaceRequestPayload` and stamps the marker on the replaced request; acks land on the fetch intercept's `onResponse`). omp has no header events at all — its native adapter reads the extension-stamped `prompt_cache_key` body field as the conversation id inside `beforeSend`, stamps the plugin headers directly on the request init (identity upgrades, never forks: wire mode already resolves the same value via `preferPromptCacheKeyIdentity`), and acks on `onResponse`. Non-Anthropic wire shapes (OpenAI `tool_calls`/`role:"tool"`) are detected by `wireMessagesLookAnthropic` and left on full resend — the lane's message ingestion is the anthropic wire. Planned turns are single-in-flight per URL and age out after 30s (a network-dead request must not block the URL forever). The shared client state machine lives in `src/agent/incremental-client.ts` (probe TTL cache, prefix-hash diffing, sticky system, per-conversation dead flag) — new native hosts should reuse it rather than re-implement the diffing. diff --git a/src/agent/incremental-client.ts b/src/agent/incremental-client.ts index d5b1ef4b0e..4bb8df28b0 100644 --- a/src/agent/incremental-client.ts +++ b/src/agent/incremental-client.ts @@ -16,7 +16,7 @@ // messages ({role, content: string | blocks[]}) — the lane derives stable // content ids server-side, so hosts without their own message ids work. -import { createHash } from "node:crypto"; +import { createHash, randomUUID } from "node:crypto"; import type { DirectSendPrep } from "./native-intercept.js"; /** Marker header the lane's divert recognizes. Client-side copy of @@ -216,6 +216,21 @@ interface ConvState { const convs = new Map(); +/** #4868: per-process identity stamped on EVERY incremental envelope. Two + * hosts sharing one conversation id flip each other's canonical with their + * resyncs until both burn REJECTION_CIRCUIT strikes into the sticky full- + * resend degradation — without a source fingerprint on the wire that race is + * indistinguishable in server logs from ordinary drift. One value per process + * lifetime (a restart is exactly when the stickiness clears); older servers + * ignore unknown envelope fields, older clients omit it (the lane then cannot + * fingerprint — no false alarm). */ +const INSTANCE_ID = randomUUID(); + +/** Test seam for the stamped per-process fingerprint (stable within a process). */ +export function __incrementalInstanceIdForTests(): string { + return INSTANCE_ID; +} + // #4491 host msgids: ids this client has already put on an envelope for the // conversation — the client-side mirror of the lane's canonical id set. The // lane HARD-FAILS a non-resync turn (400) whose msgIds reuse a live canonical @@ -389,6 +404,7 @@ function buildBody(args: PlanArgs, sinceSeq: number, messages: IncrementalWireMe path: args.path ?? "/v1/messages", request, messages, + instanceId: INSTANCE_ID, }; if (resync) body.resync = true; if (includeSystem) body.system = args.system === undefined ? null : args.system; diff --git a/src/incremental-lane.ts b/src/incremental-lane.ts index fb2a0a14be..057868ca9a 100644 --- a/src/incremental-lane.ts +++ b/src/incremental-lane.ts @@ -9,6 +9,7 @@ * "sinceSeq": 0, // canonical length the client has acked * "resync": false, // true → messages[] is a full replacement * "system": "..." | [blocks], // latest system text (sticky when omitted) + * "instanceId": "", // OPTIONAL per-process client fingerprint (#4868 dual-writer alert) * "path": "/v1/messages", // optional forward path: bare relative (default, rides the lane's default upstream) or /bili//… embedded * "request": { model, max_tokens, tools, stream, ... }, // anthropic params (allowlist) * "messages": [ ...anthropic wire messages... ] @@ -165,6 +166,42 @@ function warnDroppedRequestParams(conversationId: string, request: Record(); +const INSTANCE_ID_PATTERN = /^[A-Za-z0-9._:-]{1,128}$/; + +function noteResyncSource(conversationId: string, raw: unknown, log: (level: string, msg: string) => void): void { + const now = Date.now(); + const instanceId = typeof raw === "string" && INSTANCE_ID_PATTERN.test(raw) ? raw : ""; + const prev = resyncSources.get(conversationId); + // Same bounded discipline as droppedParamWarned: past the budget, drop the + // oldest epoch rather than grow without bound. + if (resyncSources.size >= RESYNC_SOURCE_BUDGET && prev === undefined) resyncSources.clear(); + const entry: ResyncSource = { instanceId, at: now, warnedAt: prev?.warnedAt ?? 0 }; + resyncSources.set(conversationId, entry); + const distinct = prev !== undefined && instanceId !== "" && prev.instanceId !== "" && prev.instanceId !== instanceId; + if (!distinct || now - prev.at > DUAL_WRITER_WINDOW_MS || now - entry.warnedAt < DUAL_WRITER_WARN_COOLDOWN_MS) return; + entry.warnedAt = now; + log("warn", `[incremental] ${conversationId}: suspected dual writers — distinct sources "${prev.instanceId}" and "${instanceId}" replaced this canonical ${Math.round((now - prev.at) / 1000)}s apart; each resync invalidates the other's acks, so both hosts will burn 3 consecutive 409s into the sticky full-resend fallback unless they stop sharing the conversation id (#4868)`); +} + /** Wire headers the client may contribute to the forwarded request. */ const PASSTHROUGH_HEADERS = ["authorization", "anthropic-version", "anthropic-beta"] as const; @@ -443,7 +480,11 @@ function applyTurn( const serverSeq = entry.msgs.length; if (resync) { // Full replacement: the client is the source of truth for the whole - // history (edit / branch reset / recovery). Marks and system survive. + // history (edit / branch reset / recovery). System survives (sticky + // state, never positional); marks are pruned below to the live set — + // ids are unique so a surviving mark can only attach to a live + // message, but a replaced branch's marks would otherwise linger in + // the map (and its ext key) forever (#4868). } else if (sinceSeq !== serverSeq) { // Strict ack contract — including sinceSeq 0 against a populated // canonical: a client that believes it is seeding a fresh server must @@ -480,6 +521,10 @@ function applyTurn( } const base = resync ? [] : entry.msgs; entry.msgs = [...base, ...deltaMsgs]; + if (resync) { + const live = new Set(entry.msgs.map((m) => m.id)); + for (const id of entry.marks.keys()) if (!live.has(id)) entry.marks.delete(id); + } for (const [id, mark] of flat.cacheControls) entry.marks.set(id, mark); entry.boundary = deltaWire.length > 0 ? hashWireBoundary(deltaWire[deltaWire.length - 1]) @@ -509,6 +554,7 @@ export function __resetIncrementalLaneForTests(): void { pendingPtrWork.clear(); droppedParamWarned.clear(); droppedParamWarnPairs = 0; + resyncSources.clear(); } /** Marker header native extensions stamp on an ordinary model request to @@ -674,7 +720,10 @@ export async function handleIncrementalTurn( } touchCache(conversationId, entry); // Record the epoch BEFORE the forward lands the new branch's rows. - if (resync) await recordResyncBaseSeq(sessionId, log); + if (resync) { + await recordResyncBaseSeq(sessionId, log); + noteResyncSource(conversationId, payload.instanceId, log); + } persistLaneState(sessionId, entry.system, entry.systemSet, entry.marks, log); const body: Record = { ...forwarded, messages: coreToAnthropic(entry.msgs, entry.marks) as never }; diff --git a/tests/incremental-client-recovery.test.ts b/tests/incremental-client-recovery.test.ts index c92363460d..8ae1da8c1d 100644 --- a/tests/incremental-client-recovery.test.ts +++ b/tests/incremental-client-recovery.test.ts @@ -8,7 +8,7 @@ import assert from "node:assert/strict"; process.env.NODE_TEST_CONTEXT = "1"; -const { __resetIncrementalClientForTests, noteIncrementalAck, noteIncrementalRejection, planIncrementalTurn } = await import("../src/agent/incremental-client.ts"); +const { __incrementalInstanceIdForTests, __resetIncrementalClientForTests, noteIncrementalAck, noteIncrementalRejection, planIncrementalTurn } = await import("../src/agent/incremental-client.ts"); const SID = "inc-recovery-1"; @@ -63,3 +63,20 @@ test("#3369: rejection with no prior state is a no-op reset (a seed/resync canno const next = planIncrementalTurn({ conversationId: SID, messages: msgs(2), system: undefined }); assert.equal(next.mode, "seed"); }); + +test("#4868: every envelope carries the per-process instanceId fingerprint, stable across plans", () => { + __resetIncrementalClientForTests(); + const expected = __incrementalInstanceIdForTests(); + assert.ok(typeof expected === "string" && expected.length > 0, "process fingerprint is minted at module load"); + + const seed = planIncrementalTurn({ conversationId: SID, messages: msgs(2), system: undefined }); + assert.equal(seed.mode, "seed"); + const seedBody = JSON.parse(seed.body) as Record; + assert.equal(seedBody.instanceId, expected, "seed envelope stamps the fingerprint"); + + noteIncrementalAck(SID, 2, msgs(2), undefined); + const delta = planIncrementalTurn({ conversationId: SID, messages: msgs(4), system: undefined }); + assert.equal(delta.mode, "delta"); + const deltaBody = JSON.parse(delta.body) as Record; + assert.equal(deltaBody.instanceId, expected, "delta envelope stamps the SAME fingerprint (one per process lifetime)"); +}); diff --git a/tests/incremental-lane.test.ts b/tests/incremental-lane.test.ts index 294af1b356..f3fc1bd26f 100644 --- a/tests/incremental-lane.test.ts +++ b/tests/incremental-lane.test.ts @@ -14,6 +14,7 @@ import { _resetSessionsForTest, peekSession } from "../src/session.ts"; import { _setForTest } from "../src/registry.ts"; import { resetToolRingForTest } from "../src/tool-ring.ts"; import { __resetIncrementalLaneForTests } from "../src/incremental-lane.ts"; +import { setLogCapture } from "../src/logger.ts"; process.env.NODE_ENV = "test"; const testRoot = mkdtempSync(join(tmpdir(), "bili-incr-")); @@ -25,6 +26,10 @@ interface Turn { resync?: boolean; system?: string; request?: Record; + /** #4868: client source fingerprint stamped on the envelope (dual-writer alert). */ + instanceId?: string; + /** Defaults to "conv-inc-1" (header and body must agree). */ + conversationId?: string; } interface Harness { @@ -36,10 +41,11 @@ interface Harness { markerRaw: (reqPath: string, anthropicBody: Record) => Promise<{ status: number; body: unknown }>; forwarded: Record[]; dir: string; + store: SessionStore; close: () => Promise; } -async function harness(storageMode: "full" | "legacy", opts: { upstreamDelayMs?: number; upstreamStatus?: number } = {}): Promise { +async function harness(storageMode: "full" | "legacy", opts: { upstreamDelayMs?: number; upstreamStatus?: number; logEnabled?: boolean } = {}): Promise { const dir = mkdtempSync(join(testRoot, "run-")); for (const key of ["XDG_STATE_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME", "XDG_CONFIG_HOME"]) process.env[key] = dir; process.env.BILI_STORAGE_UNIFIED = storageMode === "full" ? "full" : "legacy"; @@ -69,15 +75,16 @@ async function harness(storageMode: "full" | "legacy", opts: { upstreamDelayMs?: const addr = upstream.address(); assert(addr && typeof addr === "object"); const upstreamUrl = `http://127.0.0.1:${addr.port}`; - const proxy = await startServer({ port: 0, host: "127.0.0.1", upstream: upstreamUrl, routes: { [upstreamUrl]: { models: { "claude-test": { context: 400000 } } } }, modelContextLimit: 400000, kernelConfig: defaultConfig(400000), compress: { injectTool: true, injectNudge: false }, promptCache: { routing: "auto" }, sessionHeader: "x-acp-session", log: false, debug: false, passthrough: false, autoUpdate: false, mitm: { enabled: false, domains: [] } } as ProxyOptions); + const proxy = await startServer({ port: 0, host: "127.0.0.1", upstream: upstreamUrl, routes: { [upstreamUrl]: { models: { "claude-test": { context: 400000 } } } }, modelContextLimit: 400000, kernelConfig: defaultConfig(400000), compress: { injectTool: true, injectNudge: false }, promptCache: { routing: "auto" }, sessionHeader: "x-acp-session", log: opts.logEnabled ?? false, debug: false, passthrough: false, autoUpdate: false, mitm: { enabled: false, domains: [] } } as ProxyOptions); if (!(proxy as unknown as { listening?: boolean }).listening) await once(proxy, "listening"); const proxyUrl = `http://127.0.0.1:${(proxy.address() as { port: number }).port}`; const path = "/v1/messages"; const turn = async (t: Turn) => { + const cid = t.conversationId ?? "conv-inc-1"; const response = await fetch(`${proxyUrl}/__bili/plugin/turn`, { method: "POST", - headers: { "content-type": "application/json", "x-bili-plugin": "dsh", "x-bili-plugin-conversation": "conv-inc-1" }, - body: JSON.stringify({ conversationId: "conv-inc-1", sinceSeq: t.sinceSeq, ...(t.resync ? { resync: true } : {}), ...(t.system !== undefined ? { system: t.system } : {}), path, request: { model: "claude-test", max_tokens: 1024, ...(t.request ?? {}) }, messages: t.messages }), + headers: { "content-type": "application/json", "x-bili-plugin": "dsh", "x-bili-plugin-conversation": cid }, + body: JSON.stringify({ conversationId: cid, sinceSeq: t.sinceSeq, ...(t.resync ? { resync: true } : {}), ...(t.system !== undefined ? { system: t.system } : {}), path, request: { model: "claude-test", max_tokens: 1024, ...(t.request ?? {}) }, messages: t.messages, ...(t.instanceId !== undefined ? { instanceId: t.instanceId } : {}) }), }); const body: unknown = await response.json().catch(() => null); return { status: response.status, headers: response.headers, body }; @@ -106,7 +113,7 @@ async function harness(storageMode: "full" | "legacy", opts: { upstreamDelayMs?: const body: unknown = await response.json().catch(() => null); return { status: response.status, body }; }; - return { proxyUrl, path, turn, probe, markerTurn, markerRaw, forwarded, dir, close: async () => { await new Promise((resolve) => proxy.close(() => resolve())); store.closeQuietly(); await new Promise((resolve) => upstream.close(() => resolve())); } }; + return { proxyUrl, path, turn, probe, markerTurn, markerRaw, forwarded, dir, store, close: async () => { await new Promise((resolve) => proxy.close(() => resolve())); store.closeQuietly(); await new Promise((resolve) => upstream.close(() => resolve())); } }; } const userTurn = (n: number, text = "hello") => ({ id: `u${n}`, role: "user", contentType: "text", content: text.repeat(400) }); @@ -584,3 +591,98 @@ test("#3433: lane rejects non-anthropic message shapes with 400, never half-conv await h.close(); } }); + +// #4868: two DISTINCT client sources replacing the same canonical in quick +// succession = two hosts sharing one conversation id. Each resync invalidates +// the other's acks, so both burn consecutive 409s into the client-side +// REJECTION_CIRCUIT=3 sticky full-resend degradation — without the source +// fingerprint nothing in the log distinguishes that race from ordinary drift. +// The lane must name the race (once, per cooldown); same-source resyncs are +// normal recovery and must NOT alarm (gate discipline: both directions). +test("#4868: alternating resync sources on one conversation warn once; same source never alarms", async () => { + const h = await harness("full", { logEnabled: true }); + const warnings: string[] = []; + setLogCapture((level, msg) => { if (level === "warn") warnings.push(msg); }); + try { + const seed = await h.turn({ sinceSeq: 0, resync: true, messages: [userTurn(1)], instanceId: "host-a" }); + assert.equal(seed.status, 200); + // host-b believes it owns the conversation too: its stale delta 409s + // against host-a's canonical... + const clash = await h.turn({ sinceSeq: 0, messages: [userTurn(9)], instanceId: "host-b" }); + assert.equal(clash.status, 409); + // ...and its resync replaces the canonical from a DISTINCT source → + // exactly one clear warn naming both sides. + const resync = await h.turn({ sinceSeq: 0, resync: true, messages: [userTurn(1), userTurn(9)], instanceId: "host-b" }); + assert.equal(resync.status, 200); + const alarms = warnings.filter((w) => w.includes("dual writers")); + assert.equal(alarms.length, 1, `expected exactly one dual-writer warn, got: ${JSON.stringify(warnings)}`); + for (const needle of ["conv-inc-1", "host-a", "host-b"]) assert.ok(alarms[0].includes(needle), `warn must name ${needle}: ${alarms[0]}`); + // Same source resyncing again is ordinary recovery — no second alarm. + const again = await h.turn({ sinceSeq: 0, resync: true, messages: [userTurn(1), userTurn(9), assistantTurn(1)], instanceId: "host-b" }); + assert.equal(again.status, 200); + assert.equal(warnings.filter((w) => w.includes("dual writers")).length, 1); + } finally { + setLogCapture(null); + await h.close(); + } +}); + +test("#4868: missing or malformed instanceId cannot trigger the dual-writer alarm", async () => { + const h = await harness("full", { logEnabled: true }); + const warnings: string[] = []; + setLogCapture((level, msg) => { if (level === "warn") warnings.push(msg); }); + try { + // Older clients stamp nothing: two anonymous resyncs must stay silent. + const seed = await h.turn({ sinceSeq: 0, resync: true, messages: [userTurn(1)] }); + assert.equal(seed.status, 200); + const clash = await h.turn({ sinceSeq: 0, messages: [userTurn(9)] }); + assert.equal(clash.status, 409); + const resync = await h.turn({ sinceSeq: 0, resync: true, messages: [userTurn(1), userTurn(9)] }); + assert.equal(resync.status, 200); + // A malformed fingerprint degrades to anonymous for that side only — + // still no alarm against a valid other side. + const seed2 = await h.turn({ sinceSeq: 0, resync: true, messages: [userTurn(2)], conversationId: "conv-inc-2", instanceId: "bad id!" }); + assert.equal(seed2.status, 200); + const clash2 = await h.turn({ sinceSeq: 0, messages: [userTurn(8)], conversationId: "conv-inc-2", instanceId: "host-b" }); + assert.equal(clash2.status, 409); + const resync2 = await h.turn({ sinceSeq: 0, resync: true, messages: [userTurn(2), userTurn(8)], conversationId: "conv-inc-2", instanceId: "host-b" }); + assert.equal(resync2.status, 200); + assert.equal(warnings.filter((w) => w.includes("dual writers")).length, 0, JSON.stringify(warnings)); + } finally { + setLogCapture(null); + await h.close(); + } +}); + +// #4868: resync replaces the canonical — marks keyed by ids that no longer +// exist in the new branch must be pruned (they would linger in the map and be +// rewritten into the inc:cache-marks ext key forever, O(turns across branches)). +// Live marks survive; new marks land as before. +test("#4868: resync prunes cache marks of the replaced branch; live and new marks survive", async () => { + const h = await harness("full"); + try { + const marked = { role: "user", content: [{ type: "text", text: "hello".repeat(400), cache_control: { type: "ephemeral" } }] }; + const seed = await h.turn({ sinceSeq: 0, resync: true, messages: [marked] }); + assert.equal(seed.status, 200); + let ext = h.store.getSessionExt("conv-inc-1", "inc:cache-marks"); + assert.ok(Array.isArray(ext) && ext.length === 1, `seed mark persisted: ${JSON.stringify(ext)}`); + + // Resync onto a DIFFERENT branch (distinct text → distinct derived id): + // the old mark's id is gone from the canonical → pruned; unmarked + // history adds nothing. Same-content messages share their id and would + // legitimately keep the mark. + const resync = await h.turn({ sinceSeq: 0, resync: true, messages: [userTurn(7, "branch seven")] }); + assert.equal(resync.status, 200); + ext = h.store.getSessionExt("conv-inc-1", "inc:cache-marks"); + assert.ok(Array.isArray(ext) && ext.length === 0, `dead-branch mark pruned: ${JSON.stringify(ext)}`); + + // A delta carrying a fresh mark appends normally. + const marked2 = { role: "user", content: [{ type: "text", text: "world".repeat(400), cache_control: { type: "ephemeral" } }] }; + const delta = await h.turn({ sinceSeq: 1, messages: [marked2] }); + assert.equal(delta.status, 200); + ext = h.store.getSessionExt("conv-inc-1", "inc:cache-marks"); + assert.ok(Array.isArray(ext) && ext.length === 1, `live mark retained: ${JSON.stringify(ext)}`); + } finally { + await h.close(); + } +}); From d577f24c089f1c1391cb0dd7f2bc770e8571a19e Mon Sep 17 00:00:00 2001 From: ci-check Date: Tue, 27 Oct 2026 01:09:34 +0800 Subject: [PATCH 2/2] docs: list the msgIds envelope field in PLUGIN.md (in-passing fix) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The #4491 host message-id field is accepted by the lane (payload.msgIds, src/incremental-lane.ts) but was never added to the documented envelope shape — readers could not discover it. One-line addition to the field list while this PR edits that very sentence for instanceId. --- PLUGIN.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/PLUGIN.md b/PLUGIN.md index eef9c2bef2..2e9efbf7c8 100644 --- a/PLUGIN.md +++ b/PLUGIN.md @@ -202,7 +202,7 @@ Boundary handling is `exact` when complete blocks fit inside the prefix, `expand Lineage-only declaration (#2408): a host that cannot drive the full fork protocol (no snapshot matching, e.g. claude `--fork-session`) may still declare the parent by sending a plain identity register — `POST /__bili/plugin/register` with `conversationId` (the fresh child id), `identity: true` and `parentConversationId` — from its SessionStart hook. The declaration scopes the proxy's content-match resume inheritance (#1486) to that parent chain, so a byte-exact replay of the parent's wire history seeds refs/blocks on the child's first request, and a non-matching replay falls back to the read-only parent link (#1333). A later register without `parentConversationId` never erases the declared parent. This is strictly weaker than §8 (no dead-ref namespace, no nudge inheritance, no receipt verification) but needs no snapshot access. -Incremental turns (#3248; not yet advertised in the plugin manifest — detect by trying the endpoint and handling the 503): a host that keeps its own full local history may stop resending it. `POST /__bili/plugin/turn` (same `/__bili/` gate; headers `x-bili-plugin` + optional `x-bili-plugin-model`, `x-bili-plugin-context-window`, `x-bili-plugin-max-output`) with `{ conversationId, sinceSeq, resync?, system?, path?, request, messages, instanceId? }`. `instanceId` is an OPTIONAL per-process client fingerprint (printable tokens, ≤128 chars): the lane uses it ONLY to warn when two DISTINCT sources replace the same canonical within a short window — two hosts sharing one conversation id flip each other's history into a 409 storm until both degrade to sticky full resend, and the warn is the only log evidence of that race (#4868); it is never persisted nor forwarded upstream, and older clients omitting it simply cannot be fingerprinted (no false alarm). The first turn (or any recovery) sends `sinceSeq: 0` + the full `messages[]` (`resync: true` is implied on seed — a wholesale replacement never 409s and works whether the canonical is empty or already live); every later turn sends ONLY what the client added since the last ack — including the previous assistant reply (client-owned). `request` carries anthropic params (model, max_tokens, temperature, top_p, top_k, stop_sequences, stream, tools, tool_choice, metadata, thinking, service_tier, parallel_tool_calls); `system` is sticky (omit to keep the last one); `path` defaults to `/v1/messages` (bare relative rides the lane's default upstream; the `/bili//…` form is explicit). The response is the upstream body streamed through, with `x-bili-applied-seq` = the canonical length after applying the delta — echo that value as the next turn's `sinceSeq`. `409 { error: "seq-mismatch" | "turn-in-flight", serverSeq }` means the client MUST recover with a resync turn (`sinceSeq: serverSeq` if it still has the history, else `resync: true` full seed). `503` means the lane runs without `storage.unified` (the canonical log lives in the unified messages table). The lane owns no session identity of its own: the assembled request carries the client's own `x-bili-plugin-conversation`, so incremental and normal full-resend traffic for one conversation share one session, one fold ledger and one canonical — switching modes needs no ceremony, and a conversation may interleave them (e.g. fall back to full resend while a resync is pending). Folding, nudges, tools and the plugin tool endpoints all work unchanged on the assembled body; the model never sees the difference. v1 caveats: the canonical log is the session's messages table — a turn is acked after the assembled body is persisted (best-effort immediate flush; a crash in between surfaces as 409 → resync), and the pipeline still walks the assembled prefix (O(full) CPU, cheap and zero-hit — the server-authored fast path is tracked on #3248). A `resync: true` that replaces a diverged canonical records a base-seq epoch (`session_ext` key `inc:base-seq`): the stale branch's rows stay in the append-only log for audit, but cold rebuilds slice past them — one resync heals, restarts included. On legacy-only storage (no ext surface) the marker degrades and cold rebuilds serve the full row union (409 → resync once, as before). +Incremental turns (#3248; not yet advertised in the plugin manifest — detect by trying the endpoint and handling the 503): a host that keeps its own full local history may stop resending it. `POST /__bili/plugin/turn` (same `/__bili/` gate; headers `x-bili-plugin` + optional `x-bili-plugin-model`, `x-bili-plugin-context-window`, `x-bili-plugin-max-output`) with `{ conversationId, sinceSeq, resync?, system?, path?, request, messages, msgIds?, instanceId? }` (`msgIds`: optional array of host-side message ids, parallel to `messages`, #4491). `instanceId` is an OPTIONAL per-process client fingerprint (printable tokens, ≤128 chars): the lane uses it ONLY to warn when two DISTINCT sources replace the same canonical within a short window — two hosts sharing one conversation id flip each other's history into a 409 storm until both degrade to sticky full resend, and the warn is the only log evidence of that race (#4868); it is never persisted nor forwarded upstream, and older clients omitting it simply cannot be fingerprinted (no false alarm). The first turn (or any recovery) sends `sinceSeq: 0` + the full `messages[]` (`resync: true` is implied on seed — a wholesale replacement never 409s and works whether the canonical is empty or already live); every later turn sends ONLY what the client added since the last ack — including the previous assistant reply (client-owned). `request` carries anthropic params (model, max_tokens, temperature, top_p, top_k, stop_sequences, stream, tools, tool_choice, metadata, thinking, service_tier, parallel_tool_calls); `system` is sticky (omit to keep the last one); `path` defaults to `/v1/messages` (bare relative rides the lane's default upstream; the `/bili//…` form is explicit). The response is the upstream body streamed through, with `x-bili-applied-seq` = the canonical length after applying the delta — echo that value as the next turn's `sinceSeq`. `409 { error: "seq-mismatch" | "turn-in-flight", serverSeq }` means the client MUST recover with a resync turn (`sinceSeq: serverSeq` if it still has the history, else `resync: true` full seed). `503` means the lane runs without `storage.unified` (the canonical log lives in the unified messages table). The lane owns no session identity of its own: the assembled request carries the client's own `x-bili-plugin-conversation`, so incremental and normal full-resend traffic for one conversation share one session, one fold ledger and one canonical — switching modes needs no ceremony, and a conversation may interleave them (e.g. fall back to full resend while a resync is pending). Folding, nudges, tools and the plugin tool endpoints all work unchanged on the assembled body; the model never sees the difference. v1 caveats: the canonical log is the session's messages table — a turn is acked after the assembled body is persisted (best-effort immediate flush; a crash in between surfaces as 409 → resync), and the pipeline still walks the assembled prefix (O(full) CPU, cheap and zero-hit — the server-authored fast path is tracked on #3248). A `resync: true` that replaces a diverged canonical records a base-seq epoch (`session_ext` key `inc:base-seq`): the stale branch's rows stay in the append-only log for audit, but cold rebuilds slice past them — one resync heals, restarts included. On legacy-only storage (no ext surface) the marker degrades and cold rebuilds serve the full row union (409 → resync once, as before). Marker divert (#3248 phase 2 — native extensions): instead of switching the provider URL to the turn endpoint, a host may send the incremental body on its ORDINARY model request. Stamp `x-bili-plugin-turn: 1` on the request (the lane requires `x-bili-plugin` too, as for every plugin signal); the body shape — top-level `conversationId` + `sinceSeq` + `messages`, no top-level `model` — is the final word: a marker'd body that is NOT incremental-shaped rides the normal pipeline byte-identically, so a host may stamp the marker whenever the feature is enabled and only convert turns it wants. Incremental-SHAPED turns are lane traffic and carry the same trust class as the direct turn endpoint (#3343): non-loopback / tunnel / untrusted-origin callers get 403 on them (their capability probe already 403s, so a well-behaved host never sends deltas from afar), and management prefixes (`/__bili/`, `/__acp/`) never ride the divert. The lane forwards the assembled request to the ORIGINAL request path (bare or `/bili/…` — no provider reconfiguration). Capability probe: `GET /__bili/plugin/turn` → `{ ok, protocol: 1, feature: "pluginIncrementalTurns", enabled, storage }` (loopback-gated like every `/__bili/` surface; a 404 = older server, stay full-resend). Gating: rollout feature `pluginIncrementalTurns` (default 0% = every install byte-identical to before) — enable with `rollout.features.pluginIncrementalTurns: true` (or a percent) in the bili config, or `BILI_ROLLOUT_PLUGIN_INCREMENTAL_TURNS=1`; the feature additionally requires a live unified sidecar (shadow or full). When disabled mid-flight the lane answers 503 on incremental-shaped turns — fall back to a plain full-resend request. Reference host wiring: pi rides its `before_provider_headers` (marker) + `before_provider_request` (payload replacement) + `after_provider_response` (ack) extension events; dsh-native rides the fetch patch's `beforeSend` (body conversion) + `headersFor` (marker) + `onResponse` (ack). pi-native inherits the pi wiring unchanged (same extension factory — only proxy discovery differs). opencode rides its `http.request` hook (an `incremental` adapter callback appended after routing/stamping swaps the request body via `replaceRequestPayload` and stamps the marker on the replaced request; acks land on the fetch intercept's `onResponse`). omp has no header events at all — its native adapter reads the extension-stamped `prompt_cache_key` body field as the conversation id inside `beforeSend`, stamps the plugin headers directly on the request init (identity upgrades, never forks: wire mode already resolves the same value via `preferPromptCacheKeyIdentity`), and acks on `onResponse`. Non-Anthropic wire shapes (OpenAI `tool_calls`/`role:"tool"`) are detected by `wireMessagesLookAnthropic` and left on full resend — the lane's message ingestion is the anthropic wire. Planned turns are single-in-flight per URL and age out after 30s (a network-dead request must not block the URL forever). The shared client state machine lives in `src/agent/incremental-client.ts` (probe TTL cache, prefix-hash diffing, sticky system, per-conversation dead flag) — new native hosts should reuse it rather than re-implement the diffing.