Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag

## [Unreleased]

### Fixed

- **增量车道:拉取回执修复 ack 传输盲区 + 漂移回退取富侧(#4352/#4323)**。2026-10-22 事故双根因修复:① pi 的 after_provider_response 只在部分 provider 传输下暴露响应头,x-bili-applied-seq 丢失时 noteIncrementalAck 永不执行,后续每轮都整包 re-seed(sinceSeq=0 重发 21K 条历史)。现在 GET /__bili/plugin/turn?conversation=<id> 返回热 canonical 的 {appliedSeq, boundary}(boundary=末条 wire 消息哈希,冷重建诚实返回 null),pi 扩展在规划下一轮前拉取回执,仅在 boundary 证明 canonical 吸收了我方末条消息时采纳——绝不按 seq 前进盲目采纳(外来的全量重发也会推进 canonical);② #4323 多实例写同一目录时,漂移回退无条件吃旧信封,把索引里更新的折叠世代(26,971 refs/16 active)换成旧世代(11,849/12),覆盖与 canonical 失配→0 可压区间→preflight 8.29M 假性死局 502。现在漂移两侧按 mapped ref 数取富侧(平局信封赢,平局语义=原子整文件)。新增 tests/incremental-ack-pull.test.ts 3 例(响应头 ack 回归钉/拉取采纳/冷 boundary null 拒采纳)+ storage-full-flip 4b(陈旧信封让位新索引世代)。

### Changes

- **kernel: aged compress receipts fold (#4348).** `compress` exchanges previously enjoyed a hard-coded, age-independent folding exemption (`ALWAYS_PROTECTED_TOOLS`), so every successful compression receipt stayed on the wire forever — a long session accumulated hundreds of `[Compressed mX-mY ...]` stubs (measured: 148 receipts / ~170K chars riding every request) that no range could reclaim, and ranges over them died with "Range would split N turn(s)" once the receipts interleaved with re-anchored coverage. Now only the newest `COMPRESS_EXCHANGE_KEEP_RECENT = 3` compress exchanges keep the exemption (working memory: the model can still see what it just folded); older exchanges fold like ordinary tool messages, with call+receipt+reasoning kept whole by the existing turn-integrity gate. The summaries the model wrote live on verbatim in the rendered block anchors, so decompress/search lose nothing. Host `protectedTools`/`isToolProtected` patterns still win for non-compress tools; for compress-dialect exchanges the kernel aging policy outranks host pinning (same doctrine as #4289 — without it, MCP-lane forms pinned via the host `isToolProtected` hook could never age out). `acp_rule` keeps its blanket exemption.
Expand Down
41 changes: 41 additions & 0 deletions src/agent/incremental-client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,47 @@ function wireHash(m: IncrementalWireMessage): string {
return createHash("sha256").update(`${m.role}\u0000${JSON.stringify(m.content)}`).digest("hex").slice(0, 16);
}

/** Public twin of the internal wire hash for boundary comparisons. The
* server-side hashWireBoundary (src/incremental-lane.ts) must stay
* byte-identical to this formula. */
export function incrementalWireHash(m: IncrementalWireMessage): string {
return wireHash(m);
}

/** Receipt served by GET /__bili/plugin/turn?conversation=<id> — the warm
* canonical's appliedSeq plus the wire-hash of its last applied message.
* A null boundary means "cold, cannot vouch" and must never be adopted. */
interface IncrementalReceipt {
appliedSeq: number;
boundary: string | null;
}

/** Fresh (never cached) receipt pull for hosts whose transport hides the
* x-bili-applied-seq response header (#4352: pi's after_provider_response
* exposes headers only for some provider transports — the ack channel
* cannot depend on it). Any failure → null; the caller keeps its pending
* plan and the next turn retries. */
export async function pullIncrementalReceipt(
proxyBase: string,
conversationId: string,
fetchImpl: typeof fetch = fetch,
): Promise<IncrementalReceipt | null> {
try {
const url = `${proxyBase.replace(/\/$/, "")}/__bili/plugin/turn?conversation=${encodeURIComponent(conversationId)}`;
const res = await fetchImpl(url, { headers: { "x-bili-plugin": "pi" }, signal: AbortSignal.timeout(3000) });
if (res.status !== 200) return null;
const body = (await res.json()) as { ok?: unknown; conversation?: { appliedSeq?: unknown; boundary?: unknown } };
if (body.ok !== true || typeof body.conversation !== "object" || body.conversation === null) return null;
const seq = body.conversation.appliedSeq;
const boundary = body.conversation.boundary;
if (typeof seq !== "number" || !Number.isSafeInteger(seq) || seq < 0) return null;
if (boundary !== null && typeof boundary !== "string") return null;
return { appliedSeq: seq, boundary: boundary as string | null };
} catch {
return null;
}
}

type IncrementalPlan =
| { mode: "seed"; body: string } // full history, sinceSeq 0 (server may 409 → resync)
| { mode: "delta"; body: string } // suffix only, acked sinceSeq
Expand Down
21 changes: 20 additions & 1 deletion src/agent/pi.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import { disposeSubagentSelfReg, selfRegisterForSession, type SubagentSelfRegSta
import { isModelApiUrl, nativeInterceptInstalled } from "./native-intercept.js";
import { wirePiSubagents } from "./pi-subagents.js";
import { wireToolGuardrails } from "./tool-guardrails.js";
import { clearIncrementalProbe, INCREMENTAL_TURN_MARKER_HEADER, incrementalEnvelopeConversationId, noteIncrementalRejection, noteIncrementalAck, planIncrementalTurn, probeIncrementalTurns, sweepStale, type IncrementalWireMessage } from "./incremental-client.js";
import { clearIncrementalProbe, INCREMENTAL_TURN_MARKER_HEADER, incrementalEnvelopeConversationId, incrementalWireHash, noteIncrementalRejection, noteIncrementalAck, planIncrementalTurn, probeIncrementalTurns, pullIncrementalReceipt, sweepStale, type IncrementalWireMessage } from "./incremental-client.js";
import type { DirectSendPrep } from "./native-intercept.js";
import { asciiHeaderValue, detectProxyBase, destinationRoutedThroughProxy, fetchManifest, forwardTool, fetchStatus, fetchProxyVersion, postIdentityRegister, reportRuntimeInfoOnChange, armedIdleNotice, noSessionWarning, nonHttpProvidersFromEnv, type ForwardedToolResult, type ManifestTool } from "./shared.js";
import { createForkAdopter } from "./fork-adopt.js";
Expand Down Expand Up @@ -1339,6 +1339,25 @@ export function createBiliPlugin(agentOverride?: string, opts?: { retryIntervalM
if (proxyBase === undefined) proxyBase = await awaitNativeProxyOrigin();
if (proxyBase === undefined) return undefined;
if (!(await probeIncrementalTurns(proxyBase))) return undefined;
// #4352: the response-header ack channel is transport-dependent —
// pi's after_provider_response only exposes headers for some
// provider transports, and a missed ack makes EVERY next turn
// re-seed the full history (the 2026-10-22 incident: turn 2 sent
// sinceSeq=0 again). Before planning, reconcile any still-pending
// un-acked turn by PULLING the lane's receipt and adopting it only
// when the boundary hash proves the canonical absorbed OUR last
// message — never on mere seq progress (a foreign full-resend can
// advance the canonical too).
const pendingAck = pendingIncrementalAcks.get(sid);
if (pendingAck !== undefined && pendingAck.messages.length > 0) {
const receipt = await pullIncrementalReceipt(proxyBase, sid);
const expected = incrementalWireHash(pendingAck.messages[pendingAck.messages.length - 1]);
if (receipt !== null && receipt.appliedSeq > 0 && receipt.boundary === expected) {
pendingIncrementalAcks.delete(sid);
piIncrementalOriginals.delete(sid);
noteIncrementalAck(sid, receipt.appliedSeq, pendingAck.messages, pendingAck.system);
}
}
const plan = planIncrementalTurn({
conversationId: sid,
// The marker divert uses the ORIGINAL request URL server-side;
Expand Down
36 changes: 34 additions & 2 deletions src/incremental-lane.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@
* (R1 message ids), durable acks.
*/
import http from "node:http";
import { createHash } from "node:crypto";
import { Readable } from "node:stream";
import { anthropicToCore, coreToAnthropic, type BiliMessage } from "acp-kernel/wire";
import { getStore } from "./persist.js";
Expand Down Expand Up @@ -162,6 +163,12 @@ interface Canonical {
marks: Map<string, unknown>;
system: unknown;
systemSet: boolean;
/** #4352 ack pull: hash of the last WIRE message the canonical was
* advanced with (the client-side incrementalWireHash formula over the
* delta's final message). Warm-cache only — a cold rebuild has no wire
* bytes to hash and reports null; clients treat a null boundary as
* "cannot adopt, keep full resend" and self-heal on the next turn. */
boundary: string | null;
}
const canonicalCache = new Map<string, Canonical>();
const inFlight = new Set<string>();
Expand Down Expand Up @@ -386,6 +393,14 @@ function loadSystem(sessionId: string): unknown {
}

/** Seed/append/resync validation + application. Pure over the canonical. */
/** Byte-identical to the client-side incrementalWireHash
* (src/agent/incremental-client.ts) over the raw wire message — the two
* formulas must never diverge or the pull-ack boundary check degrades into
* permanent false mismatches. */
function hashWireBoundary(m: { role: string; content: unknown }): string {
return createHash("sha256").update(`${m.role}\u0000${JSON.stringify(m.content)}`).digest("hex").slice(0, 16);
}

function applyTurn(
entry: Canonical,
sinceSeq: number,
Expand All @@ -410,6 +425,9 @@ function applyTurn(
const base = resync ? [] : entry.msgs;
entry.msgs = [...base, ...flat.msgs];
for (const [id, mark] of flat.cacheControls) entry.marks.set(id, mark);
entry.boundary = deltaWire.length > 0
? hashWireBoundary(deltaWire[deltaWire.length - 1])
: (resync ? null : entry.boundary);
return { ok: true, appliedSeq: entry.msgs.length, marks: flat.cacheControls };
}

Expand Down Expand Up @@ -457,20 +475,34 @@ export function incrementalTurnsEnabled(): boolean {
/** GET probe on the turn path: capabilities for native extensions. A 404 on
* older servers means "no incremental support" — clients fall back to full
* resend, which is the default everywhere. */
export function handleIncrementalProbe(res: http.ServerResponse): void {
export function handleIncrementalProbe(res: http.ServerResponse, conversationId?: string): void {
const sidecar = (() => {
try {
return getStore().replicasAvailable();
} catch {
return false;
}
})();
// #4352 ack pull: `?conversation=<id>` upgrades the probe into a receipt
// poll. Served from the WARM canonical only — a cold rebuild here must
// not touchCache (a probe is not a turn), and a rebuilt entry has no wire
// boundary to hash, so an honest null forces the client to keep full
// resend for one more turn instead of adopting a guess.
let conversation: { appliedSeq: number; boundary: string | null } | undefined;
if (conversationId !== undefined && conversationId.length > 0) {
const entry = canonicalCache.get(conversationId);
conversation = {
appliedSeq: entry !== undefined && entry.sessionId === laneSessionId(conversationId) ? entry.msgs.length : 0,
boundary: entry !== undefined && entry.sessionId === laneSessionId(conversationId) ? entry.boundary : null,
};
}
jsonReply(res, 200, {
ok: true,
protocol: 1,
feature: "pluginIncrementalTurns",
enabled: featureEnabled("pluginIncrementalTurns") && sidecar,
storage: sidecar ? "unified" : "legacy-only",
...(conversation !== undefined ? { conversation } : {}),
});
}

Expand Down Expand Up @@ -554,7 +586,7 @@ export async function handleIncrementalTurn(
log("warn", `[incremental] canonical rebuild failed for ${sessionId}: ${rebuilt.err} — requesting client resync`);
return jsonReply(res, 409, { error: "seq-mismatch", serverSeq: 0, reason: rebuilt.err, conversationId });
}
entry = { sessionId, msgs: rebuilt.msgs, marks: loadMarks(sessionId), system: loadSystem(sessionId), systemSet: false };
entry = { sessionId, msgs: rebuilt.msgs, marks: loadMarks(sessionId), system: loadSystem(sessionId), systemSet: false, boundary: null };
}
const applied = applyTurn(entry, sinceSeq, resync, deltaWire as never, "system" in payload);
if (!applied.ok) {
Expand Down
21 changes: 21 additions & 0 deletions src/persist.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1439,6 +1439,27 @@ export class SessionStore {
const driftedEnvelope = this.loadEnvelope(id, meta);
if (driftedEnvelope) {
const drifted = buildSession(driftedEnvelope.payload);
// #4323: "drifted" only proves the two copies DIVERGED —
// not which side is ahead. With multiple writers sharing a
// sessions directory (the 2026-10-22 incident) the INDEX
// can hold the newer generation: serving the stale
// envelope resurrected an older fold state whose coverage
// no longer matched the canonical rows, the raw history
// replayed unfolded (0 compressible ranges) and preflight
// dead-ended at a phantom 8.29M tokens. Compare both
// sides' mapped-ref counts and serve the richer one; the
// envelope keeps winning ties (its bytes are whole-file
// atomic, the index's superiority must be visible).
const unifiedView = this.loadUnified(id);
if (unifiedView) {
const envRefs = Object.keys((drifted.state as { messageRefs?: { byRef?: Record<string, unknown> } }).messageRefs?.byRef ?? {}).length;
const idxState = unifiedView.state as { messageRefs?: { byRef?: Record<string, unknown> } };
const idxRefs = Object.keys(idxState.messageRefs?.byRef ?? {}).length;
if (idxRefs > envRefs) {
this.log("warn", `[persist] drift on ${id}: unified index carries ${idxRefs} mapped ref(s) vs envelope ${envRefs} — serving the richer index side (multi-writer generation, #4323)`);
return this.afterLoad(id, unifiedView);
}
}
if (hasNegativePersistedTokens(driftedEnvelope.payload)) {
this.scheduleSave(drifted);
this.log("info", `[persist] clamped negative token stats on reload for ${id} (#408)`);
Expand Down
3 changes: 2 additions & 1 deletion src/server/handle.ts
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,8 @@ export async function handle(
res.end(JSON.stringify({ error: denial }));
return;
}
handleIncrementalProbe(res);
const conv = new URL(req.url ?? "/", "http://localhost").searchParams.get("conversation");
handleIncrementalProbe(res, conv !== null && conv.length > 0 ? conv : undefined);
return;
}
// #3248 marker divert: a native extension may stamp
Expand Down
Loading
Loading