diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 09e953c4c5..d59b7e9cc6 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -174,8 +174,6 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | Repeat interval for the EPERM alert; 0 logs it once. | | `storage` | object | {} (all defaults) | — | Unified SQLite storage sidecar settings (#2671). | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | Tiered storage-migration switch (#2671): shadow = unified SQLite double-write dry-run with digest reconciliation; full = drop-in read path (unified-first loads with organic import, legacy fallback). Absent everywhere now defaults to FULL (#3818 flip - the shadow dry-run reconciled cleanly since #2815, so the index is the load path; JSON envelopes keep being written, and legacy remains the one-line escape hatch). File tier is strict legacy/shadow/full; the env tier also accepts 1/true (=shadow) and unified (=full) and wins over the file. Resolved once per lane boot - hot edits need a restart. Encryption (BILI_ENCRYPTION_KEY) hard-blocks the sidecar. | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). | -| `storage.unified` | legacy\|shadow\|full | unset (shadow) | BILI_STORAGE_UNIFIED | Tiered storage-migration switch (#2671): shadow = unified SQLite double-write dry-run with digest reconciliation; full = drop-in read path (unified-first loads with organic import, legacy fallback). Absent everywhere now defaults to shadow (JSON stays the sole source of truth; the index is digest-reconciled telemetry - set legacy to opt out). File tier is strict legacy/shadow/full; the env tier also accepts 1/true (=shadow) and unified (=full) and wins over the file. Resolved once per lane boot - hot edits need a restart. Encryption (BILI_ENCRYPTION_KEY) hard-blocks the sidecar. | | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 758505a848..457bc4cbb3 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -174,8 +174,6 @@ | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | EPERM 告警的重复间隔;0 只记录一次。 | | `storage` | object | {} (all defaults) | — | 统一 SQLite 存储边车设置(#2671)。 | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | 分层存储迁移开关(#2671):shadow = 统一 SQLite 双写干跑 + digest 对账;full = 平替读路径(unified-first 加载 + 有机导入 + legacy 兜底)。两处均未设置时默认 full(#3818 翻转 —— shadow 干跑自 #2815 起对账一直干净,索引因此成为默认加载路径;JSON 信封照写,legacy 仍是一行逃生口)。文件层只接受 legacy/shadow/full;环境变量层还接受 1/true(=shadow)与 unified(=full)并优先于文件。每个 lane 启动时解析一次 —— 热改动需重启生效。加密(BILI_ENCRYPTION_KEY)会硬阻断边车。 | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。 | -| `storage.unified` | legacy\|shadow\|full | unset (shadow) | BILI_STORAGE_UNIFIED | 分层存储迁移开关(#2671):shadow = 统一 SQLite 双写干跑 + digest 对账;full = 平替读路径(unified-first 加载 + 有机导入 + legacy 兜底)。两处均未设置时默认 shadow(JSON 仍是唯一事实源,索引只做对账遥测 —— 设 legacy 可退出)。文件层只接受 legacy/shadow/full;环境变量层还接受 1/true(=shadow)与 unified(=full)并优先于文件。每个 lane 启动时解析一次 —— 热改动需重启生效。加密(BILI_ENCRYPTION_KEY)会硬阻断边车。 | | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | diff --git a/src/agent/incremental-client.ts b/src/agent/incremental-client.ts index dc1451d52b..0a1ed6d72e 100644 --- a/src/agent/incremental-client.ts +++ b/src/agent/incremental-client.ts @@ -241,7 +241,7 @@ interface PlanArgs { * optional — the lane defaults a bare `path` to /v1/messages. */ path?: string; messages: IncrementalWireMessage[]; - /** Raw wire system field; omit key entirely when the host sent none. */ + /** System from the complete host request; undefined means the host sent none. */ system: unknown; /** Anthropic request params from the original body (model, max_tokens…); * optional — missing params ride the lane's defaults. */ @@ -263,7 +263,7 @@ function buildBody(args: PlanArgs, sinceSeq: number, messages: IncrementalWireMe messages, }; if (resync) body.resync = true; - if (includeSystem) body.system = args.system; + if (includeSystem) body.system = args.system === undefined ? null : args.system; return JSON.stringify(body); } @@ -272,8 +272,8 @@ function buildBody(args: PlanArgs, sinceSeq: number, messages: IncrementalWireMe export function planIncrementalTurn(args: PlanArgs, forceResync = false): IncrementalPlan { const state = convs.get(args.conversationId); if ((state !== undefined && state.dead) || args.messages.length === 0) return { mode: "skip" }; + const systemJson = args.system === undefined ? undefined : JSON.stringify(args.system); if (state === undefined || forceResync) { - const systemJson = args.system === undefined ? undefined : JSON.stringify(args.system); const includeSystem = state === undefined || systemJson !== state.systemJson; // First attempt (and every forced replan) sends the FULL set with // resync:true — a wholesale canonical replacement never 409s, so the @@ -291,23 +291,21 @@ export function planIncrementalTurn(args: PlanArgs, forceResync = false): Increm if (state.sent[i] !== hashes[i]) { aligned = false; break; } } if (!aligned || hashes.length < state.sent.length) { - return { mode: "resync", body: buildBody(args, 0, args.messages, true, args.system === undefined ? false : JSON.stringify(args.system) !== state.systemJson) }; + return { mode: "resync", body: buildBody(args, 0, args.messages, true, systemJson !== state.systemJson) }; } if (hashes.length === state.sent.length) { // Equal message set: normally a retry of the same turn → normal path. // But a SYSTEM change is still a real outgoing turn — send an empty // delta so the lane forwards the assembled request with the new // sticky system (the canonical itself is unchanged). - const sysJson = args.system === undefined ? undefined : JSON.stringify(args.system); - if (sysJson !== undefined && sysJson !== state.systemJson) { + if (systemJson !== state.systemJson) { return { mode: "delta", body: buildBody(args, state.sinceSeq, [], false, true) }; } return { mode: "skip" }; // retry of the same turn → normal path } - const systemJson = args.system === undefined ? undefined : JSON.stringify(args.system); return { mode: "delta", - body: buildBody(args, state.sinceSeq, args.messages.slice(state.sent.length), false, systemJson !== undefined && systemJson !== state.systemJson), + body: buildBody(args, state.sinceSeq, args.messages.slice(state.sent.length), false, systemJson !== state.systemJson), }; } diff --git a/src/log-mask.ts b/src/log-mask.ts index ac2440b279..d34a619b1b 100644 --- a/src/log-mask.ts +++ b/src/log-mask.ts @@ -118,7 +118,7 @@ export function maskUrlsInText(text: string): string { * preference must not weaken the publish boundary (same stance as * redactSecretsInText, which is never gated by the switch). Userinfo, query * and hash are dropped exactly as in maskUrlForLog; the path is kept. */ -export function maskUrlForPublish(url: string): string { +function maskUrlForPublish(url: string): string { let u: URL; try { u = new URL(url); diff --git a/tests/incremental-lane.test.ts b/tests/incremental-lane.test.ts index cf71b9edf2..34a7d48be7 100644 --- a/tests/incremental-lane.test.ts +++ b/tests/incremental-lane.test.ts @@ -436,6 +436,52 @@ test("incremental marker divert: system-only change rides an empty delta — can } }); +for (const scenario of ["append", "system-only", "edit", "forced resync", "fresh client"] as const) { + test(`incremental client: removed system clears the upstream on ${scenario}`, async () => { + const { __resetIncrementalClientForTests, planIncrementalTurn, noteIncrementalAck } = await import("../src/agent/incremental-client.js"); + const priorRollout = process.env.BILI_ROLLOUT_PLUGIN_INCREMENTAL_TURNS; + process.env.BILI_ROLLOUT_PLUGIN_INCREMENTAL_TURNS = "1"; + __resetIncrementalClientForTests(); + let h: Harness | undefined; + try { + h = await harness("full"); + const conversationId = "conv-system-clear"; + const original = [{ role: "user", content: "original request" }]; + const seed = planIncrementalTurn({ conversationId, messages: original, system: "OLD-SYSTEM-INSTRUCTION" }); + assert.notEqual(seed.mode, "skip"); + if (seed.mode === "skip") return; + const seeded = await h.markerTurn(h.path, JSON.parse(seed.body)); + assert.equal(seeded.status, 200); + assert.equal(seeded.appliedSeq, original.length); + assert.ok(JSON.stringify(h.forwarded.at(-1)?.system).includes("OLD-SYSTEM-INSTRUCTION")); + noteIncrementalAck(conversationId, original.length, original, "OLD-SYSTEM-INSTRUCTION"); + if (scenario === "fresh client") __resetIncrementalClientForTests(); + const messages = scenario === "append" + ? [...original, { role: "assistant", content: "answer" }, { role: "user", content: "next request" }] + : scenario === "edit" ? [{ role: "user", content: "edited request" }] : original; + const plan = planIncrementalTurn({ conversationId, messages, system: undefined }, scenario === "forced resync"); + assert.equal(plan.mode, scenario === "fresh client" ? "seed" : scenario === "edit" || scenario === "forced resync" ? "resync" : "delta"); + const cleared = await h.markerTurn(h.path, JSON.parse(plan.body)); + assert.equal(cleared.status, 200); + assert.equal(cleared.appliedSeq, messages.length); + assert.equal(JSON.stringify(h.forwarded.at(-1)?.system ?? "").includes("OLD-SYSTEM-INSTRUCTION"), false, "removed original system must not be forwarded"); + noteIncrementalAck(conversationId, messages.length, messages, undefined); + const next = planIncrementalTurn({ conversationId, messages: [...messages, { role: "user", content: "another request" }], system: undefined }); + assert.equal(next.mode, "delta"); + if (next.mode !== "delta") return; + assert.equal(Object.hasOwn(JSON.parse(next.body), "system"), false, "acknowledged absence stays omitted on subsequent deltas"); + const followed = await h.markerTurn(h.path, JSON.parse(next.body)); + assert.equal(followed.status, 200); + assert.equal(JSON.stringify(h.forwarded.at(-1)?.system ?? "").includes("OLD-SYSTEM-INSTRUCTION"), false); + } finally { + if (priorRollout === undefined) delete process.env.BILI_ROLLOUT_PLUGIN_INCREMENTAL_TURNS; + else process.env.BILI_ROLLOUT_PLUGIN_INCREMENTAL_TURNS = priorRollout; + __resetIncrementalClientForTests(); + await h?.close(); + } + }); +} + // --- client helper state machine (src/agent/incremental-client.ts) ------------------ test("incremental client helper: seed implies resync, delta slices, edits resync, sticky system", async () => {