Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag

### Fixed

- **win32 统一存储降级提示:READONLY 签名点名 + OS 级拒绝不再建议删 index.db(#4614)**。现场报告(v0.1.205,win32-x64):index.db 已存在且早期可写,某次重启后打开报 `attempt to write a readonly database`——与 #3823 同一类 OS 级拒写(完整性标签/AV/并发写者)在文件已存在时的另一种表面(打开成功、第一次写入被拒 → READONLY 而非 CANTOPEN);但既有提示只认 CANTOPEN 字符串,诊断 reason 零可操作信息,warn 行结尾还跟着「delete the file to reset the index」——full 模式下该建议有害(index.db 可能是被冻结会话近期状态的唯一副本)。修法:`sqliteOpenFailureHint` 扩展覆盖 READONLY 签名(点名完整性标签嫌疑、icacls 查标签命令、递归 Low 重打标、AV、并发写者,明确 do NOT delete);degrade 尾部改为「hint 触发时不附删除建议」(CANTOPEN/READONLY 同享;真损坏文件与非 win32 平台维持原样)。测试:`tests/storage-cantopen-hint.test.ts` 新增 5 例(READONLY 正负双向 + 0444 文件确定性复现 READONLY 降级 wiring + stub-win32 删除建议抑制钉 + 原生平台双向),全绿。
- **master CI 全红修复:#4390 historian 的媒体引用释放 + #3250 WAL cap 断言预算(#4506)。** bisect 定位 f161c518..02f544da 窗口 first bad = #4390(row-identity-historian),两个破口:① 权威集清扫改墓碑后,retired 行的 message_media 映射与媒体 CAS 引用永远不释放(默认要等 archive 365 天)——图片重度会话里即 #3762/#3901 的字节泄漏,`media-cas` 家族在 windows 分片稳定红。现在 retire 时释放媒体映射+引用(文本字节照旧史官保留;图片体积大、历史价值低),rewind-redo 复活时由重摄取路径幂等重建映射与引用(sqlite+mysql 双后端同款)。② `#3250: configured cap` 断言的 CAP_SLACK=4KiB(一页余量)预算不住 #4390 新增的每行 WAL 体积(identity 列 64hex + idx_messages_identity 索引项,~192 行超 12 页/47,376B)——截断不变量本身完好,坏的是页级余量的算术;custom-cap 用例改用 64KiB 的 inter-reset 累积预算(注释引用 #4390 行体积),close-converges 等其余断言维持一页余量不变。
- **External summary: stop injecting the summary-writing rules into the conversation prompt (#4460).** With `compress.externalSummary.enabled`, the conversation model only selects ranges and the independent service writes the committed summary — yet every conversation request still carried `HOW TO COMPRESS`/`KEEP VERBATIM`/`DROP`/`PRIORITY` (~5.8K chars) plus the per-summary length budget, which the model's non-authoritative hint is never checked against. The conversation-side prompt now drops both while the chain is enabled (all five prompt assembly sites, function and hybrid builders, pack overrides included); philosophy, ACP tags, tools, summaries-in-context and marker integrity stay. The summary requests keep the full rules (`session.meta.summaryInstructions`), and classic mode is byte-identical. New tests/issue4460-external-summary-conversation-prompt.test.ts (7 cases, positive + negative).
- **增量车道:拉取回执修复 ack 传输盲区 + 漂移回退取富侧(#4352/#4323)**。2026-10-22 事故双根因修复:① pi 的 after_provider_response 只在部分 provider 传输下暴露响应头,x-bili-applied-seq 丢失时 noteIncrementalAck 永不执行,后续每轮都整包 re-seed(sinceSeq=0 重发 21K 条历史)。现在 GET /__bili/plugin/turn?conversation=<id> 返回热 canonical 的 {appliedSeq, boundary}(boundary=末条 wire 消息哈希,冷重建诚实返回 null),pi 扩展在规划下一轮前拉取回执,仅在 boundary 证明 canonical 吸收了我方末条消息时采纳——绝不按 seq 前进盲目采纳(外来的全量重发也会推进 canonical);② #4323 多实例写同一目录时,漂移回退无条件吃旧信封,把索引里更新的折叠世代(26,971 refs/16 active)换成旧世代(11,849/12),覆盖与 canonical 失配→0 可压区间→preflight 8.29M 假性死局 502。现在漂移两侧按 mapped ref 数取富侧(平局信封赢,平局语义=原子整文件)。新增 tests/incremental-ack-pull.test.ts 3 例(响应头 ack 回归钉/拉取采纳/冷 boundary null 拒采纳)+ storage-full-flip 4b(陈旧信封让位新索引世代)。
Expand Down
47 changes: 32 additions & 15 deletions src/storage/shadow.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,20 +53,32 @@ export const VERIFY_EVERY_INGESTS = 20;
* every save, and the first clean sample re-admits the session. */
export const DRIFT_ESCALATE_STREAK = 3;

/** #3823: on win32 "unable to open database file" (SQLITE_CANTOPEN) is almost
* always an OS-level refusal — antivirus/Defender holding the dir, OneDrive
* placeholder files under the profile dir, a path past MAX_PATH, or a
* restricted/sandboxed process whose writes Windows integrity control refuses
* (#3823 field report: a DSH-launched child at low integrity hit exactly this
* class on its CA bundle) — NOT a corrupt index.db, so the generic "delete
* the file" recovery advice misleads for that class; name the real suspects
* instead. Empty elsewhere: every field instance of this failure domain is
* win32-x64 (#3338/#3385/#3788/#3823), and Linux/macOS CANTOPEN causes
* (permissions, disk full) are too varied to hint at usefully. */
/** #3823 + #4614: on win32 two open-time error strings are OS-level refusals,
* not a corrupt index.db, so the generic "delete the file" recovery advice
* misleads for that class — name the real suspects instead:
* - "unable to open database file" (SQLITE_CANTOPEN): antivirus/Defender
* holding the dir, OneDrive placeholder files, a path past MAX_PATH, or a
* restricted/sandboxed process (#3823 field report: a DSH-launched child at
* low integrity hit exactly this class on its CA bundle);
* - "attempt to write a readonly database" (SQLITE_READONLY family): the
* EXISTING index.db opened fine but the first write was refused — the
* #3823 mechanism surfacing differently once the file already exists
* (Windows integrity control / MIC No-Write-Up: a process launched at lower
* integrity than the file's label cannot write it; AV lock; exclusive
* concurrent writer). Deleting the file here is worse than useless: in
* unified-full mode it may hold the only copy of recent session state.
* Empty elsewhere: every field instance of this failure domain is win32-x64
* (#3338/#3385/#3788/#3823/#4614), and Linux/macOS causes (permissions, disk
* full) are too varied to hint at usefully. */
export function sqliteOpenFailureHint(err: string): string {
if (process.platform !== "win32") return "";
if (!/unable to open database file/i.test(err)) return "";
return " (win32: the OS refused to open the file — usually antivirus/Defender locking the sessions dir, OneDrive placeholder files under the profile dir, a path over 260 chars, or a restricted/sandboxed process (low-integrity label, app container) whose writes Windows integrity control refuses; exclude %USERPROFILE%\\.local\\share\\billion-context\\ from antivirus real-time scanning first; deleting the file will not clear a lock)";
if (/unable to open database file/i.test(err)) {
return " (win32: the OS refused to open the file — usually antivirus/Defender locking the sessions dir, OneDrive placeholder files under the profile dir, a path over 260 chars, or a restricted/sandboxed process (low-integrity label, app container) whose writes Windows integrity control refuses; exclude %USERPROFILE%\\.local\\share\\billion-context\\ from antivirus real-time scanning first; deleting the file will not clear a lock)";
}
if (/attempt to write a readonly database/i.test(err)) {
return " (win32: 'attempt to write a readonly database' on an EXISTING index.db = the file opened read-only and the first write was refused by the OS — most often Windows integrity control (a process running at Low integrity cannot write Medium-labeled files, and a DSH-spawned node.exe can itself carry a Low label, e.g. after a stray 'icacls /setintegritylevel Low' — #5154), less often antivirus/Defender or another bili/dsh process holding the files; check the PROCESS side first — run icacls on your node.exe and if it reports a Low label restore it with 'icacls <path-to-node.exe> /setintegritylevel Medium' — do NOT relabel the sessions dir Low to match (that cements the mislabel and drags every file down with it); do NOT delete the file — in unified-full mode it may hold the only copy of recent session state)";
}
return "";
}

// Canonical home for StorageMode/resolveStorageMode is src/knobs.ts (env-var
Expand Down Expand Up @@ -241,11 +253,16 @@ export class UnifiedShadow {
// blobs (#2671 Phase 2) — destructive recovery is operator policy.
const errStr = String(err);
// Deleting the index cannot fix a no-engine failure (this Node.js has neither builtin
// node:sqlite nor a loadable better-sqlite3) — that advice only fits a genuine open
// failure; the Node.js upgrade lever is surfaced via /__bili/status + the bootstrap warn.
// node:sqlite nor a loadable better-sqlite3), and it cannot clear an OS-level refusal
// either (#3823 lock; #4614 readonly — where it would DESTROY the only copy of recent
// session state in unified-full mode). The delete advice fits only a genuine corrupt
// file; the Node.js upgrade lever is surfaced via /__bili/status + the bootstrap warn.
const hint = sqliteOpenFailureHint(errStr);
const tail = /no SQLite engine available/i.test(errStr)
? ""
: `${sqliteOpenFailureHint(errStr)}; delete the file to reset the index`;
: hint !== ""
? hint
: "; delete the file to reset the index";
loggerLog(
"warn",
`[storage] unified sidecar failed to open ${dir}/index.db (${errStr}) — staying legacy-only${tail}`,
Expand Down
3 changes: 3 additions & 0 deletions tests/issue4323-drift-escalation.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,9 @@ test("#4323 A: three consecutive sampled drifts escalate once and freeze envelop
assert.equal(escalated.length, 1, "exactly ONE escalation alarm for the whole streak");
assert.ok(escalated[0].msg.includes("active block(s) vs index"), `alarm names both sides' fold counts: ${escalated[0].msg}`);
assert.ok(escalated[0].msg.includes("#394"), "alarm names the multi-writer anti-pattern");
// #4614: the alarm must name BOTH known producers — a single host re-writing
// slots below the watermark on every save produces the identical signature.
assert.ok(escalated[0].msg.includes("#4614"), "alarm also names the single-writer repeating-rewrite producer");
const resolved = captured.filter((e) => e.level === "info" && e.msg.includes(`drift resolved for ${ids}`));
assert.equal(resolved.length, 1, "re-admission is logged");
const driftErrors = captured.filter((e) => e.level === "error" && e.msg.startsWith(`[storage] unified shadow DIGEST DRIFT for ${ids}:`));
Expand Down
133 changes: 126 additions & 7 deletions tests/storage-cantopen-hint.test.ts
Original file line number Diff line number Diff line change
@@ -1,23 +1,33 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, mkdirSync } from "node:fs";
import { chmodSync, mkdtempSync, mkdirSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import * as path from "node:path";
import { sqliteOpenFailureHint } from "../src/storage/shadow.ts";
import { _resetStorageRuntimeStatusForTest, storageStatusWire } from "../src/storage/runtime-status.ts";
import { SessionStore } from "../src/persist.ts";
import { setLogCapture } from "../src/logger.ts";
import { rmrf } from "./tmp-rm.ts";

/**
* #3823: on win32, CANTOPEN ("unable to open database file") at index.db open
* time is almost always an OS-level environment refusal (AV/Defender lock,
* OneDrive placeholder files, MAX_PATH) — NOT a corrupt file, so the generic
* "delete the file to reset the index" advice misleads for that class. The
* hint fires exactly when win32 && CANTOPEN and stays silent otherwise —
* both directions pinned (#3617 gate discipline).
* #3823 + #4614: on win32, two open-time error strings are OS-level
* environment refusals, not a corrupt file, so the generic "delete the file
* to reset the index" advice misleads for that class (for READONLY it is
* worse than useless — deleting may destroy the only copy of recent session
* state in unified-full mode):
* - CANTOPEN ("unable to open database file"): AV/Defender lock, OneDrive
* placeholders, MAX_PATH, low-integrity process;
* - READONLY ("attempt to write a readonly database"): an EXISTING index.db
* opened but the first write was refused — the same integrity-control
* mechanism once the file already exists (#4614 field report).
* Each hint fires exactly when win32 && its signature and stays silent
* otherwise; when a hint fires the degrade line drops the delete advice,
* and a genuine corrupt-file failure keeps it — all directions pinned
* (#3617 gate discipline).
*/

const CANT_OPEN = "Error: [storage:node:sqlite] Error: unable to open database file";
const READONLY = "Error: [storage:node:sqlite] Error: attempt to write a readonly database";

function stubPlatform(platform: string): () => void {
const original = Object.getOwnPropertyDescriptor(process, "platform");
Expand Down Expand Up @@ -62,6 +72,31 @@ test("hint: non-win32 platforms stay silent even on CANTOPEN", () => {
}
});

test("hint: win32 + READONLY names integrity-control suspects and forbids deletion (#4614)", () => {
const restore = stubPlatform("win32");
try {
const hint = sqliteOpenFailureHint(READONLY);
assert.ok(hint.includes("win32"), "names the platform");
assert.ok(hint.includes("readonly"), "echoes the failure class");
assert.ok(hint.includes("integrity"), "points at Windows integrity control");
assert.ok(hint.includes("icacls"), "gives the label-check command");
assert.ok(hint.includes("do NOT delete"), "forbids deleting the live index");
} finally {
restore();
}
});

test("hint: non-win32 platforms stay silent even on READONLY", () => {
for (const p of ["linux", "darwin"]) {
const restore = stubPlatform(p);
try {
assert.equal(sqliteOpenFailureHint(READONLY), "", `expected no hint on ${p}`);
} finally {
restore();
}
}
});

const ENV_KEYS = ["BILI_SESSIONS_DIR", "BILI_CONFIG_FILE", "XDG_CACHE_HOME", "XDG_STATE_HOME", "BILI_STORAGE_UNIFIED", "BILI_ENCRYPTION_KEY"] as const;

function isolateEnv(base: string, sessions: string): Record<string, string | undefined> {
Expand Down Expand Up @@ -92,6 +127,8 @@ test("#3823 wiring: an unopenable index.db degrades with the platform-appropriat
// reports carry, minus the environment-specific cause.
mkdirSync(path.join(sessions, "index.db"));
const snap = isolateEnv(base, sessions);
const logs: string[] = [];
setLogCapture((level, msg) => logs.push(`${level} ${msg}`));
try {
_resetStorageRuntimeStatusForTest();
new SessionStore({ dir: sessions, enabled: true });
Expand All @@ -102,12 +139,94 @@ test("#3823 wiring: an unopenable index.db degrades with the platform-appropriat
wire.reason?.startsWith("index database failed to open"),
`reason names the open failure (got: ${wire.reason})`,
);
const degradeLine = logs.find((l) => l.includes("unified sidecar failed to open"));
assert.ok(degradeLine, "the degrade warn was logged");
if (process.platform === "win32") {
assert.ok(wire.reason?.includes("(win32:"), "win32 runner carries the OS-level hint");
assert.ok(
!degradeLine.includes("delete the file to reset the index"),
"an OS-level refusal must not carry the delete advice",
);
} else {
assert.ok(!wire.reason?.includes("(win32:"), "non-win32 runner stays un-hinted");
assert.ok(
degradeLine.includes("; delete the file to reset the index"),
"a genuine open failure keeps the delete advice off-hint platforms",
);
}
} finally {
setLogCapture(null);
_resetStorageRuntimeStatusForTest();
restoreEnv(snap);
rmrf(base);
}
});

// #4614: the field-reported READONLY signature — an EXISTING index.db that
// opens but refuses the first write. Deterministic cross-platform repro: a
// 0444 regular file (root bypasses permission bits, so skip under root;
// chmod is inert on win32, so skip there — the win32 branch of the hint is
// unit-pinned above with a platform stub instead).
test("#4614 wiring: an unwritable existing index.db degrades carrying the readonly signature", () => {
if (process.platform === "win32") return;
if (typeof process.getuid === "function" && process.getuid() === 0) return;
const base = mkdtempSync(path.join(tmpdir(), "readonly-hint-"));
const sessions = path.join(base, "sessions");
mkdirSync(sessions, { recursive: true });
writeFileSync(path.join(sessions, "index.db"), "");
chmodSync(path.join(sessions, "index.db"), 0o444);
const snap = isolateEnv(base, sessions);
const logs: string[] = [];
setLogCapture((level, msg) => logs.push(`${level} ${msg}`));
try {
_resetStorageRuntimeStatusForTest();
new SessionStore({ dir: sessions, enabled: true });
const wire = storageStatusWire();
assert.equal(wire.degraded, true, "the sidecar must be degraded");
assert.equal(wire.effective, "legacy", "legacy JSON keeps serving");
assert.ok(
wire.reason?.includes("attempt to write a readonly database"),
`reason carries the readonly signature (got: ${wire.reason})`,
);
const degradeLine = logs.find((l) => l.includes("unified sidecar failed to open"));
assert.ok(degradeLine, "the degrade warn was logged");
assert.ok(!degradeLine.includes("(win32:"), "non-win32 stays un-hinted");
} finally {
setLogCapture(null);
_resetStorageRuntimeStatusForTest();
restoreEnv(snap);
rmrf(base);
}
});

// #3617 negative direction on ALL runners: when the OS-level hint fires
// (stubbed win32 + CANTOPEN), the degrade line must drop the delete advice —
// including for READONLY-class refusals where deleting destroys the only copy
// of recent session state in unified-full mode.
test("wiring: an OS-level refusal drops the delete advice from the degrade line", () => {
const base = mkdtempSync(path.join(tmpdir(), "readonly-hint-"));
const sessions = path.join(base, "sessions");
mkdirSync(sessions, { recursive: true });
mkdirSync(path.join(sessions, "index.db"));
const snap = isolateEnv(base, sessions);
const logs: string[] = [];
setLogCapture((level, msg) => logs.push(`${level} ${msg}`));
const restore = stubPlatform("win32");
try {
_resetStorageRuntimeStatusForTest();
new SessionStore({ dir: sessions, enabled: true });
const wire = storageStatusWire();
assert.ok(wire.reason?.includes("(win32:"), "stubbed win32 carries the OS-level hint in the diagnostic");
const degradeLine = logs.find((l) => l.includes("unified sidecar failed to open"));
assert.ok(degradeLine, "the degrade warn was logged");
assert.ok(degradeLine.includes("(win32:"), "the warn line carries the hint");
assert.ok(
!degradeLine.includes("delete the file to reset the index"),
"an OS-level refusal must not carry the delete advice",
);
} finally {
restore();
setLogCapture(null);
_resetStorageRuntimeStatusForTest();
restoreEnv(snap);
rmrf(base);
Expand Down
Loading