Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -665,6 +665,8 @@ bili storage gc [--days N] [--vacuum full|incremental] [--dry-run]

`archive`/`restore`/`gc` require the unified store (`BILI_STORAGE_UNIFIED=shadow|full` or `storage.unified` in the config file) — legacy-only installs refuse with a hint. `--days` defaults to `storage.archiveAfterDays` (365). No scheduler: these run on demand. All commands accept `--json`.

**Full vacuum is an offline operation on large databases (#4612).** `--vacuum full` (the default) rewrites the entire index.db under an exclusive write lock — O(db size), minutes to tens of minutes on multi-GB databases, with no intermediate progress (SQLite's VACUUM is atomic; it cannot be split). Running it while a proxy is serving from the same sessions dir starves that proxy's writes until its storage layer degrades, so `storage gc --vacuum full` **refuses to start when another live process holds the database** (`index.db.lock` pid discipline, #4175): the CLI exits 1 with an actionable message, the panel endpoint returns HTTP 409. Stop the other instance first (or wait for it to exit) and retry; for online-safe shrink use `--vacuum incremental`, which reclaims freed pages in short transactions without taking the exclusive lock. The zero-ref delete sweep itself is batched with per-batch progress output either way.

## Status

Early. Protocol handling and compression work against mock tests (500+ passing). Real-model integration testing is the next milestone. Expect rough edges.
Expand Down
2 changes: 2 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -483,6 +483,8 @@ bili storage gc [--days N] [--vacuum full|incremental] [--dry-run]

`archive`/`restore`/`gc` 需要统一存储(`BILI_STORAGE_UNIFIED=shadow|full` 或配置文件里的 `storage.unified`)—— legacy-only 安装会拒绝并给出提示。`--days` 默认取 `storage.archiveAfterDays`(365)。没有调度器:全部按需运行。所有命令都支持 `--json`。

**大库上 full vacuum 是离线操作(#4612)。** `--vacuum full`(默认)会在独占写锁下重写整个 index.db —— O(库大小),多 GB 的库要几分钟到几十分钟,且没有中间进度(SQLite 的 VACUUM 是原子操作,拆不开)。在有代理正从同一会话目录服务时跑它,会把那个代理的写入饿死到其存储层降级为止,所以 `storage gc --vacuum full` **在另一个存活进程持有该数据库时会拒绝启动**(`index.db.lock` pid 纪律,#4175):CLI 以退出码 1 给出可操作提示,面板端点返回 HTTP 409。先停掉另一个实例(或等它退出)再重试;要在线安全地缩库用 `--vacuum incremental`,它以短事务回收空闲页、不拿独占锁。零引用删除清扫无论哪种模式都分批执行并逐批打印进度。

## 状态

早期。协议处理和压缩已通过 mock 测试(500+ 项通过)。真实模型集成测试是下一里程碑。预期会有粗糙的地方。
Expand Down
3 changes: 3 additions & 0 deletions scripts/live-files.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@ export const LIVE_FILES = [
// #4434 (split-brain surface): spawns bili CLI children against a throwaway
// origin; own probe server is listen(0) so only the zone band matters
"tests/issue-split-brain-surface.test.ts",
// #4612: spawns an idle node child as the "other live process" fixture;
// binds no ports, mkdtemp-isolated per test
"tests/issue4612-gc-online-guard.test.ts",
];

/** Band width per live file: the zone ladder drifts +1 per EADDRINUSE retry;
Expand Down
2 changes: 1 addition & 1 deletion src/agent/client-build-note.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ import path from "node:path";
import { isVersionNewer } from "../update.js";
import { logHostEvent } from "./host-log.js";

export type ClientBuildCheck = {
type ClientBuildCheck = {
/** Short human label used in the note text ("dsh", "pi", ...). */
label: string;
/** The client's npm id, as shown in the note (@deepseek-ai/dsh, ...). */
Expand Down
12 changes: 12 additions & 0 deletions src/persist.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1764,6 +1764,18 @@ export class SessionStore {
}
}

/** #4612: live foreign writer on the unified db (#4175 lock discipline) —
* the gc full-vacuum preflight signal for the CLI/HTTP surfaces. Null when
* the unified store is unavailable or nobody else is provably writing. */
unifiedActiveWriter(): number | null {
if (!this.shadow || this.shadow.stats.disabled) return null;
try {
return this.shadow.activeForeignWriter();
} catch {
return null; // sidecar degraded mid-call: no evidence of a writer
}
}

/** #3901/#3463: restore an archived session from its export file. */
unifiedRestoreArchived(sessionId: string, archiveDir?: string): { ok: boolean; reason?: string } | null {
if (!this.shadow || this.shadow.stats.disabled) return null;
Expand Down
11 changes: 11 additions & 0 deletions src/server/admin.ts
Original file line number Diff line number Diff line change
Expand Up @@ -540,6 +540,17 @@ export async function handleAdminRoute(req: http.IncomingMessage, res: http.Serv
}
const qVacuum = (() => { try { return new URL(req.url!, "http://localhost").searchParams.get("vacuum"); } catch { return null; } })();
const vacuum = qVacuum === "full" || qVacuum === "incremental" ? qVacuum : parsed.vacuum === "full" ? "full" : parsed.vacuum === "incremental" ? "incremental" : undefined;
// #4612: full VACUUM refuses to start while another live process
// writes the same db (exclusive write lock across the whole
// O(db-size) rewrite) — same preflight as the CLI lane.
if (vacuum === "full") {
const writer = store.unifiedActiveWriter();
if (writer !== null) {
res.writeHead(409, { "content-type": "application/json" });
res.end(JSON.stringify({ ok: false, error: `another process (pid ${writer}) is actively writing index.db — full VACUUM holds the exclusive write lock across the whole rewrite; stop the other process first or use ?vacuum=incremental`, writerPid: writer }));
return;
}
}
const result = await store.unifiedGc({
archiveAfterDays: typeof parsed.archiveAfterDays === "number" && Number.isFinite(parsed.archiveAfterDays) && parsed.archiveAfterDays >= 0 ? parsed.archiveAfterDays : undefined,
vacuum,
Expand Down
15 changes: 15 additions & 0 deletions src/storage-maintenance.ts
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,21 @@ async function runGc(store: SessionStore, days: number | undefined, vacuum: "ful
if (json) return { code: 0, text: `${JSON.stringify(data, null, 2)}\n` };
return { code: 0, text: `${text}\ndry-run: nothing executed\n` };
}
// #4612: full VACUUM is an O(db-size) atomic rewrite under the exclusive
// write lock with no intermediate progress — running it while another live
// process (e.g. a running proxy) writes the same db starves that process
// until its storage layer degrades. Refuse with an actionable message;
// incremental / no-vacuum sweeps are short transactions and stay allowed.
if (vacuumMode === "full") {
const writer = store.unifiedActiveWriter();
if (writer !== null) {
data.refused = true;
data.writerPid = writer;
const reason = `refused: another process (pid ${writer}) is actively writing index.db — 'storage gc --vacuum full' rewrites the whole database under an exclusive write lock (minutes on large dbs, no intermediate progress) and would starve the other instance's writes until its storage layer degrades. Stop the other process (or wait for it to exit) and retry, or use --vacuum incremental for an online-safe shrink.`;
if (json) return { code: 1, text: `${JSON.stringify(data, null, 2)}\n` };
return fail(`${text}\n${reason}`, json, data);
}
}
const dbBefore = indexDbSize(store.dir);
// Same facade call the HTTP endpoint makes (admin.ts POST /__bili/storage/gc)
// — results are therefore comparable lane-for-lane.
Expand Down
8 changes: 8 additions & 0 deletions src/storage/shadow.ts
Original file line number Diff line number Diff line change
Expand Up @@ -725,6 +725,14 @@ export class UnifiedShadow {
return this.sync.gcPlan(archiveAfterDays, now);
}

/** #4612: live foreign writer on the unified db (#4175 lock discipline) —
* the gc full-vacuum preflight signal. Null when unavailable or nobody
* else is provably writing. */
activeForeignWriter(): number | null {
if (this.sync === null) return null;
return this.sync.activeForeignWriter();
}

rawMessages(sessionId: string): { seq: number; content: string }[] {
if (this.sync === null) return [];
this.assertLive();
Expand Down
105 changes: 100 additions & 5 deletions src/storage/store.ts
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,24 @@ export class StorageIntegrityError extends Error {
}
}

/** #4612: zero-ref sweep batching — minimum rows per commit transaction, and a
* cap on how many progress lines one sweep may emit (the batch grows with the
* table, so a 1M-row sweep logs ~500 lines instead of 1000). */
const GC_SWEEP_BATCH_MIN_ROWS = 1000;
const GC_SWEEP_MAX_PROGRESS_LINES = 500;

function formatDbSize(bytes: number): string {
if (bytes < 0) return "?";
const units = ["B", "KiB", "MiB", "GiB"];
let v = bytes;
let i = 0;
while (v >= 1024 && i < units.length - 1) {
v /= 1024;
i++;
}
return i === 0 ? `${v} B` : `${v.toFixed(1)} ${units[i]}`;
}

/** #4175: true when a pid belongs to a live process (EPERM = alive, foreign user). */
function processAlive(pid: number): boolean {
try {
Expand All @@ -330,13 +348,16 @@ export class UnifiedStore {
/** Where the engine came from (#2833): builtin / host / self-downloaded — logged at enable so field diagnosis can tell which acquisition path a lane ran on. */
readonly engineSource: SqliteEngineSource;
private readonly db: SqliteDatabase;
/** #4612: main database file path (VACUUM sizing logs). */
private readonly path: string;
/** #3250: successful-ingest counter driving the WAL_CHECKPOINT_EVERY_INGESTS cadence. */
private ingestCount = 0;
/** #4175: the live foreign pid found holding <db>.lock at open, if any. */
private readonly foreignLockHolder: { pid: number } | null;
private readonly lockPath: string | null;

private constructor(db: SqliteDatabase, engine: SqliteEngineName, engineSource: SqliteEngineSource, lockPath: string | null, foreignLockHolder: { pid: number } | null) {
private constructor(path: string, db: SqliteDatabase, engine: SqliteEngineName, engineSource: SqliteEngineSource, lockPath: string | null, foreignLockHolder: { pid: number } | null) {
this.path = path;
this.db = db;
this.engine = engine;
this.engineSource = engineSource;
Expand Down Expand Up @@ -438,7 +459,7 @@ export class UnifiedStore {
} catch (err) {
loggerLog("warn", `[storage] WAL checkpoint on open failed for ${path}: ${String(err)}`);
}
return new UnifiedStore(db, used, source, lockPath, foreignLockHolder);
return new UnifiedStore(path, db, used, source, lockPath, foreignLockHolder);
}

close(): void {
Expand Down Expand Up @@ -469,6 +490,40 @@ export class UnifiedStore {
return this.foreignLockHolder;
}

/** #4612: a live pid besides ours currently writing this db — the union of
* two evidence sources: (1) the foreign holder captured at open, re-validated
* for liveness (it may have exited since); (2) a fresh read of <db>.lock
* pointing at another live pid — every opener overwrites the lockfile with
* its own pid on open, so this catches an instance that started AFTER us.
* Null when no live foreign writer is provably present; an unreadable
* lockfile means "no evidence" (fail-open) — callers need proof of
* liveness to refuse, not certainty of absence. */
activeForeignWriter(): number | null {
if (this.foreignLockHolder !== null && processAlive(this.foreignLockHolder.pid)) {
return this.foreignLockHolder.pid;
}
if (this.lockPath !== null) {
try {
if (existsSync(this.lockPath)) {
const parsed = JSON.parse(readFileSync(this.lockPath, "utf8")) as { pid?: unknown };
const pid = typeof parsed.pid === "number" ? parsed.pid : null;
if (pid !== null && pid !== process.pid && processAlive(pid)) return pid;
}
} catch {
/* unreadable lockfile: no evidence of a writer */
}
}
return null;
}

private dbFileSize(): number {
try {
return statSync(this.path).size;
} catch {
return -1;
}
}

/**
* #4175: online consistent snapshot via SQLite's native VACUUM INTO —
* safe while WAL writers are active (it reads a consistent snapshot and
Expand Down Expand Up @@ -1685,6 +1740,19 @@ export class UnifiedStore {
* pages incrementally; a one-time VACUUM reclaims the initial bloat of
* databases created before that pragma). */
gc(opts: { archiveAfterDays: number; archiveDir: string; now?: number; vacuum?: "full" | "incremental" }): { archived: number; contentRowsDeleted: number } {
// #4612: full VACUUM is an O(db-size) atomic rewrite under the exclusive
// write lock with no intermediate progress. Running it against a db that
// another live process is writing starves that process until its storage
// layer degrades (field report: shadow 3-strike during a 14-min run on a
// 14 GB db). Refuse up front with an actionable error instead of silently
// contending. Incremental / no-vacuum sweeps are short transactions — the
// existing busy_timeout handling covers them.
if (opts.vacuum === "full") {
const writer = this.activeForeignWriter();
if (writer !== null) {
throw new Error(`[#4612] gc full VACUUM refused: another process (pid ${writer}) is actively writing this database — the rewrite holds the exclusive write lock across the whole pass and would starve the peer's writes until its storage layer degrades; stop the other process (or wait for it to exit) and retry, or use vacuum:"incremental"`);
}
}
const now = opts.now ?? Date.now();
let archived = 0;
let contentRowsDeleted = 0;
Expand All @@ -1699,14 +1767,41 @@ export class UnifiedStore {
}
const zero = this.db.prepare("SELECT COUNT(*) AS n FROM content WHERE refcount <= 0").get() as { n: number };
if (zero.n > 0) {
this.db.prepare("DELETE FROM content WHERE refcount <= 0").run();
contentRowsDeleted += zero.n;
// #4612: batched sweep. One unbounded DELETE held the exclusive write
// lock across the entire scan (field: 82k rows / 11.3 GiB, 14 minutes
// of silence) with no progress visible. Per-batch commits release the
// lock between batches and give the operator something to watch. Rows
// are garbage by definition (refcount <= 0), so partial progress is
// safe: whatever survives a crash is collected by the next sweep. The
// batch grows with the table so total log lines stay bounded.
const batch = Math.max(GC_SWEEP_BATCH_MIN_ROWS, Math.ceil(zero.n / GC_SWEEP_MAX_PROGRESS_LINES));
// SQLite DELETE takes no LIMIT clause: bound the batch through a
// rowid subselect instead.
const del = this.db.prepare("DELETE FROM content WHERE rowid IN (SELECT rowid FROM content WHERE refcount <= 0 ORDER BY rowid LIMIT ?)");
let deleted = 0;
for (;;) {
const r = del.run(batch);
const n = Number(r.changes);
if (n === 0) break;
deleted += n;
loggerLog("info", `[#4612] gc zero-ref sweep: ${deleted}/${zero.n} rows (${Math.round((deleted / zero.n) * 100)}%)`);
}
contentRowsDeleted += deleted;
}
if (opts.vacuum === "full") {
loggerLog("info", "[#3901] gc: running full VACUUM (this can take a while on large databases)");
const sizeBefore = this.dbFileSize();
let walPart = "";
try {
walPart = `, ${formatDbSize(statSync(`${this.path}-wal`).size)} WAL`;
} catch {
/* no WAL file present */
}
loggerLog("info", `[#4612] gc: full VACUUM starting — ${formatDbSize(sizeBefore)} main file${walPart}; atomic rewrite holding the exclusive write lock until done (no intermediate progress)`);
const startedAt = Date.now();
// #4390: delegate to vacuum() — a bare VACUUM leaves the compacted
// image in the WAL; only the TRUNCATE checkpoint shrinks the files.
this.vacuum("full");
loggerLog("info", `[#4612] gc: full VACUUM finished in ${((Date.now() - startedAt) / 1000).toFixed(1)} s — ${formatDbSize(sizeBefore)} -> ${formatDbSize(this.dbFileSize())}`);
} else if (opts.vacuum === "incremental") {
this.vacuum("incremental");
}
Expand Down
Loading
Loading