diff --git a/CHANGELOG.md b/CHANGELOG.md index d18a29b629..716a49def1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag ### 存储收尾九项一车(#3901,修复 #3463/#3762/#3841/#3798/#3633;#3723 落地) -- **GC 与归档(#3463)**:新 `storage.archiveAfterDays`(环境变量 `BILI_STORAGE_ARCHIVE_AFTER_DAYS`,缺省 365,0=关)——闲置超期的会话归档(完整导出到 sessions 目录下 `archive/` + 墓碑行保标签 + 释放全部内容/媒体引用),`POST /__bili/storage/gc`(可带 `vacuum`)手动触发,`POST /__bili/storage/restore` 恢复;`auto_vacuum=INCREMENTAL` 常开(存量库首次需 `?vacuum=full` 收缩)。 +- **GC 与归档(#3463)**:新 `storage.archiveAfterDays`(环境变量 `BILI_STORAGE_ARCHIVE_AFTER_DAYS`,缺省 365,0=关)——闲置超期的会话归档(单文件追加式归档日志 `archive/archive.jsonl`,每会话一行——不撒小文件;墓碑行保标签 + 释放全部内容/媒体引用),`POST /__bili/storage/gc`(可带 `vacuum`)手动触发,`POST /__bili/storage/restore` 恢复;`auto_vacuum=INCREMENTAL` 常开(存量库首次需 `?vacuum=full` 收缩)。 - **reap 台账(#3841)**:schema v6 `sessions.orphans_json`——收割形状(state.blocks 失块、blockContents 留文)现在定向记台账:被收割块行删除(引用计数释放),孤儿内容并入视图但不复活 state.blocks;重启不再复活已收割块、不再污染信封。 - **媒体 CAS(#3762)**:schema v7 `message_media`——图片 base64 以内容哈希进 CAS、消息行只存 `{"$media":hash}` 引用(读回逐字节水合),同图多会话只存一份;删除/归档时引用计数释放。 - **增量指针(#3798)**:`inc:canonical-ptrs` 逐行指针替代 base-seq 计数切片——resync 共享前缀的行复用不再丢历史;resync 时点先失效旧指针(响应前持久化),崩溃窗口退回旧切片语义(永不供错分支);turn 锁重排:响应阶段 409 turn-in-flight 照旧、ack 之后的回写链被下一轮 await(ack 永不超前行落盘)。 diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 84614d5c52..f0cda90f8d 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -178,7 +178,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | -| `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - full export to the archive dir under the sessions directory, a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). Env wins over the file tier. | +| `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). Env wins over the file tier. | | `sessions` | object | {} (all defaults) | — | Session-table size cap and opt-in garbage collection. | | `sessions.max` | number | 256 | BILI_MAX_SESSIONS | In-memory session table cap (LRU eviction of idle sessions). | | `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | Bounded startup (#2803/#3050): max most-recently-saved sessions hydrated at boot before listening; older state reloads on demand. Default 0 (no eager warm set — first touch pays one on-demand load; set 8 to restore the eager behavior). Capped by sessions.max. | @@ -333,7 +333,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | Key | Type | Default | Env | Description | |-----|------|---------|-----|-------------| | `BILI_DSH_AUTHORED_COMPACTION` | boolean | true (on, #3901 flip) | — | Let bili ANSWER dsh compaction summarize calls locally with the folded snapshot instead of refusing (#3733): identity header + kernel-rendered block summaries, zero egress; replay recognized via the #2432 rebase. | -| `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | integer | 365 | — | GC age sweep window in days (#3463): sessions idle longer than this are archived (full export to the archive dir + tombstone row + content/media refs released). 0 disables the age sweep; the manual entry POST /__bili/storage/gc always runs. Config-file key storage.archiveAfterDays is the persistent form. | +| `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | integer | 365 | — | GC age sweep window in days (#3463): sessions idle longer than this are archived (one JSON line appended to the single archive log (archive/archive.jsonl) + tombstone row + content/media refs released). 0 disables the age sweep; the manual entry POST /__bili/storage/gc always runs. Config-file key storage.archiveAfterDays is the persistent form. | | `BILI_SNAPSHOT_SHADOW` | boolean | true (on) | — | Snapshot-render shadow observability (#3756 follow-up): compute the outbound snapshot view on every armed turn WITHOUT using it, classify the diff against the legacy outbound, and surface per-session counters in the web panel + the snapshot-shadow diagnostics topic. 0/false/off disables; 1/true/on enables; otherwise the config-file snapshot.shadow boolean applies (default on). The render lane itself stays rollout-gated (BILI_ROLLOUT_SNAPSHOT_RENDER), so the shadow lane never changes outbound bytes - it is the dry-run evidence that flips the rollout later. | | `BILI_CONFIG_FILE` | string | unset (XDG config path) | — | Override the config file path (path relocation). | | `BILI_SESSIONS_DIR` | string | unset (XDG state path) | — | Where session records are stored (path relocation). | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index e60ec8f26a..8115d9fe12 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -178,7 +178,7 @@ | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | -| `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 完整导出到会话目录下的 archive/,墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。环境变量优先于配置文件层。 | +| `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。环境变量优先于配置文件层。 | | `sessions` | object | {} (all defaults) | — | 会话表容量上限与可选垃圾回收。 | | `sessions.max` | number | 256 | BILI_MAX_SESSIONS | 内存会话表上限(空闲会话 LRU 逐出)。 | | `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | 有界启动(#2803/#3050):启动监听前预热加载的最近保存会话数上限,更早的状态按需重新加载。默认 0(不预热——首次访问按需加载一次;设 8 恢复预热行为)。受 sessions.max 封顶。 | @@ -333,7 +333,7 @@ | Key | Type | Default | Env | Description | |-----|------|---------|-----|-------------| | `BILI_DSH_AUTHORED_COMPACTION` | boolean | true (on, #3901 flip) | — | 让 bili 本地应答 dsh 压缩摘要请求(返回折叠快照而非拒绝,#3733):身份头 + 内核渲染格式的块摘要,零出口流量;重放经 #2432 再基线识别。 | -| `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | integer | 365 | — | GC 年龄清扫窗口(天,#3463):闲置超过该时长的会话被归档(完整导出到归档目录 + 墓碑行保留标签 + 释放内容/媒体引用)。0 关闭年龄清扫;手动入口 POST /__bili/storage/gc 始终可用。配置文件键 storage.archiveAfterDays 为持久形式。 | +| `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | integer | 365 | — | GC 年龄清扫窗口(天,#3463):闲置超过该时长的会话被归档(单文件归档日志 archive/archive.jsonl 追加一行 + 墓碑行保留标签 + 释放内容/媒体引用)。0 关闭年龄清扫;手动入口 POST /__bili/storage/gc 始终可用。配置文件键 storage.archiveAfterDays 为持久形式。 | | `BILI_SNAPSHOT_SHADOW` | boolean | true (on) | — | 快照渲染影子对账(#3756 后续):每轮已折叠会话都用引擎现算一次出站快照视图但不使用,与 legacy 实际出站比对分类,逐会话计数进 web 面板与 snapshot-shadow 诊断主题。0/false/off 关闭;1/true/on 开启;否则读配置文件 snapshot.shadow 布尔(默认开)。渲染车道本身仍受灰度开关控制(BILI_ROLLOUT_SNAPSHOT_RENDER),影子车道绝不改变出站字节 —— 它是后续翻转灰度的干跑证据。 | | `BILI_CONFIG_FILE` | string | unset (XDG config path) | — | 覆盖配置文件路径(路径重定位)。 | | `BILI_SESSIONS_DIR` | string | unset (XDG state path) | — | 会话记录的存放目录(路径重定位)。 | diff --git a/src/storage/store.ts b/src/storage/store.ts index 92537a6658..4a5c568715 100644 --- a/src/storage/store.ts +++ b/src/storage/store.ts @@ -17,7 +17,7 @@ */ import { createHash } from "node:crypto"; -import { mkdirSync, writeFileSync, readFileSync, existsSync } from "node:fs"; +import { mkdirSync, writeFileSync, appendFileSync, readFileSync, existsSync } from "node:fs"; import { dirname, join } from "node:path"; import { log as loggerLog } from "../logger.js"; import { openSqliteDatabase, type SqliteDatabase, type SqliteEngineName, type SqliteEngineSource } from "./driver.js"; @@ -929,20 +929,31 @@ export class UnifiedStore { }); } + /** #3901/#3463: single append-only archive log — one JSON line per + * archived session, not one tiny file per session. */ + static readonly ARCHIVE_LOG_NAME = "archive.jsonl"; + /** #3901/#3463: archive one session. The full row bodies (messages, * blocks, refs, dead refs, ccr, session_ext, unpinned checkpoints) are - * exported verbatim — content bodies included — to - * `/.json`, then swept with the deleteSession - * refcount discipline. Unlike deleteSession the sessions row survives - * as a tombstone (status 'archived', label intact — the web list keeps + * exported verbatim — content bodies included — as ONE JSON line + * appended to `/archive.jsonl` (the single append-only log; + * a crash mid-append leaves at most one truncated line which the + * restore reader skips), then swept with the deleteSession refcount + * discipline. Unlike deleteSession the sessions row survives as a + * tombstone (status 'archived', label intact — the web list keeps * showing it) and the conversations identity binding survives (identity * is forever; a restore must land on the same session id). Pinned * checkpoints keep their rows + refs. Idempotent: an already-archived - * session is a no-op returning the existing export path. */ + * session is a no-op returning the log path. */ + archiveSession(sessionId: string, opts: { archiveDir: string }): { ok: boolean; exportPath?: string; reason?: string; contentRowsDeleted?: number } { const row = this.getSessionRow(sessionId); if (!row) return { ok: false, reason: "not-found" }; - if (row.status === "archived") return { ok: true, exportPath: join(opts.archiveDir, `${sessionId}.json`) }; + // #3901 review: sessions are small and numerous — a per-session file + // per archive would spray tiny files across the archive dir. One + // append-only JSONL holds every archived session, one line each. + const exportPath = join(opts.archiveDir, UnifiedStore.ARCHIVE_LOG_NAME); + if (row.status === "archived") return { ok: true, exportPath }; // --- read everything the session owns (before any mutation) --- const messages = (this.db.prepare("SELECT seq, msg_id, ref, ref_num, role, kind, content_hash FROM messages WHERE session_id = ? ORDER BY seq").all(sessionId) as Record[]); const mediaMaps = (this.db.prepare("SELECT seq, media_hash FROM message_media WHERE session_id = ?").all(sessionId) as Record[]); @@ -964,9 +975,11 @@ export class UnifiedStore { const c = this.db.prepare("SELECT bytes FROM content WHERE hash = ?").get(hash) as { bytes: Buffer | Uint8Array } | undefined; if (c) content[hash as string] = Buffer.from(c.bytes).toString("utf8"); } - const exportPath = join(opts.archiveDir, `${sessionId}.json`); mkdirSync(opts.archiveDir, { recursive: true }); - writeFileSync(exportPath, JSON.stringify({ version: 1, archivedAt: Date.now(), session: row, content, messages, mediaMaps, blocks, refs: refsRows, deadRefs, ccrEntries: ccr, sessionExt: ext, checkpoints })); + // One JSONL line per archived session: append-only (cheap, crash = + // at worst one truncated line which the reader skips), restore + // rewrites the file without the restored session's lines. + appendFileSync(exportPath, JSON.stringify({ version: 1, sessionId, archivedAt: Date.now(), session: row, content, messages, mediaMaps, blocks, refs: refsRows, deadRefs, ccrEntries: ccr, sessionExt: ext, checkpoints }) + "\n"); // --- sweep the row bodies (refcount discipline; pinned checkpoints survive) --- let contentRowsDeleted = 0; this.db.transaction(() => { @@ -999,7 +1012,7 @@ export class UnifiedStore { * re-claims ON CONFLICT DO NOTHING (first claimant wins; the binding * never left, so this is normally a no-op). */ restoreArchived(sessionId: string, opts: { archiveDir: string }): { ok: boolean; reason?: string } { - const exportPath = join(opts.archiveDir, `${sessionId}.json`); + const exportPath = join(opts.archiveDir, UnifiedStore.ARCHIVE_LOG_NAME); if (!existsSync(exportPath)) return { ok: false, reason: "export-missing" }; let dump: { session: ReturnType; @@ -1012,8 +1025,29 @@ export class UnifiedStore { sessionExt: Record[]; checkpoints: Record[]; }; + // Scan the JSONL for the session's line (skip malformed/truncated + // trailing lines — an archive append interrupted by a crash must not + // block restoring OTHER sessions). + let lineNo = -1; + let rawLines: string[] = []; + try { + rawLines = readFileSync(exportPath, "utf8").split("\n"); + for (let i = 0; i < rawLines.length; i++) { + if (!rawLines[i].trim()) continue; + let candidate: { sessionId?: string }; + try { + candidate = JSON.parse(rawLines[i]) as { sessionId?: string }; + } catch { + continue; + } + if (candidate.sessionId === sessionId) { lineNo = i; break; } + } + } catch { + return { ok: false, reason: "export-corrupt" }; + } + if (lineNo === -1) return { ok: false, reason: "export-missing" }; try { - dump = JSON.parse(readFileSync(exportPath, "utf8")) as typeof dump; + dump = JSON.parse(rawLines[lineNo]) as typeof dump; } catch { return { ok: false, reason: "export-corrupt" }; } @@ -1049,7 +1083,20 @@ export class UnifiedStore { sess.lastIngestedSeq ?? null, sess.orphansJson ?? null, sessionId, ); }); - loggerLog("info", `[#3901] restored archived session ${sessionId} from ${exportPath}`); + // Drop the restored session's line(s) from the JSONL (a same-session + // archive can theoretically appear more than once across reinstalls; + // rewrite removes them all). Other sessions' lines are untouched. + const kept: string[] = []; + for (const l of rawLines) { + if (!l.trim()) continue; + try { + const head = JSON.parse(l) as { sessionId?: string }; + if (head.sessionId === sessionId) continue; + } catch { continue; } + kept.push(l); + } + writeFileSync(exportPath, kept.length > 0 ? kept.join("\n") + "\n" : ""); + loggerLog("info", `[#3901] restored archived session ${sessionId} from ${exportPath} (line removed from the archive log)`); return { ok: true }; } diff --git a/tests/storage-archive-gc.test.ts b/tests/storage-archive-gc.test.ts index 5257c4fe03..3ce3715c3b 100644 --- a/tests/storage-archive-gc.test.ts +++ b/tests/storage-archive-gc.test.ts @@ -60,7 +60,8 @@ test("archiveSession: tombstone + CAS release; shared rows survive; restore re-i store.ingestSession(planIngest(mkSession("sess-A", [shared, onlyA]))); store.ingestSession(planIngest(mkSession("sess-B", [shared]))); - const archived = store.archiveSession("sess-A", { archiveDir: path.join(dir, "archive") }); + const archiveDir = path.join(dir, "archive"); + const archived = store.archiveSession("sess-A", { archiveDir }); assert.ok(archived.ok); assert.ok(archived.exportPath, "archive returns the export path"); assert.ok(fs.existsSync(archived.exportPath), "archive export file written"); @@ -85,6 +86,62 @@ test("archiveSession: tombstone + CAS release; shared rows survive; restore re-i } }); +test("archive log: one append-only JSONL, one line per session, no per-session files (#3901 review)", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "storage-archive-log-")); + try { + const store = UnifiedStore.open(path.join(dir, "index.db")); + store.ingestSession(planIngest(mkSession("sess-A", [mkMsg("m-a", "body a ".repeat(50))]))); + store.ingestSession(planIngest(mkSession("sess-B", [mkMsg("m-b", "body b ".repeat(50))]))); + const archiveDir = path.join(dir, "archive"); + const a = store.archiveSession("sess-A", { archiveDir }); + const b = store.archiveSession("sess-B", { archiveDir }); + assert.ok(a.ok && b.ok); + assert.equal(a.exportPath, b.exportPath, "all archives land in the single log"); + assert.equal(fs.readdirSync(archiveDir).length, 1, "no per-session files sprayed"); + const lines = fs.readFileSync(a.exportPath!, "utf8").split("\n").filter((l) => l.trim()); + assert.equal(lines.length, 2, "one JSON line per archived session"); + assert.ok(JSON.parse(lines[0]).sessionId === "sess-A" && JSON.parse(lines[1]).sessionId === "sess-B", "lines carry their session ids"); + // restore removes exactly the restored session's line + const restored = store.restoreArchived("sess-A", { archiveDir }); + assert.ok(restored.ok); + const after = fs.readFileSync(a.exportPath!, "utf8").split("\n").filter((l) => l.trim()); + assert.equal(after.length, 1, "restore rewrites the log without the restored line"); + assert.equal(JSON.parse(after[0]).sessionId, "sess-B", "other sessions' lines survive the rewrite"); + // re-archive lands a fresh line again (append semantics) + assert.ok(store.archiveSession("sess-A", { archiveDir }).ok); + assert.equal(fs.readFileSync(a.exportPath!, "utf8").split("\n").filter((l) => l.trim()).length, 2); + } finally { + try { fs.rmSync(dir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); } catch { /* Windows stray handles */ } + } +}); + +test("archive log: a crashed append's truncated line cannot block other restores (#3901 review)", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "storage-archive-trunc-")); + try { + const store = UnifiedStore.open(path.join(dir, "index.db")); + store.ingestSession(planIngest(mkSession("sess-A", [mkMsg("m-a", "body a ".repeat(50))]))); + store.ingestSession(planIngest(mkSession("sess-B", [mkMsg("m-b", "body b ".repeat(50))]))); + const archiveDir = path.join(dir, "archive"); + assert.ok(store.archiveSession("sess-B", { archiveDir }).ok); + // Simulate a crash mid-append for sess-A: a partial JSON line after sess-B's. + const logPath = path.join(archiveDir, UnifiedStore.ARCHIVE_LOG_NAME); + fs.appendFileSync(logPath, '{"version":1,"sessionId":"sess-A","arc'); + // Restoring sess-B still works despite the malformed trailing line… + const restored = store.restoreArchived("sess-B", { archiveDir }); + assert.ok(restored.ok, "malformed trailing line does not block other restores"); + assert.equal(store.getMessages("sess-B").length, 1, "restored session has its message back"); + // …and the rewrite drops the truncated line (its DB rows were never swept — + // the append precedes the sweep — so nothing restorable is lost). + const lines = fs.readFileSync(logPath, "utf8").split("\n").filter((l) => l.trim()); + assert.equal(lines.length, 0, "truncated line dropped by the restore rewrite"); + // Restoring the crashed session itself reports honestly: its data still lives in the DB, + // and the next gc pass re-archives it with a fresh complete line. + assert.deepEqual(store.restoreArchived("sess-A", { archiveDir }), { ok: false, reason: "export-missing" }); + } finally { + try { fs.rmSync(dir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); } catch { /* Windows stray handles */ } + } +}); + test("gc(): age sweep archives only past-retention sessions; dormant shared rows collected", () => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), "storage-gc-")); try { diff --git a/website/config-reference/env-only.yaml b/website/config-reference/env-only.yaml index d1bcbd5468..c63a37593b 100644 --- a/website/config-reference/env-only.yaml +++ b/website/config-reference/env-only.yaml @@ -16,8 +16,8 @@ entries: kind: env-only audience: user group: envonly - desc_en: "GC age sweep window in days (#3463): sessions idle longer than this are archived (full export to the archive dir + tombstone row + content/media refs released). 0 disables the age sweep; the manual entry POST /__bili/storage/gc always runs. Config-file key storage.archiveAfterDays is the persistent form." - desc_zh: "GC 年龄清扫窗口(天,#3463):闲置超过该时长的会话被归档(完整导出到归档目录 + 墓碑行保留标签 + 释放内容/媒体引用)。0 关闭年龄清扫;手动入口 POST /__bili/storage/gc 始终可用。配置文件键 storage.archiveAfterDays 为持久形式。" + desc_en: "GC age sweep window in days (#3463): sessions idle longer than this are archived (one JSON line appended to the single archive log (archive/archive.jsonl) + tombstone row + content/media refs released). 0 disables the age sweep; the manual entry POST /__bili/storage/gc always runs. Config-file key storage.archiveAfterDays is the persistent form." + desc_zh: "GC 年龄清扫窗口(天,#3463):闲置超过该时长的会话被归档(单文件归档日志 archive/archive.jsonl 追加一行 + 墓碑行保留标签 + 释放内容/媒体引用)。0 关闭年龄清扫;手动入口 POST /__bili/storage/gc 始终可用。配置文件键 storage.archiveAfterDays 为持久形式。" - key: BILI_SNAPSHOT_SHADOW type: boolean default: "true (on)" diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index b82733d859..155f4e82ee 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -278,8 +278,8 @@ entries: default: "365" env: BILI_STORAGE_ARCHIVE_AFTER_DAYS group: process - desc_en: "GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - full export to the archive dir under the sessions directory, a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). Env wins over the file tier." - desc_zh: "GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 完整导出到会话目录下的 archive/,墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。环境变量优先于配置文件层。" + desc_en: "GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). Env wins over the file tier." + desc_zh: "GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。环境变量优先于配置文件层。" - key: sessions type: object default: "{} (all defaults)"