diff --git a/CHANGELOG.md b/CHANGELOG.md index f0878e9053..79c36fbae5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,15 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag ## [Unreleased] +### 存储收尾九项一车(#3901,修复 #3463/#3762/#3841/#3798/#3633;#3723 落地) + +- **GC 与归档(#3463)**:新 `storage.archiveAfterDays`(环境变量 `BILI_STORAGE_ARCHIVE_AFTER_DAYS`,缺省 365,0=关)——闲置超期的会话归档(完整导出到 sessions 目录下 `archive/` + 墓碑行保标签 + 释放全部内容/媒体引用),`POST /__bili/storage/gc`(可带 `vacuum`)手动触发,`POST /__bili/storage/restore` 恢复;`auto_vacuum=INCREMENTAL` 常开(存量库首次需 `?vacuum=full` 收缩)。 +- **reap 台账(#3841)**:schema v6 `sessions.orphans_json`——收割形状(state.blocks 失块、blockContents 留文)现在定向记台账:被收割块行删除(引用计数释放),孤儿内容并入视图但不复活 state.blocks;重启不再复活已收割块、不再污染信封。 +- **媒体 CAS(#3762)**:schema v7 `message_media`——图片 base64 以内容哈希进 CAS、消息行只存 `{"$media":hash}` 引用(读回逐字节水合),同图多会话只存一份;删除/归档时引用计数释放。 +- **增量指针(#3798)**:`inc:canonical-ptrs` 逐行指针替代 base-seq 计数切片——resync 共享前缀的行复用不再丢历史;resync 时点先失效旧指针(响应前持久化),崩溃窗口退回旧切片语义(永不供错分支);turn 锁重排:响应阶段 409 turn-in-flight 照旧、ack 之后的回写链被下一轮 await(ack 永不超前行落盘)。 +- **dsh 代压缩默认开(#3723/#3733 落地)**:`dsh.authoredCompaction` 缺省 true(环境变量 `BILI_DSH_AUTHORED_COMPACTION=0` 或文件 false 恢复 #1729 拒绝路径)——0.1.203 狗粮窗口守住,失败模式只是 checkpoint 不缩,不丢数据。 +- **多实例姿态文档(#3633)**:README 新章节「Running multiple instances」:一会话一实例、L7 按会话头哈希才是正确的 sticky 路由(ClientIP affinity 是故障转移反模式)、RWO 一实例一卷(NFS 上 WAL=静默损坏)、/__bili/replicas 是 checkpoint 不是 pod 副本。 + ### 快照渲染影子对账(#3756 后续,默认开启) - 新增影子车道:灰度仍为 0% 的出站快照引擎在每轮已折叠会话上现算视图,与 legacy 实际出站比对后丢弃 —— 出站字节零变化,纯观测。 diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 0f4160037e..610b9c4c8a 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -117,6 +117,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `compat.noCacheControl` | boolean | false | BILI_NO_CACHE_CONTROL | Stop stamping Anthropic-lane cache_control breakpoints entirely (escape hatch for upstreams/relays with their own breakpoint policy). | | `compat.keepResponseId` | boolean | false | ACP_KEEP_RESPONSE_ID | Preserve previous_response_id on kernel-rebuilt Responses requests (default strips it so rebuilt bodies never reference ids the upstream never issued). | | `dsh.allowDshCompaction` | boolean | false | BILI_ALLOW_DSH_COMPACTION | Lift bili's local 422 refusal of dsh native compaction calls on EVERY wire lane bili serves — openai, anthropic, and responses (#1729/#2028; responses added by #2360, the lane dsh desktop's compaction actually rides): when true, dsh's compaction-basic may execute through bili; its checkpoint durably shadows raw history, so this is an explicit irreversible opt-in. | +| `dsh.authoredCompaction` | boolean | true (on, #3901 flip) | BILI_DSH_AUTHORED_COMPACTION | bili answers dsh compaction summarize calls locally with the folded snapshot instead of refusing (#3723/#3733). ON by default since the 0.1.203 dogfood window held; false (or env BILI_DSH_AUTHORED_COMPACTION=0) restores the #1729 refusal path. | | `resign` | scheme → { enabled?, passthrough?, credentialRef? } | {} (armed; built-in scheme sdk-hmac-sha256) | BILI_RESIGN, BILI_RESIGN_PASSTHROUGH, BILI_CODEARTS_REF, BILI_RESIGN_BENEFIT | Re-sign arm keyed by Authorization scheme: signed model requests tunnel with every egress body re-signed; unresignable requests are refused locally 403 unless passthrough opts that scheme into verbatim forwarding. | | `promptCache.routing` | "auto" \| "enabled" \| "disabled" | auto | ACP_PROMPT_CACHE_ROUTING | Prompt-cache routing posture for cache-aware lane selection. | | `native.attachExternal` | boolean | false | BILI_NATIVE_ATTACH_EXTERNAL | Let launcher-less native plugins attach to an external (lane'd, unarmed) daemon instead of spawning their own. | @@ -177,6 +178,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | +| `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - full export to the archive dir under the sessions directory, a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). Env wins over the file tier. | | `sessions` | object | {} (all defaults) | — | Session-table size cap and opt-in garbage collection. | | `sessions.max` | number | 256 | BILI_MAX_SESSIONS | In-memory session table cap (LRU eviction of idle sessions). | | `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | Bounded startup (#2803/#3050): max most-recently-saved sessions hydrated at boot before listening; older state reloads on demand. Default 0 (no eager warm set — first touch pays one on-demand load; set 8 to restore the eager behavior). Capped by sessions.max. | @@ -330,7 +332,8 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | Key | Type | Default | Env | Description | |-----|------|---------|-----|-------------| -| `BILI_DSH_AUTHORED_COMPACTION` | boolean | false | — | Let bili ANSWER dsh compaction summarize calls locally with the folded snapshot instead of refusing (#3733): identity header + kernel-rendered block summaries, zero egress; replay recognized via the #2432 rebase. | +| `BILI_DSH_AUTHORED_COMPACTION` | boolean | true (on, #3901 flip) | — | Let bili ANSWER dsh compaction summarize calls locally with the folded snapshot instead of refusing (#3733): identity header + kernel-rendered block summaries, zero egress; replay recognized via the #2432 rebase. | +| `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | integer | 365 | — | GC age sweep window in days (#3463): sessions idle longer than this are archived (full export to the archive dir + tombstone row + content/media refs released). 0 disables the age sweep; the manual entry POST /__bili/storage/gc always runs. Config-file key storage.archiveAfterDays is the persistent form. | | `BILI_SNAPSHOT_SHADOW` | boolean | true (on) | — | Snapshot-render shadow observability (#3756 follow-up): compute the outbound snapshot view on every armed turn WITHOUT using it, classify the diff against the legacy outbound, and surface per-session counters in the web panel + the snapshot-shadow diagnostics topic. 0/false/off disables; 1/true/on enables; otherwise the config-file snapshot.shadow boolean applies (default on). The render lane itself stays rollout-gated (BILI_ROLLOUT_SNAPSHOT_RENDER), so the shadow lane never changes outbound bytes - it is the dry-run evidence that flips the rollout later. | | `BILI_CONFIG_FILE` | string | unset (XDG config path) | — | Override the config file path (path relocation). | | `BILI_SESSIONS_DIR` | string | unset (XDG state path) | — | Where session records are stored (path relocation). | @@ -1717,6 +1720,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_CODEX_COMPACT` | `codexCompact` | "intercept" | | `BILI_DECODED_BODY_BYTES` | `network.decodedBodyBytes` | 10737418240 | | `BILI_DECOMPRESS_TMP_CAP` | `decompressTmpCap` | 50 | +| `BILI_DSH_AUTHORED_COMPACTION` | `dsh.authoredCompaction` | true (on, #3901 flip) | | `BILI_DUMP_4XX` | `diagnostics.dump4xx` | false | | `BILI_DUMP_4XX_MAX_BYTES` | `diagnostics.dump4xxMaxBytes` | 2097152 (floor 1024) | | `BILI_DUMP_MAX_AGE_DAYS` | `diagnostics.maxAgeDays` | unset (off) | @@ -1778,6 +1782,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_SESSION_WARM_MAX_BYTES` | `sessions.warmMaxBytes` | 67108864 | | `BILI_SQLITE_AUTO_DOWNLOAD` | `storage.sqliteAutoDownload` | true | | `BILI_STABLE_SYSTEM_ANCHOR` | `stableSystemAnchor` | false | +| `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | `storage.archiveAfterDays` | 365 | | `BILI_STORAGE_JSON_WRITER` | `storage.jsonWriter` | on | | `BILI_STORAGE_UNIFIED` | `storage.unified` | unset (full) | | `BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES` | `storage.walJournalSizeLimitBytes` | 8388608 | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 11a4f9a397..565d8c570a 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -117,6 +117,7 @@ | `compat.noCacheControl` | boolean | false | BILI_NO_CACHE_CONTROL | 完全停止 Anthropic 通道的 cache_control 断点打标(面向拒收该字段或有自有断点策略的上游/中继的逃生门)。 | | `compat.keepResponseId` | boolean | false | ACP_KEEP_RESPONSE_ID | 在内核重建的 Responses 请求上保留 previous_response_id(默认剔除,使重建请求体永不引用上游从未签发过的响应 id)。 | | `dsh.allowDshCompaction` | boolean | false | BILI_ALLOW_DSH_COMPACTION | 解除 bili 对 dsh 原生压缩调用的本地 422 拒绝——覆盖 bili 服务的全部线路:openai、anthropic、responses(#1729/#2028;responses 由 #2360 补上,正是 dsh 桌面端压缩实际走的线路):为 true 时 dsh 的 compaction-basic 可经 bili 执行;其 checkpoint 会永久覆盖原始历史,属显式的不可逆 opt-in。 | +| `dsh.authoredCompaction` | boolean | true (on, #3901 flip) | BILI_DSH_AUTHORED_COMPACTION | bili 用折叠快照本地应答 dsh 压缩摘要请求而非拒绝(#3723/#3733)。0.1.203 狗粮窗口守住后默认开启;false(或环境变量 BILI_DSH_AUTHORED_COMPACTION=0)恢复 #1729 拒绝路径。 | | `resign` | scheme → { enabled?, passthrough?, credentialRef? } | {} (armed; built-in scheme sdk-hmac-sha256) | BILI_RESIGN, BILI_RESIGN_PASSTHROUGH, BILI_CODEARTS_REF, BILI_RESIGN_BENEFIT | 按 Authorization 方案键控的重签名臂:可重签名的模型请求全程重签转发;无法重签的请求本地 403 拒绝,除非该方案经 passthrough 选择原文透传。 | | `promptCache.routing` | "auto" \| "enabled" \| "disabled" | auto | ACP_PROMPT_CACHE_ROUTING | 面向缓存感知路由选择的提示词缓存路由姿态。 | | `native.attachExternal` | boolean | false | BILI_NATIVE_ATTACH_EXTERNAL | 允许无启动器的原生插件挂到外部(车道化、未武装看门狗)守护进程,而不是自行派生。 | @@ -177,6 +178,7 @@ | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | +| `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 完整导出到会话目录下的 archive/,墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。环境变量优先于配置文件层。 | | `sessions` | object | {} (all defaults) | — | 会话表容量上限与可选垃圾回收。 | | `sessions.max` | number | 256 | BILI_MAX_SESSIONS | 内存会话表上限(空闲会话 LRU 逐出)。 | | `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | 有界启动(#2803/#3050):启动监听前预热加载的最近保存会话数上限,更早的状态按需重新加载。默认 0(不预热——首次访问按需加载一次;设 8 恢复预热行为)。受 sessions.max 封顶。 | @@ -330,7 +332,8 @@ | Key | Type | Default | Env | Description | |-----|------|---------|-----|-------------| -| `BILI_DSH_AUTHORED_COMPACTION` | boolean | false | — | 让 bili 本地应答 dsh 压缩摘要请求(返回折叠快照而非拒绝,#3733):身份头 + 内核渲染格式的块摘要,零出口流量;重放经 #2432 再基线识别。 | +| `BILI_DSH_AUTHORED_COMPACTION` | boolean | true (on, #3901 flip) | — | 让 bili 本地应答 dsh 压缩摘要请求(返回折叠快照而非拒绝,#3733):身份头 + 内核渲染格式的块摘要,零出口流量;重放经 #2432 再基线识别。 | +| `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | integer | 365 | — | GC 年龄清扫窗口(天,#3463):闲置超过该时长的会话被归档(完整导出到归档目录 + 墓碑行保留标签 + 释放内容/媒体引用)。0 关闭年龄清扫;手动入口 POST /__bili/storage/gc 始终可用。配置文件键 storage.archiveAfterDays 为持久形式。 | | `BILI_SNAPSHOT_SHADOW` | boolean | true (on) | — | 快照渲染影子对账(#3756 后续):每轮已折叠会话都用引擎现算一次出站快照视图但不使用,与 legacy 实际出站比对分类,逐会话计数进 web 面板与 snapshot-shadow 诊断主题。0/false/off 关闭;1/true/on 开启;否则读配置文件 snapshot.shadow 布尔(默认开)。渲染车道本身仍受灰度开关控制(BILI_ROLLOUT_SNAPSHOT_RENDER),影子车道绝不改变出站字节 —— 它是后续翻转灰度的干跑证据。 | | `BILI_CONFIG_FILE` | string | unset (XDG config path) | — | 覆盖配置文件路径(路径重定位)。 | | `BILI_SESSIONS_DIR` | string | unset (XDG state path) | — | 会话记录的存放目录(路径重定位)。 | @@ -1716,6 +1719,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_CODEX_COMPACT` | `codexCompact` | "intercept" | | `BILI_DECODED_BODY_BYTES` | `network.decodedBodyBytes` | 10737418240 | | `BILI_DECOMPRESS_TMP_CAP` | `decompressTmpCap` | 50 | +| `BILI_DSH_AUTHORED_COMPACTION` | `dsh.authoredCompaction` | true (on, #3901 flip) | | `BILI_DUMP_4XX` | `diagnostics.dump4xx` | false | | `BILI_DUMP_4XX_MAX_BYTES` | `diagnostics.dump4xxMaxBytes` | 2097152 (floor 1024) | | `BILI_DUMP_MAX_AGE_DAYS` | `diagnostics.maxAgeDays` | unset (off) | @@ -1777,6 +1781,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_SESSION_WARM_MAX_BYTES` | `sessions.warmMaxBytes` | 67108864 | | `BILI_SQLITE_AUTO_DOWNLOAD` | `storage.sqliteAutoDownload` | true | | `BILI_STABLE_SYSTEM_ANCHOR` | `stableSystemAnchor` | false | +| `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | `storage.archiveAfterDays` | 365 | | `BILI_STORAGE_JSON_WRITER` | `storage.jsonWriter` | on | | `BILI_STORAGE_UNIFIED` | `storage.unified` | unset (full) | | `BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES` | `storage.walJournalSizeLimitBytes` | 8388608 | diff --git a/README.md b/README.md index 713d5bb334..8b3ebb6cd5 100644 --- a/README.md +++ b/README.md @@ -542,7 +542,41 @@ client already supplied (e.g. a body `session_id` is forwarded upstream as A child session (subagent/fork starting from an empty history) reports its lineage at birth — its identity registration carries `parentConversationId` and the proxy records a read-only `derivedFrom` link — so `decompress` / `search_context` fall back along the parent chain for content the child never saw itself (cycle-guarded, depth cap 8). Nothing is copied into the child, the parent is never modified; if the parent is unknown the child simply starts fresh. Per-lane parent signals (pi/omp `parentSession` header, OpenCode V1/V2 `parentID`): [SESSION-IDENTITY.md](SESSION-IDENTITY.md#derived-child-sessions-lineage-at-birth-1333-1362). claude/codex/dsh need nothing here: they share one session id across subagents or have no child-session concept at all. -### Windows: exclude the sessions dir from antivirus (#362) +### Running multiple instances (#3633) + +**One conversation belongs to one instance for its whole life.** Fold state, +prefix affinity and the summary budget are keyed to a session id whose home is +the client's own history — and the session's rows live in that instance's +`index.db`. There is no shared store between instances: nothing replicates, +nothing fails over, nothing rehydrates a session onto a different instance. + +Consequences for anyone tempted to put several instances behind one Service: + +- **Sticky routing is required, and `ClientIP` affinity is the wrong kind.** + The conversation identity lives in a header or the request body, not the + source IP — a Service can only hash on IP, so a client pinned to replica N + silently loses *every* session when replica N dies (buying throughput while + selling away availability, exactly when you scaled out because of load). + Correct routing is an L7 proxy hashing a conversation header + (`x-bili-plugin-conversation`, `x-opencode-session-id`, `x-acp-session`, …) + onto a backend — and the client actually sending one (header-less clients + get prefix affinity instead, which an L7 proxy cannot hash). +- **Instance count is volume count.** The state directory is one local + directory with SQLite in WAL mode (`busy_timeout=5000`); a coherent shared + `-shm` and working byte-range locks across writers require an RWO volume + that is never shared. Pointing two instances at the same RWX/NFS path does + not give you a shared store — it gives you silent corruption behind a + timeout. One RWO volume per instance, `storage.unified` included: "unified" + means one store *for that instance* (sessions + CCR + CAS in one + `index.db`), not one store for the fleet. +- **The workload that actually scales** is *many independent short-lived + conversations* — one proxy per client or per lane — not *one long + conversation load-balanced across proxies*. Long conversations are + single-instance by design. + +Naming note: `POST /__bili/replicas` creates a session **checkpoint fork** +(from stored checkpoints), not a pod replica — read it as `/__bili/checkpoints`. + The proxy rewrites each session's state file every turn of a long session; on Windows, real-time AV (Defender), the search indexer, or a sync tool (OneDrive) can lock the sessions dir mid-write, so persists fail with `EPERM` until the lock clears. Fix at the root: add `%USERPROFILE%\.local\share\billion-context\` to your antivirus exclusions (Defender steps included) and keep sync tools off that path — [CONFIGURATION.md](CONFIGURATION.md#windows-exclude-the-sessions-dir-from-antivirus-362). The same exclusion also covers the unified-storage `index.db` refusing to open (`unable to open database file`, #3823) — the storage-degrade warning names it as the first suspect. diff --git a/src/config.ts b/src/config.ts index 0819018246..6d26b6e4e0 100644 --- a/src/config.ts +++ b/src/config.ts @@ -912,6 +912,10 @@ export type ProxyOptions = { export interface DshFileSettings { /** Enable with `{ "dsh": { "allowDshCompaction": true } }`; full behavior contract on the FileConfig side. */ allowDshCompaction?: boolean; + /** #3723/#3901: bili-authored compaction answers (ON by default since + * the 0.1.203 dogfood window; `false` or env `BILI_DSH_AUTHORED_COMPACTION=0` + * restores the #1729 refusal path). */ + authoredCompaction?: boolean; } /** The routing fields a provider entry can carry — exactly what @@ -1841,6 +1845,10 @@ type FileConfig = { * the file/default tier. Applied when the store opens (lane boot) — * hot edits apply on the next restart. */ walJournalSizeLimitBytes?: number; + /** #3901/#3463: session retention window in days for the GC age + * sweep (archive + tombstone + CAS release). 0 disables; env + * BILI_STORAGE_ARCHIVE_AFTER_DAYS wins. Default 365. */ + archiveAfterDays?: number; }; /** Network & timing knobs (#2030) — every field was an env-only variable * before (BILI_*_MS / BILI_REPLAY_* family); env still wins when set. diff --git a/src/incremental-lane.ts b/src/incremental-lane.ts index 6900a24888..7f60d49aff 100644 --- a/src/incremental-lane.ts +++ b/src/incremental-lane.ts @@ -95,6 +95,14 @@ export function incrementalTurnBodyShape(raw: Buffer): boolean { const EXT_KEY_SYSTEM = "inc:system"; const EXT_KEY_CACHE_MARKS = "inc:cache-marks"; const EXT_KEY_BASE_SEQ = "inc:base-seq"; +/** #3798: ordered row pointers (seq numbers) of the CURRENT canonical + * branch. The base-seq COUNT cutoff assumed the new branch lands as a + * contiguous suffix, but remapPlanToAbsolute reuses rows for identical + * (msg_id, content_hash) — a resync sharing history with the rejected + * branch maps onto rows BELOW the cutoff, and the count slice silently + * drops the shared prefix (cold rebuild served a partial history and 409'd + * the client into a reseed loop). Pointers name the exact rows instead. */ +const EXT_KEY_CANONICAL_PTRS = "inc:canonical-ptrs"; /** Request-param allowlist for the forwarded anthropic body. `messages` and * `system` are owned by the lane (assembled from canonical + payload.system) @@ -157,6 +165,10 @@ interface Canonical { } const canonicalCache = new Map(); const inFlight = new Set(); +/** #3798: per-conversation pointer-recording chains. The next turn awaits + * its predecessor's chain so an applied-seq ack can never outrun the + * pointer durability of the turn before it. */ +const pendingPtrWork = new Map>(); function touchCache(conversationId: string, entry: Canonical): void { canonicalCache.delete(conversationId); @@ -193,8 +205,31 @@ function readCanonicalFromStore(sessionId: string): { kind: "ok"; msgs: BiliMess return { kind: "unavailable" }; } if (rows === null) return { kind: "unavailable" }; - // Divergent-resync epoch: rows below baseSeq belong to a branch the - // client already replaced — serve from the current branch only. + // #3798 pointer replay: the ext key names the current branch's rows in + // canonical order. Present-but-broken must NEVER serve a partial history + // — report bad and let the lane force an honest client resync. + const ptrs = readCanonicalPointers(sessionId); + // Pointer freshness is guarded at the WRITE side, not here: a resync + // invalidates them durably (null, pre-response) and the wrap-up chain + // re-records the new branch's rows. A null here therefore means either + // "no pointers yet" or "invalidated, rewrite lost to a crash" — both + // fall to the legacy epoch slice, which serves the CURRENT branch by + // construction. Identical-history resyncs deliberately rely on the + // pointers surviving (rows are REUSED below the count cutoff, #3798). + if (ptrs !== null) { + if (ptrs.length === 0) return { kind: "ok", msgs: [] }; + const bySeq = new Map(rows.map((r) => [r.seq, r])); + const picked: { seq: number; content: string }[] = []; + for (const seq of ptrs) { + const row = bySeq.get(seq); + if (row === undefined) return { kind: "bad", err: "canonical pointers reference missing rows" }; + picked.push(row); + } + const msgs = parseCanonicalRows(picked); + return msgs === null ? { kind: "bad", err: "malformed canonical rows" } : { kind: "ok", msgs }; + } + // Legacy epoch (no pointers recorded yet): rows below baseSeq belong to + // a branch the client already replaced — serve from the current branch only. const baseSeq = readBaseSeq(sessionId); if (baseSeq > 0) rows = rows.slice(baseSeq); const msgs = parseCanonicalRows(rows); @@ -245,6 +280,69 @@ function readBaseSeq(sessionId: string): number { } } +/** #3798: read the persisted canonical pointers. Null = legacy epoch (the + * base-seq count fallback); an array = pointer replay (possibly empty — an + * explicitly empty canonical); anything else = malformed → treated as + * legacy (the next successful turn rewrites them). */ +function readCanonicalPointers(sessionId: string): number[] | null { + let v: unknown; + try { + v = getStore().getSessionExt(sessionId, EXT_KEY_CANONICAL_PTRS); + } catch { + return null; + } + if (!Array.isArray(v)) return null; + if (v.every((e) => typeof e === "number" && Number.isSafeInteger(e) && e >= 0)) return v as number[]; + return null; +} + +/** Stable projection of a core message for pointer matching. The row's + * content is the PIPELINE's serialization (prepare re-derives core from the + * assembled wire body), which re-stamps presentation fields like + * wireIndex by full-body position - verbatim JSON equality would miss. + * Content-derived ids (plus the cluster suffix) are unique, so + * [id, role, contentType, text, media size] pins the exact row. */ +function stableMessageKey(m: Record): string { + const text = typeof m.text === "string" ? m.text : typeof m.content === "string" ? m.content : null; + const media = typeof m.imageBase64 === "string" ? m.imageBase64.length : null; + return JSON.stringify([m.id, m.role, m.contentType, text, media]); +} + +/** #3798: after a turn's raw tail has settled into the unified table, pin + * the canonical to the exact rows that carry it. Matching uses the stable + * projection (see stableMessageKey) — the row content is the pipeline's + * serialization, which re-stamps presentation fields like wireIndex by + * full-body position, so verbatim JSON equality would miss. Any miss → + * leave the previous pointers untouched (never a partial list) and warn; + * the next settled turn rewrites them. */ +function recordCanonicalPointers(sessionId: string, msgs: BiliMessage[], log: (l: string, m: string) => void): void { + try { + const store = getStore(); + const rows = store.unifiedRawMessages(sessionId); + if (rows === null) return; + const byStable = new Map(); + for (const r of rows) { + try { + byStable.set(stableMessageKey(JSON.parse(r.content) as Record), r.seq); + } catch { + // unparseable row -> skip; a miss keeps the previous pointers + } + } + const ptrs: number[] = []; + for (const m of msgs) { + const seq = byStable.get(stableMessageKey(m as unknown as Record)); + if (seq === undefined) { + log("warn", `[incremental] canonical pointers not recorded for ${sessionId}: a message row is not settled yet — cold rebuilds use the legacy base-seq slice until the next turn`); + return; + } + ptrs.push(seq); + } + store.setSessionExt(sessionId, EXT_KEY_CANONICAL_PTRS, ptrs); + } catch (err) { + log("warn", `[incremental] canonical pointers not recorded for ${sessionId}: ${String(err)}`); + } +} + async function recordResyncBaseSeq(sessionId: string, log: (l: string, m: string) => void): Promise { try { const store = getStore(); @@ -255,7 +353,14 @@ async function recordResyncBaseSeq(sessionId: string, log: (l: string, m: string if (warm) await store.writeNow(warm); const rows = store.unifiedRawMessages(sessionId); const count = rows ? rows.length : 0; - if (count > 0) store.setSessionExt(sessionId, EXT_KEY_BASE_SEQ, count); + if (count > 0) { + store.setSessionExt(sessionId, EXT_KEY_BASE_SEQ, count); + // #3798: the old branch's pointers are now meaningless — dropping + // them HERE (pre-response, durable with the epoch) means a crash + // can never resurrect the replaced branch through stale pointers; + // the post-turn chain re-records the new branch's pointers. + store.setSessionExt(sessionId, EXT_KEY_CANONICAL_PTRS, null); + } } catch (err) { log("warn", `[incremental] resync epoch not recorded for ${sessionId}: ${String(err)} — cold rebuilds will serve the full row union until the next resync`); } @@ -315,9 +420,19 @@ function applyTurn( */ /** Test seam: drop the warm canonical caches and in-flight locks so a test * can simulate a process restart (cold rebuild from the unified table). */ +/** Test seam: the per-conversation pointer-durability chain for the most + * recent turn, so tests can model a REAL client (whose next turn awaits + * the chain server-side) before simulating a restart. A crash inside the + * chain's window leaves the pointers one turn stale — same window as the + * unified rows themselves. */ +export function __incrementalPtrChainForTests(conversationId: string): Promise | undefined { + return pendingPtrWork.get(conversationId); +} + export function __resetIncrementalLaneForTests(): void { canonicalCache.clear(); inFlight.clear(); + pendingPtrWork.clear(); droppedParamWarned.clear(); droppedParamWarnPairs = 0; } @@ -405,9 +520,24 @@ export async function handleIncrementalTurn( const deltaWire = payload.messages; // Per-conversation turn lock: the ack contract is strictly sequential. + // The 409 fires only for turns arriving while the holder is INSIDE its + // response phase (between entry and forward completion) — a concurrent + // sender gets the documented refusal. Turns arriving after the ack + // (holder in wrap-up: rows + pointer durability) pass the guard and + // AWAIT that wrap-up chain first, so an ack can never outrun the + // previous turn's rows (#3798). A turn that slips in behind such a + // waiter re-checks inFlight and takes the retriable 409. + if (inFlight.has(conversationId)) return jsonReply(res, 409, { error: "turn-in-flight", conversationId }); + const pendingPtrs = pendingPtrWork.get(conversationId); + if (pendingPtrs) await pendingPtrs; if (inFlight.has(conversationId)) return jsonReply(res, 409, { error: "turn-in-flight", conversationId }); inFlight.add(conversationId); - let release: (() => void) | undefined; + let releaseChain: (() => void) | undefined; + let settled = false; + const chain = new Promise((resolve) => { + releaseChain = resolve; + }); + pendingPtrWork.set(conversationId, chain); try { const sessionId = laneSessionId(conversationId); let entry = canonicalCache.get(conversationId); @@ -500,24 +630,38 @@ export async function handleIncrementalTurn( } return (originalWriteHead as (...a: unknown[]) => ReturnType)(...args); }; - release = (): void => { + const releaseWriteHead = (): void => { (res as { writeHead?: unknown }).writeHead = originalWriteHead as unknown as never; }; log("info", `[incremental] turn ${conversationId} since=${String(sinceSeq)} applied=${String(appliedSeq)} agent=${agent} model=${String(model)} → forward ${path}`); // Best-effort durability close-out: flush the session once the // response has fully left so the canonical table catches up with the // ack before the client can act on the next turn. - res.on("finish", () => { - try { - const session = peekSession(sessionId); - if (session) void getStore().writeNow(session).catch(() => undefined); - } catch { - // flushing is opportunistic; the debounced save path covers it anyway - } - }); await forward(synthetic, res); - } finally { - release?.(); + releaseWriteHead(); + // #3798: forward() completing means the response phase is over — the + // holder leaves inFlight here, and the wrap-up chain (rows + pointer + // durability) floats behind the ack; the next turn awaits it at + // entry. Persist failures only delay pointer freshness — the epoch + // guard covers a cold rebuild in the window. inFlight.delete(conversationId); + const session = peekSession(sessionId); + settled = true; + void (session + ? getStore().writeNow(session).then(() => recordCanonicalPointers(sessionId, entry.msgs, log)) + : Promise.resolve() + ) + .catch(() => undefined) + .finally(() => { + releaseChain?.(); + if (pendingPtrWork.get(conversationId) === chain) pendingPtrWork.delete(conversationId); + }); + } finally { + // Exception path (never entered the wrap-up): release everything. + if (!settled) { + releaseChain?.(); + inFlight.delete(conversationId); + if (pendingPtrWork.get(conversationId) === chain) pendingPtrWork.delete(conversationId); + } } } diff --git a/src/knobs.ts b/src/knobs.ts index 974a1b896e..40c5f0b899 100644 --- a/src/knobs.ts +++ b/src/knobs.ts @@ -511,7 +511,14 @@ export function codexCompactMode(): CodexCompactMode { export function dshAuthoredCompactionEnabled(): boolean { const envRaw = process.env.BILI_DSH_AUTHORED_COMPACTION; if (envRaw !== undefined) return envRaw.trim() !== "0"; - return false; + // #3723/#3901 flip: ON by default since the 0.1.203 dogfood window held + // (the guard answers locally, zero egress — the failure mode is a dsh + // checkpoint not shrinking, never data loss). The config-file key + // `dsh.authoredCompaction` mirrors the env; "0"/false is the escape + // hatch back to the #1729 refusal path. + const file = loadConfigFile().dsh?.authoredCompaction; + if (typeof file === "boolean") return file; + return true; } /** BILI_DECOMPRESS_TMP_CAP > decompressTmpCap > 50. */ @@ -701,6 +708,11 @@ export function fileStorageAutoDownload(): boolean | undefined { return typeof v === "boolean" ? v : undefined; } +export function fileStorageArchiveAfterDays(): number | undefined { + const v = loadConfigFile().storage?.archiveAfterDays; + return typeof v === "number" && Number.isFinite(v) && v >= 0 ? v : undefined; +} + export function fileStorageWalJournalSizeLimit(): number | undefined { const v = loadConfigFile().storage?.walJournalSizeLimitBytes; return typeof v === "number" && Number.isInteger(v) && v > 0 ? v : undefined; @@ -719,6 +731,21 @@ export type StorageMode = "legacy" | "shadow" | "full"; /** Resolve the unified-storage level. Pure: env + file tier value in, mode out * (total function — never throws; an explicit legacy or a garbage spelling * degrades to legacy, only full absence lands on the default). */ +/** #3901/#3463: session retention window in days. Sessions whose + * updated_at is older than the cutoff are archived by gc() (export + + * tombstone + CAS release). 0 disables the age sweep entirely. env + * BILI_STORAGE_ARCHIVE_AFTER_DAYS > file storage.archiveAfterDays > + * default 365. */ +export function resolveArchiveAfterDays(env: NodeJS.ProcessEnv = process.env, fileDays?: number): number { + const raw = env.BILI_STORAGE_ARCHIVE_AFTER_DAYS?.trim(); + if (raw !== undefined && raw !== "") { + const n = Number.parseInt(raw, 10); + if (Number.isFinite(n) && n >= 0) return n; + } + if (typeof fileDays === "number" && Number.isFinite(fileDays) && fileDays >= 0) return fileDays; + return 365; +} + export function resolveStorageMode(env: NodeJS.ProcessEnv = process.env, fileUnified?: string): StorageMode { const raw = env.BILI_STORAGE_UNIFIED?.trim().toLowerCase(); if (raw) { diff --git a/src/persist.ts b/src/persist.ts index 1f631053fa..b71ecc080b 100644 --- a/src/persist.ts +++ b/src/persist.ts @@ -2,6 +2,7 @@ import { createHash, randomUUID } from "node:crypto"; import { closeSync, existsSync, fsyncSync, mkdirSync, openSync, readFileSync, renameSync, rmSync, statSync, writeFileSync } from "node:fs"; import { readdir, readFile, rm } from "node:fs/promises"; import * as path from "node:path"; +const { join } = path; import { StateStore, flatFileNameFor, type PersistedEnvelope, type StateStoreCodec } from "acp-kernel/persist"; import { sessionsDir } from "./paths.js"; import { @@ -12,8 +13,10 @@ import { persistEpermAlertThreshold as knobPersistEpermAlertThreshold, persistEpermAlertRepeatMs as knobPersistEpermAlertRepeatMs, resolveJsonWriterOff as knobResolveJsonWriterOff, + resolveArchiveAfterDays, fileStorageUnified, fileStorageJsonWriter, + fileStorageArchiveAfterDays, fileStorageWalJournalSizeLimit, } from "./knobs.js"; import { log as loggerLog } from "./logger.js"; @@ -1454,6 +1457,32 @@ export class SessionStore { * #3166 watermark), so the table is a faithful canonical log even while * the JSON envelope stays the truth base. Null when the unified store is * unavailable (legacy-only lanes) — callers must not guess. */ + /** #3901/#3463: run the unified-store GC (age-sweep archive + zero-ref + * collection + optional vacuum) through the live sidecar. Returns null + * when the unified store is unavailable (legacy-only lanes). The + * archive dir defaults to /archive next to the envelopes. */ + unifiedGc(opts: { archiveAfterDays?: number; vacuum?: "full" | "incremental"; archiveDir?: string } = {}): { archived: number; contentRowsDeleted: number } | null { + if (!this.shadow || this.shadow.stats.disabled) return null; + try { + const days = opts.archiveAfterDays ?? resolveArchiveAfterDays(undefined, fileStorageArchiveAfterDays()); + return this.shadow.gc({ archiveAfterDays: days, archiveDir: opts.archiveDir ?? join(this.sessionsDir, "archive"), vacuum: opts.vacuum }); + } catch (err) { + this.log("warn", `[persist] unified gc failed: ${String(err)}`); + return null; + } + } + + /** #3901/#3463: restore an archived session from its export file. */ + unifiedRestoreArchived(sessionId: string, archiveDir?: string): { ok: boolean; reason?: string } | null { + if (!this.shadow || this.shadow.stats.disabled) return null; + try { + return this.shadow.restoreArchived(sessionId, { archiveDir: archiveDir ?? join(this.sessionsDir, "archive") }); + } catch (err) { + this.log("warn", `[persist] unified restore failed: ${String(err)}`); + return null; + } + } + unifiedRawMessages(sessionId: string): { seq: number; content: string }[] | null { if (!this.shadow || this.shadow.stats.disabled) return null; try { diff --git a/src/server/admin.ts b/src/server/admin.ts index bdbbcc0340..222ad0897b 100644 --- a/src/server/admin.ts +++ b/src/server/admin.ts @@ -457,6 +457,67 @@ export async function handleAdminRoute(req: http.IncomingMessage, res: http.Serv res.end(JSON.stringify({ ok: true })); return; } + // #3901/#3463: unified-storage GC — age-sweep archive + zero-ref + // collection + optional vacuum. Loopback-gated like every /__bili/ + // route; 501 in legacy mode (same discipline as /__bili/replicas). + // #3901 review: the documented form is `?vacuum=full` (CONFIGURATION.md / + // issue spec), but exact-match routing dropped any query string — accept + // both, query wins over body. + if (req.method === "POST" && (req.url === "/__bili/storage/gc" || req.url?.startsWith("/__bili/storage/gc?"))) { + try { + const parsed = JSON.parse((await readBody(req)).toString("utf8")) as { archiveAfterDays?: unknown; vacuum?: unknown }; + const store = getStore(); + if (!store.replicasAvailable()) { + res.writeHead(501, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: false, error: "storage gc requires the unified store — start this instance with BILI_STORAGE_UNIFIED=shadow|full (or storage.unified in the config file)" })); + return; + } + const qVacuum = (() => { try { return new URL(req.url!, "http://localhost").searchParams.get("vacuum"); } catch { return null; } })(); + const vacuum = qVacuum === "full" || qVacuum === "incremental" ? qVacuum : parsed.vacuum === "full" ? "full" : parsed.vacuum === "incremental" ? "incremental" : undefined; + const result = store.unifiedGc({ + archiveAfterDays: typeof parsed.archiveAfterDays === "number" && Number.isFinite(parsed.archiveAfterDays) && parsed.archiveAfterDays >= 0 ? parsed.archiveAfterDays : undefined, + vacuum, + }); + if (!result) { + res.writeHead(503, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: false, error: "unified store unavailable mid-call" })); + return; + } + log("info", `[admin] POST /__bili/storage/gc — archived=${result.archived} contentRowsDeleted=${result.contentRowsDeleted} vacuum=${vacuum ?? "none"}`); + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: true, ...result })); + return; + } catch (err) { + res.writeHead(err instanceof BodyTooLargeError ? 413 : 400, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: false, error: String(err) })); + return; + } + } + if (req.method === "POST" && req.url === "/__bili/storage/restore") { + try { + const parsed = JSON.parse((await readBody(req)).toString("utf8")) as { sessionId?: unknown; archiveDir?: unknown }; + if (typeof parsed.sessionId !== "string" || !parsed.sessionId) { + res.writeHead(400, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: false, error: "expected { sessionId: string, archiveDir?: string }" })); + return; + } + const store = getStore(); + if (!store.replicasAvailable()) { + res.writeHead(501, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: false, error: "restore requires the unified store" })); + return; + } + const result = store.unifiedRestoreArchived(parsed.sessionId, typeof parsed.archiveDir === "string" && parsed.archiveDir ? parsed.archiveDir : undefined); + log("info", `[admin] POST /__bili/storage/restore ${parsed.sessionId} → ${JSON.stringify(result)}`); + res.writeHead(result?.ok ? 200 : 404, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: result?.ok === true, error: result?.reason ?? (result ? undefined : "unified store unavailable") })); + return; + } catch (err) { + res.writeHead(err instanceof BodyTooLargeError ? 413 : 400, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: false, error: String(err) })); + return; + } + } // #2671 Phase 2: replica/checkpoint management over the unified store. // Loopback-gated like every /__bili/ route; 501 in legacy mode so the // operator knows the knob, not the URL, is what is missing. diff --git a/src/storage/ingest.ts b/src/storage/ingest.ts index 429a8a8f57..9b27270502 100644 --- a/src/storage/ingest.ts +++ b/src/storage/ingest.ts @@ -90,6 +90,31 @@ export const ingestSerializations = { count: 0 }; /** One messages-log entry with its ABSOLUTE seq already assigned (intern * assigns first-seen order; the delta planner continues after the trusted * prefix). */ +/** + * #3762 media CAS split: pull the top-level imageBase64 payload out of the + * serialized message. The stored content keeps the key POSITION but carries + * a {"$media": sha256(bytes)} reference; the bytes themselves live in the + * content CAS, refcounted via message_media rows. Key order is preserved by + * JSON.parse, so hydrate-and-restringify is byte-identical — every existing + * reader (getMessages hydrates before returning) sees the verbatim original. + * No media (or empty) → passthrough with the plain content hash. + */ +function splitMediaContent(content: string): { content: string; hash: string; media: { hash: string; bytes: string }[] } { + if (!content.includes("\"imageBase64\"")) return { content, hash: sha256Hex(content), media: [] }; + let parsed: { imageBase64?: unknown }; + try { + parsed = JSON.parse(content) as { imageBase64?: unknown }; + } catch { + return { content, hash: sha256Hex(content), media: [] }; + } + const bytes = parsed.imageBase64; + if (typeof bytes !== "string" || bytes.length === 0) return { content, hash: sha256Hex(content), media: [] }; + const mediaHash = sha256Hex(bytes); + parsed.imageBase64 = { $media: mediaHash }; + const split = JSON.stringify(parsed); + return { content: split, hash: sha256Hex(split), media: [{ hash: mediaHash, bytes }] }; +} + interface LogEntry { seq: number; msgId: string; @@ -98,6 +123,7 @@ interface LogEntry { role: string; kind: string; ref: string | null; + media?: { hash: string; bytes: string }[]; } /** Adapts any Session-shaped object (persist loadSync output) to the @@ -147,16 +173,17 @@ export function planIngest(session: LegacySessionLike): IngestPlan { const log: LogEntry[] = []; const seqByIdentity = new Map(); const intern = (m: CoreMessage): number => { - const content = JSON.stringify(m); + const raw = JSON.stringify(m); ingestSerializations.count++; - const parsed = JSON.parse(content) as { role: string; contentType?: string }; - const entry = { msgId: m.id, content, hash: sha256Hex(content), role: parsed.role, kind: parsed.contentType ?? "message" }; + const parsed = JSON.parse(raw) as { role: string; contentType?: string }; + const split = splitMediaContent(raw); + const entry = { msgId: m.id, content: split.content, hash: split.hash, role: parsed.role, kind: parsed.contentType ?? "message", media: split.media }; const key = `${m.id}\u0000${entry.hash}`; const existing = seqByIdentity.get(key); if (existing !== undefined) return existing; const seq = log.length; seqByIdentity.set(key, seq); - log.push({ seq, msgId: m.id, content: entry.content, hash: entry.hash, role: entry.role, kind: entry.kind, ref: byRawRef[m.id] ?? null }); + log.push({ seq, msgId: m.id, content: entry.content, hash: entry.hash, role: entry.role, kind: entry.kind, ref: byRawRef[m.id] ?? null, ...(entry.media.length > 0 ? { media: entry.media } : {}) }); return seq; }; const pluginSnapshotSeqs = session.pluginSnapshot ? session.pluginSnapshot.map(intern) : null; @@ -221,15 +248,18 @@ export function planIngestDelta( // IS the message the watermark was set from. const probe = JSON.stringify(tail[trustedLen - 1]); ingestSerializations.count++; - if (tail[trustedLen - 1].id !== boundary.msgId || sha256Hex(probe) !== boundary.contentHash) return null; + // #3762: the stored boundary hash covers the MEDIA-SPLIT serialization; + // split the probe the same way so image messages verify identically. + if (tail[trustedLen - 1].id !== boundary.msgId || splitMediaContent(probe).hash !== boundary.contentHash) return null; const byRawRef = session.state.messageRefs.byRaw; const log: LogEntry[] = []; const appended: number[] = []; for (let i = trustedLen; i < tail.length; i++) { const m = tail[i]; - const content = JSON.stringify(m); + const raw = JSON.stringify(m); ingestSerializations.count++; - const parsed = JSON.parse(content) as { role: string; contentType?: string }; + const parsed = JSON.parse(raw) as { role: string; contentType?: string }; + const split = splitMediaContent(raw); // Placeholder seqs under the aligned assumption (no rows above the // watermark); ingestSession renumbers from max(seq)+1 inside the // transaction when a fold left rows above the trusted tip. @@ -238,11 +268,12 @@ export function planIngestDelta( log.push({ seq, msgId: m.id, - content, - hash: sha256Hex(content), + content: split.content, + hash: split.hash, role: parsed.role, kind: parsed.contentType ?? "message", ref: byRawRef[m.id] ?? null, + ...(split.media.length > 0 ? { media: split.media } : {}), }); } // planIngest's snapshot ternary replicated on the ORIGINAL value: an @@ -284,6 +315,7 @@ function finishPlan( kind: e.kind, content: e.content, contentHash: e.hash, + ...(e.media && e.media.length > 0 ? { media: e.media } : {}), })); const blocks: BlockRow[] = session.state.blocks.map((b, ord) => { @@ -396,6 +428,7 @@ export function loadLegacyView( kstateJson: string; tailJson: string; ccrJson: string | null; + orphansJson?: string | null; }; messages: { seq: number; msgId: string; ref: string | null; content: string }[]; blocks: { blockId: string; summary: string; one: string | null; payloadJson: string }[]; @@ -469,6 +502,19 @@ export function loadLegacyView( full: JSON.parse(b.summary) as BlockView, }); } + // #3841 reap journal: journal'd orphan contents (reaped blocks) ride + // blockContents so the derived-decompress fallback survives restarts — + // WITHOUT resurrecting state.blocks (the reaped entries stay reaped). + if (s.orphansJson !== null && s.orphansJson !== undefined && s.orphansJson !== "") { + try { + const orphans = JSON.parse(s.orphansJson) as Record; + for (const [blockId, view] of Object.entries(orphans)) { + if (!blockContents.has(blockId)) blockContents.set(blockId, view); + } + } catch { + // corrupt journal → skip; the JSON envelope still carries the text + } + } const messagesBySeq = new Map(rows.messages.map((m) => [m.seq, JSON.parse(m.content) as CoreMessage])); const lastMessages = tail.tailSeqs diff --git a/src/storage/schema.ts b/src/storage/schema.ts index 3267fd460f..a354971ca2 100644 --- a/src/storage/schema.ts +++ b/src/storage/schema.ts @@ -63,7 +63,7 @@ // index. Both directions compatible: an older build opening a v5 // file no-ops at the version gate (extra column ignored); a newer // build upgrades a v4 file with one idempotent ALTER TABLE. -const STORAGE_SCHEMA_VERSION = 5; +const STORAGE_SCHEMA_VERSION = 7; /** #3250: cap for `journal_size_limit` (see STORAGE_PRAGMAS). Passive and * auto checkpoints move frames into the main db and reset the write offset @@ -87,6 +87,11 @@ export function storagePragmas(walJournalSizeLimitBytes: number): string[] { "PRAGMA busy_timeout = 5000", "PRAGMA foreign_keys = ON", `PRAGMA journal_size_limit = ${walJournalSizeLimitBytes}`, + // #3901/#3463: freed pages return to the pool incrementally so an + // archive/GC sweep actually shrinks the file going forward. Databases + // created before this pragma keep their original layout until a + // one-time full VACUUM (gc({vacuum:"full"}) / ?vacuum=full). + "PRAGMA auto_vacuum = INCREMENTAL", ]; } @@ -113,7 +118,8 @@ const STORAGE_DDL_V1 = `CREATE TABLE IF NOT EXISTS sessions ( kstate_json TEXT NOT NULL DEFAULT '{}', tail_json TEXT NOT NULL DEFAULT '{}', ccr_json TEXT, - last_ingested_seq INTEGER + last_ingested_seq INTEGER, + orphans_json TEXT ); CREATE TABLE IF NOT EXISTS content ( @@ -243,6 +249,17 @@ CREATE TABLE IF NOT EXISTS kv_store ( PRIMARY KEY (namespace, key) ) WITHOUT ROWID; +-- #3762 media CAS: one row per (message, media blob) reference. The message +-- content stores {"$media": hash} in the imageBase64 slot; the bytes live in +-- the content CAS, refcounted by these mapping rows. Swept with the session +-- (and on idempotent re-receipt undo, same discipline as messages.content). +CREATE TABLE IF NOT EXISTS message_media ( + session_id TEXT NOT NULL, + seq INTEGER NOT NULL, + media_hash TEXT NOT NULL REFERENCES content(hash), + PRIMARY KEY (session_id, seq, media_hash) +) WITHOUT ROWID; + CREATE INDEX IF NOT EXISTS idx_messages_session ON messages(session_id, seq); CREATE INDEX IF NOT EXISTS idx_messages_msg_id ON messages(session_id, msg_id); CREATE INDEX IF NOT EXISTS idx_blocks_session ON blocks(session_id, ord); @@ -290,6 +307,12 @@ export function migrateSchema(db: { db.exec("CREATE TABLE IF NOT EXISTS session_ext (\n session_id TEXT NOT NULL,\n key TEXT NOT NULL,\n value_json TEXT NOT NULL,\n updated_at INTEGER NOT NULL,\n PRIMARY KEY (session_id, key)\n ) WITHOUT ROWID"); db.exec("CREATE TABLE IF NOT EXISTS kv_store (\n namespace TEXT NOT NULL,\n key TEXT NOT NULL,\n value_json TEXT NOT NULL,\n updated_at INTEGER NOT NULL,\n PRIMARY KEY (namespace, key)\n ) WITHOUT ROWID"); } + if (from < 7) { + // v7 media CAS (#3762): message→media mapping table. Idempotent + // CREATE — pre-v7 rows keep their inline bytes and read back + // verbatim (lazy: no backfill rehash). + db.exec("CREATE TABLE IF NOT EXISTS message_media (\n session_id TEXT NOT NULL,\n seq INTEGER NOT NULL,\n media_hash TEXT NOT NULL REFERENCES content(hash),\n PRIMARY KEY (session_id, seq, media_hash)\n ) WITHOUT ROWID"); + } if (from < 5) { // v5 ingest watermark (#3133): additive NULLable column on // sessions. Probe first — same duplicate-column discipline as v2 @@ -300,6 +323,19 @@ export function migrateSchema(db: { db.exec("ALTER TABLE sessions ADD COLUMN last_ingested_seq INTEGER"); } } + if (from < 6) { + // v6 reap journal (#3841): nullable column holding the block + // contents of REAPED blocks (#2695 reap keeps blockContents + // while state.blocks loses the entry — the journal carries that + // content across restarts so reaped blocks never resurrect). + // NULL = the session has no reaped orphans (pre-v6 rows keep + // old behavior). + const cols = (db.prepare("PRAGMA table_info(sessions)").all() as unknown as { name?: unknown }[]) + .map((c) => (typeof c?.name === "string" ? c.name : "")); + if (!cols.includes("orphans_json")) { + db.exec("ALTER TABLE sessions ADD COLUMN orphans_json TEXT"); + } + } db.exec(`PRAGMA user_version = ${STORAGE_SCHEMA_VERSION}`); }); return { from, to: STORAGE_SCHEMA_VERSION }; diff --git a/src/storage/shadow.ts b/src/storage/shadow.ts index 47187de8f2..9362d6399f 100644 --- a/src/storage/shadow.ts +++ b/src/storage/shadow.ts @@ -23,6 +23,7 @@ import { log as loggerLog } from "../logger.js"; import { planIngest, planIngestDelta, loadLegacyView, completeBlockContents, type LegacySessionLike } from "./ingest.ts"; +import type { BlockView } from "../session.js"; import { resolveStorageMode, resolveStorageWalJournalSizeLimit, type StorageMode } from "../knobs.ts"; import { UnifiedStore, type CheckpointRow, type SessionSummaryRow, type WebSummaryRow } from "./store.ts"; import { canonicalize, sha256 } from "./canonical.ts"; @@ -195,6 +196,28 @@ export class UnifiedShadow { // skipping the mirror here is free; the JSON envelope stays // the superset on disk. if (orphanContents(loaded)) { + // #3841 reap journal: distinguish the REAL kernel reap shape + // (#2695: reaped block rows exist in the store but the + // incoming state no longer claims them) from never-ingested + // hand-built shapes. For reaps, drop exactly the reaped rows + // and journal the orphan contents — without this the rows + // resurrect into every viewFor rebuild after a restart and + // the next envelope save re-pollutes. Envelope stays the + // serving path either way (owner-pinned semantics). + const claimed = new Set(loaded.state.blocks.map((b) => b.blockId)); + const reapedIds = new Set(); + const orphans = new Map(); + try { + for (const row of this.store.getBlocks(loaded.id)) { + if (!claimed.has(row.blockId)) reapedIds.add(row.blockId); + } + for (const [blockId, view] of loaded.blockContents) { + if (!claimed.has(blockId)) orphans.set(blockId, view); + } + if (reapedIds.size > 0 || orphans.size > 0) this.store.reapBlocks(loaded.id, reapedIds, orphans); + } catch { + // journal failure must not break the save path + } this.stats.orphanSkips++; return; } @@ -447,6 +470,15 @@ export class UnifiedShadow { * SQL instead of walking + parsing every legacy file. */ /** #3248: raw canonical rows for the incremental lane (core-form * message JSON in seq order — exactly what planIngest interned). */ + /** #3901/#3463: pass-through GC + restore (see UnifiedStore.gc). */ + gc(opts: { archiveAfterDays: number; archiveDir: string; vacuum?: "full" | "incremental" }): { archived: number; contentRowsDeleted: number } { + return this.store.gc(opts); + } + + restoreArchived(sessionId: string, opts: { archiveDir: string }): { ok: boolean; reason?: string } { + return this.store.restoreArchived(sessionId, opts); + } + rawMessages(sessionId: string): { seq: number; content: string }[] { this.assertLive(); return this.store.getMessages(sessionId).map((m) => ({ seq: m.seq, content: m.content })); diff --git a/src/storage/store.ts b/src/storage/store.ts index fd4feecb2c..92537a6658 100644 --- a/src/storage/store.ts +++ b/src/storage/store.ts @@ -17,8 +17,8 @@ */ import { createHash } from "node:crypto"; -import { mkdirSync } from "node:fs"; -import { dirname } from "node:path"; +import { mkdirSync, writeFileSync, readFileSync, existsSync } from "node:fs"; +import { dirname, join } from "node:path"; import { log as loggerLog } from "../logger.js"; import { openSqliteDatabase, type SqliteDatabase, type SqliteEngineName, type SqliteEngineSource } from "./driver.js"; import { migrateSchema, storagePragmas, WAL_JOURNAL_SIZE_LIMIT } from "./schema.js"; @@ -73,6 +73,10 @@ export interface SessionRow { * fully). Computed by ingestSession only (delta provenance or full-path * alignment) — never by planners or callers; read via ingestWatermark(). */ lastIngestedSeq?: number | null; + /** #3841 reap journal: contents of reaped blocks (state.blocks lost + * the entry, blockContents keeps the text). Served via loadLegacyView + * merge into blockContents; never resurrects state.blocks. */ + orphansJson?: string | null; } export interface MessageRow { @@ -90,6 +94,10 @@ export interface MessageRow { * lets ingestSession skip re-hashing every message on every save. * Identical value to sha256Hex(content); optional for direct callers. */ contentHash?: string; + /** #3762: media payloads split out of `content` (bytes live in the + * content CAS, refcounted via message_media rows). Ingest-only — + * readers always receive hydrated content. */ + media?: { hash: string; bytes: string }[]; } export interface BlockRow { @@ -447,7 +455,17 @@ export class UnifiedStore { for (const row of staleRows) { if (incomingIds.has(`${row.msg_id}\u0000${row.content_hash}`)) continue; this.releaseContent(row.content_hash); + // #3762 review (#3901): media mappings ride the message + // sweep — release each referenced media hash and drop + // the rows with their message, or the CAS bytes leak + // forever (deleteSession/archiveSession already do). + for (const mm of this.db + .prepare("SELECT media_hash FROM message_media WHERE session_id = ? AND seq = ?") + .all(s.sessionId, row.seq) as { media_hash: string }[]) { + this.releaseContent(mm.media_hash); + } this.db.prepare("DELETE FROM messages WHERE session_id = ? AND seq = ?").run(s.sessionId, row.seq); + this.db.prepare("DELETE FROM message_media WHERE session_id = ? AND seq = ?").run(s.sessionId, row.seq); } } } @@ -525,6 +543,18 @@ export class UnifiedStore { // Idempotent re-receipt: the CAS row is still referenced // by the existing message row — undo our refcount bump. this.releaseContent(hash); + } else if (m.media && m.media.length > 0) { + // #3762: put the media bytes + mapping rows; refcount per + // message row that actually landed (INSERT OR IGNORE on + // the mapping keeps re-ingest of the same (seq, hash) + // from double-bumping). + for (const md of m.media) { + this.putContent(md.bytes, md.hash); + const mm = this.db + .prepare("INSERT OR IGNORE INTO message_media (session_id, seq, media_hash) VALUES (?, ?, ?)") + .run(s.sessionId, m.seq, md.hash); + if (mm.changes === 0) this.releaseContent(md.hash); + } } } @@ -815,12 +845,58 @@ export class UnifiedStore { * of the session: PINNED ones survive (their state blob keeps its * refcount — replica GC protection, #2671 Phase 2); unpinned ones are * swept here together with the session. */ + /** #3841 reap journal: targeted update for the orphan shape (#2695 reap — + * state.blocks loses the entry, blockContents keeps the text). The full + * ingest skips orphan-shaped records (envelope-served, owner-pinned + * semantics: imports 0, never gutted into the index), which used to + * leave the REAPED block rows in the table — a restart's viewFor rebuild + * resurrected them and the next save polluted the envelope. Instead: + * drop exactly the reaped rows (releasing their content refs, + * deleteSession discipline) and journal the orphan contents into + * sessions.orphans_json so loadLegacyView can rebuild blockContents + * without resurrecting state.blocks. Idempotent; unknown ids no-op. */ + reapBlocks(sessionId: string, reapedIds: Set, orphanContents: Map): void { + if (reapedIds.size === 0 && orphanContents.size === 0) return; + this.db.transaction(() => { + for (const blockId of reapedIds) { + const row = this.db + .prepare("SELECT summary_hash, one_hash FROM blocks WHERE session_id = ? AND block_id = ?") + .get(sessionId, blockId) as { summary_hash: string; one_hash: string | null } | undefined; + if (!row) continue; + this.releaseContent(row.summary_hash); + if (row.one_hash !== null) this.releaseContent(row.one_hash); + this.db.prepare("DELETE FROM blocks WHERE session_id = ? AND block_id = ?").run(sessionId, blockId); + } + if (orphanContents.size > 0) { + const sess = this.db + .prepare("SELECT orphans_json FROM sessions WHERE session_id = ?") + .get(sessionId) as { orphans_json: string | null } | undefined; + if (sess) { + let merged: Record = {}; + try { + const parsed = JSON.parse(sess.orphans_json ?? "{}") as unknown; + if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) merged = parsed as Record; + } catch { + // corrupt journal → start fresh rather than losing the reap + } + for (const [blockId, view] of orphanContents) merged[blockId] = view; + this.db.prepare("UPDATE sessions SET orphans_json = ? WHERE session_id = ?").run(JSON.stringify(merged), sessionId); + } + } + }); + } + deleteSession(sessionId: string): void { this.db.transaction(() => { const refs: string[] = []; for (const m of this.db .prepare("SELECT content_hash FROM messages WHERE session_id = ?") .all(sessionId) as { content_hash: string }[]) refs.push(m.content_hash); + // #3762: media mappings ride the message sweep — one release per + // mapping row, then the rows go. + for (const mm of this.db + .prepare("SELECT media_hash FROM message_media WHERE session_id = ?") + .all(sessionId) as { media_hash: string }[]) refs.push(mm.media_hash); for (const b of this.db .prepare("SELECT summary_hash, one_hash FROM blocks WHERE session_id = ?") .all(sessionId) as { summary_hash: string; one_hash: string | null }[]) { @@ -839,6 +915,7 @@ export class UnifiedStore { refs.push(cp.state_hash); } this.db.prepare("DELETE FROM messages WHERE session_id = ?").run(sessionId); + this.db.prepare("DELETE FROM message_media WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM blocks WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM dead_refs WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM refs WHERE session_id = ?").run(sessionId); @@ -852,6 +929,171 @@ export class UnifiedStore { }); } + /** #3901/#3463: archive one session. The full row bodies (messages, + * blocks, refs, dead refs, ccr, session_ext, unpinned checkpoints) are + * exported verbatim — content bodies included — to + * `/.json`, then swept with the deleteSession + * refcount discipline. Unlike deleteSession the sessions row survives + * as a tombstone (status 'archived', label intact — the web list keeps + * showing it) and the conversations identity binding survives (identity + * is forever; a restore must land on the same session id). Pinned + * checkpoints keep their rows + refs. Idempotent: an already-archived + * session is a no-op returning the existing export path. */ + archiveSession(sessionId: string, opts: { archiveDir: string }): { ok: boolean; exportPath?: string; reason?: string; contentRowsDeleted?: number } { + const row = this.getSessionRow(sessionId); + if (!row) return { ok: false, reason: "not-found" }; + if (row.status === "archived") return { ok: true, exportPath: join(opts.archiveDir, `${sessionId}.json`) }; + // --- read everything the session owns (before any mutation) --- + const messages = (this.db.prepare("SELECT seq, msg_id, ref, ref_num, role, kind, content_hash FROM messages WHERE session_id = ? ORDER BY seq").all(sessionId) as Record[]); + const mediaMaps = (this.db.prepare("SELECT seq, media_hash FROM message_media WHERE session_id = ?").all(sessionId) as Record[]); + const blocks = (this.db.prepare("SELECT block_id, ord, start_seq, end_seq, start_ref, end_ref, tier, summary_hash, one_hash, supersedes_block_id, active, created_at, covered_hash, payload_json FROM blocks WHERE session_id = ? ORDER BY ord").all(sessionId) as Record[]); + const refsRows = (this.db.prepare("SELECT ord, raw_id, ref, ref_num FROM refs WHERE session_id = ? ORDER BY ord").all(sessionId) as Record[]); + const deadRefs = (this.db.prepare("SELECT ord, ref, ref_num, recorded_at FROM dead_refs WHERE session_id = ? ORDER BY ord").all(sessionId) as Record[]); + const ccr = (this.db.prepare("SELECT ord, ccr_key, content_hash FROM ccr_entries WHERE session_id = ? ORDER BY ord").all(sessionId) as Record[]); + const ext = (this.db.prepare("SELECT key, value_json FROM session_ext WHERE session_id = ?").all(sessionId) as Record[]); + const checkpoints = (this.db.prepare("SELECT cp_id, seq_at, block_set_hash, kind, created_at, pinned, label, state_hash FROM checkpoints WHERE session_id = ?").all(sessionId) as Record[]); + // --- collect the distinct CAS bodies referenced by those rows --- + const wanted = new Set(); + for (const m of messages) wanted.add(m.content_hash as string); + for (const mm of mediaMaps) wanted.add(mm.media_hash as string); + for (const b of blocks) { wanted.add(b.summary_hash as string); if (b.one_hash !== null && b.one_hash !== undefined) wanted.add(b.one_hash as string); } + for (const e of ccr) wanted.add(e.content_hash as string); + for (const cp of checkpoints) if (cp.pinned === 0) wanted.add(cp.state_hash as string); + const content: Record = {}; + for (const hash of wanted) { + const c = this.db.prepare("SELECT bytes FROM content WHERE hash = ?").get(hash) as { bytes: Buffer | Uint8Array } | undefined; + if (c) content[hash as string] = Buffer.from(c.bytes).toString("utf8"); + } + const exportPath = join(opts.archiveDir, `${sessionId}.json`); + mkdirSync(opts.archiveDir, { recursive: true }); + writeFileSync(exportPath, JSON.stringify({ version: 1, archivedAt: Date.now(), session: row, content, messages, mediaMaps, blocks, refs: refsRows, deadRefs, ccrEntries: ccr, sessionExt: ext, checkpoints })); + // --- sweep the row bodies (refcount discipline; pinned checkpoints survive) --- + let contentRowsDeleted = 0; + this.db.transaction(() => { + const rel: string[] = []; + for (const m of messages) rel.push(m.content_hash as string); + for (const mm of mediaMaps) rel.push(mm.media_hash as string); + for (const b of blocks) { rel.push(b.summary_hash as string); if (b.one_hash !== null && b.one_hash !== undefined) rel.push(b.one_hash as string); } + for (const e of ccr) rel.push(e.content_hash as string); + for (const cp of checkpoints) if (cp.pinned === 0) { rel.push(cp.state_hash as string); this.db.prepare("DELETE FROM checkpoints WHERE cp_id = ?").run(cp.cp_id); } + this.db.prepare("DELETE FROM messages WHERE session_id = ?").run(sessionId); + this.db.prepare("DELETE FROM message_media WHERE session_id = ?").run(sessionId); + this.db.prepare("DELETE FROM blocks WHERE session_id = ?").run(sessionId); + this.db.prepare("DELETE FROM dead_refs WHERE session_id = ?").run(sessionId); + this.db.prepare("DELETE FROM refs WHERE session_id = ?").run(sessionId); + this.db.prepare("DELETE FROM ccr_entries WHERE session_id = ?").run(sessionId); + this.db.prepare("DELETE FROM session_ext WHERE session_id = ?").run(sessionId); + this.db.prepare("UPDATE sessions SET status = 'archived' WHERE session_id = ?").run(sessionId); + for (const hash of rel) this.releaseContent(hash); + const zero = this.db.prepare("SELECT COUNT(*) AS n FROM content WHERE refcount <= 0").get() as { n: number }; + contentRowsDeleted = zero.n; + this.db.prepare("DELETE FROM content WHERE refcount <= 0").run(); + }); + loggerLog("info", `[#3901] archived ${sessionId} → ${exportPath} (${contentRowsDeleted} content rows collected)`); + return { ok: true, exportPath, contentRowsDeleted }; + } + + /** #3901/#3463: byte-exact restore from an archive export. Re-puts the + * content bodies (same sha256 → same hashes), re-inserts every row, + * flips the tombstone back to 'active'. The conversations binding + * re-claims ON CONFLICT DO NOTHING (first claimant wins; the binding + * never left, so this is normally a no-op). */ + restoreArchived(sessionId: string, opts: { archiveDir: string }): { ok: boolean; reason?: string } { + const exportPath = join(opts.archiveDir, `${sessionId}.json`); + if (!existsSync(exportPath)) return { ok: false, reason: "export-missing" }; + let dump: { + session: ReturnType; + content: Record; + messages: Record[]; + blocks: Record[]; + refs: Record[]; + deadRefs: Record[]; + ccrEntries: Record[]; + sessionExt: Record[]; + checkpoints: Record[]; + }; + try { + dump = JSON.parse(readFileSync(exportPath, "utf8")) as typeof dump; + } catch { + return { ok: false, reason: "export-corrupt" }; + } + if (!dump.session || dump.session.sessionId !== sessionId) return { ok: false, reason: "export-mismatch" }; + const sess: NonNullable = dump.session; + this.db.transaction(() => { + for (const [hash, body] of Object.entries(dump.content)) this.putContent(Buffer.from(body, "utf8"), hash); + const insMsg = this.db.prepare("INSERT INTO messages (session_id, seq, msg_id, ref, ref_num, role, kind, content_hash) VALUES (?, ?, ?, ?, ?, ?, ?, ?)"); + for (const m of dump.messages) insMsg.run(sessionId, m.seq, m.msg_id, m.ref ?? null, m.ref_num ?? null, m.role, m.kind ?? "message", m.content_hash); + const insMedia = this.db.prepare("INSERT OR IGNORE INTO message_media (session_id, seq, media_hash) VALUES (?, ?, ?)"); + for (const mm of (dump as { mediaMaps?: Record[] }).mediaMaps ?? []) insMedia.run(sessionId, mm.seq, mm.media_hash); + const insBlock = this.db.prepare(`INSERT INTO blocks (block_id, session_id, ord, start_seq, end_seq, start_ref, end_ref, tier, summary_hash, one_hash, supersedes_block_id, active, created_at, covered_hash, payload_json) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(session_id, block_id) DO NOTHING`); + for (const b of dump.blocks) insBlock.run(b.block_id, sessionId, b.ord, b.start_seq, b.end_seq, b.start_ref ?? null, b.end_ref ?? null, b.tier, b.summary_hash, b.one_hash ?? null, b.supersedes_block_id ?? null, b.active, b.created_at, b.covered_hash, b.payload_json); + const insRef = this.db.prepare("INSERT INTO refs (session_id, ord, raw_id, ref, ref_num) VALUES (?, ?, ?, ?, ?)"); + for (const r of dump.refs) insRef.run(sessionId, r.ord, r.raw_id, r.ref, r.ref_num ?? null); + const insDead = this.db.prepare("INSERT INTO dead_refs (session_id, ord, ref, ref_num, recorded_at) VALUES (?, ?, ?, ?, ?)"); + for (const d of dump.deadRefs) insDead.run(sessionId, d.ord, d.ref, d.ref_num, d.recorded_at); + const insCcr = this.db.prepare("INSERT INTO ccr_entries (session_id, ord, ccr_key, content_hash) VALUES (?, ?, ?, ?)"); + for (const e of dump.ccrEntries) insCcr.run(sessionId, e.ord, e.ccr_key, e.content_hash); + const insExt = this.db.prepare("INSERT INTO session_ext (session_id, key, value_json) VALUES (?, ?, ?)"); + for (const x of dump.sessionExt) insExt.run(sessionId, x.key, x.value_json); + const insCp = this.db.prepare("INSERT INTO checkpoints (cp_id, session_id, seq_at, block_set_hash, kind, created_at, pinned, label, state_hash) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(cp_id) DO NOTHING"); + for (const cp of dump.checkpoints) insCp.run(cp.cp_id, sessionId, cp.seq_at, cp.block_set_hash, cp.kind, cp.created_at, cp.pinned, cp.label ?? null, cp.state_hash); + this.db + .prepare(`UPDATE sessions SET parent_session_id = ?, fork_point_seq = ?, protocol = ?, label = ?, title = ?, host_title = ?, namespace = ?, origin = ?, + created_at = ?, updated_at = ?, next_block_id = ?, next_run_id = ?, meta_json = ?, stats_json = ?, metadata_json = ?, kstate_json = ?, tail_json = ?, ccr_json = ?, last_ingested_seq = ?, orphans_json = ?, status = 'active' WHERE session_id = ?`) + .run( + sess.parentSessionId ?? null, sess.forkPointSeq ?? null, sess.protocol ?? null, sess.label ?? null, + sess.title ?? null, sess.hostTitle ?? null, sess.namespace ?? "proxy", sess.origin ?? null, + sess.createdAt, sess.updatedAt, sess.nextBlockId, sess.nextRunId, + sess.metaJson, sess.statsJson, sess.metadataJson, sess.kstateJson, sess.tailJson, sess.ccrJson ?? null, + sess.lastIngestedSeq ?? null, sess.orphansJson ?? null, sessionId, + ); + }); + loggerLog("info", `[#3901] restored archived session ${sessionId} from ${exportPath}`); + return { ok: true }; + } + + /** #3901/#3463: age-sweep GC. Sessions whose updated_at is older than + * the retention cutoff are archived (export + sweep); zero-ref content + * rows left by any path are collected. `vacuum: 'full'` additionally + * rebuilds the database file (auto_vacuum=INCREMENTAL handles freed + * pages incrementally; a one-time VACUUM reclaims the initial bloat of + * databases created before that pragma). */ + gc(opts: { archiveAfterDays: number; archiveDir: string; now?: number; vacuum?: "full" | "incremental" }): { archived: number; contentRowsDeleted: number } { + const now = opts.now ?? Date.now(); + let archived = 0; + let contentRowsDeleted = 0; + if (opts.archiveAfterDays > 0) { + const cutoff = now - opts.archiveAfterDays * 86_400_000; + const stale = this.db.prepare("SELECT session_id FROM sessions WHERE status = 'active' AND updated_at < ?").all(cutoff) as { session_id: string }[]; + for (const s of stale) { + const r = this.archiveSession(s.session_id, { archiveDir: opts.archiveDir }); + if (r.ok) archived++; + if (r.contentRowsDeleted) contentRowsDeleted += r.contentRowsDeleted; + } + } + const zero = this.db.prepare("SELECT COUNT(*) AS n FROM content WHERE refcount <= 0").get() as { n: number }; + if (zero.n > 0) { + this.db.prepare("DELETE FROM content WHERE refcount <= 0").run(); + contentRowsDeleted += zero.n; + } + if (opts.vacuum === "full") { + loggerLog("info", "[#3901] gc: running full VACUUM (this can take a while on large databases)"); + this.db.exec("VACUUM"); + } else if (opts.vacuum === "incremental") { + this.db.exec("PRAGMA incremental_vacuum"); + } + return { archived, contentRowsDeleted }; + } + + /** Test/ops helper: refcount of the content row whose serialized body + * contains `needle` (archive-gc tests use unique body texts; the CAS + * body is the full message JSON, so prefix matching would miss). */ + contentRefCountOfBody(needle: string): number { + const row = this.db.prepare("SELECT refcount FROM content WHERE CAST(bytes AS TEXT) LIKE '%' || ? || '%'").get(needle) as { refcount: number } | undefined; + return row ? row.refcount : 0; + } + /** Freezes one session record as a checkpoint row + CAS blob. The row * owns exactly one refcount on the blob; identical consecutive states * dedup to a single CAS entry (the CoW behind cheap repeated @@ -1258,7 +1500,7 @@ export class UnifiedStore { .prepare( `SELECT session_id, parent_session_id, fork_point_seq, protocol, label, title, host_title, namespace, origin, status, created_at, updated_at, next_block_id, next_run_id, - meta_json, stats_json, metadata_json, kstate_json, tail_json, ccr_json, last_ingested_seq + meta_json, stats_json, metadata_json, kstate_json, tail_json, ccr_json, last_ingested_seq, orphans_json FROM sessions WHERE session_id = ?`, ) .get(sessionId) as Record | undefined; @@ -1285,6 +1527,7 @@ export class UnifiedStore { tailJson: row.tail_json as string, ccrJson: (row.ccr_json as string | null) ?? null, lastIngestedSeq: (row.last_ingested_seq as number | null | undefined) ?? null, + orphansJson: (row.orphans_json as string | null | null | undefined) ?? null, }; } @@ -1300,10 +1543,40 @@ export class UnifiedStore { role: r.role as string, kind: r.kind as string, contentHash: r.content_hash as string, - content: this.getContent(r.content_hash as string) ?? "", + content: this.hydrateMedia(this.getContent(r.content_hash as string) ?? ""), })); } + /** #3762: swap {"$media": hash} references back to the CAS bytes. Only + * touches content that carries the marker; key order is preserved by + * parse/stringify so the result is byte-identical to the pre-split + * serialization. A missing CAS row (corruption / manual surgery) falls + * back to the split form rather than throwing — visible, not silent. */ + private hydrateMedia(content: string): string { + if (!content.includes("\"$media\"")) return content; + try { + const parsed = JSON.parse(content) as { imageBase64?: unknown }; + const slot = parsed.imageBase64; + if (slot !== null && typeof slot === "object" && typeof (slot as { $media?: unknown }).$media === "string") { + const bytes = this.getContent((slot as { $media: string }).$media); + if (typeof bytes === "string" && bytes.length > 0) { + parsed.imageBase64 = bytes; + return JSON.stringify(parsed); + } + } + return content; + } catch { + return content; + } + } + + /** #3762 test/ops helper: refcount of the content row holding exactly + * these media bytes. */ + mediaRefCount(bytes: string): number { + const row = this.db.prepare("SELECT refcount FROM content WHERE CAST(bytes AS TEXT) = ?").get(bytes) as { refcount: number } | undefined; + return row ? row.refcount : 0; + } + getBlocks(sessionId: string): (BlockRow & { summaryHash: string; oneHash: string | null })[] { const rows = this.db .prepare( diff --git a/tests/dsh-authored-compaction.test.ts b/tests/dsh-authored-compaction.test.ts index 37f0a4f952..2dcfa97a05 100644 --- a/tests/dsh-authored-compaction.test.ts +++ b/tests/dsh-authored-compaction.test.ts @@ -348,7 +348,9 @@ test("e2e: stream:false honored — JSON body with the checkpoint text", async ( test("e2e: knob off → the #1729 refusal still stands (status quo pinned)", async () => { await withHarness(async (h) => { await armBlock(h, "s-off"); - delete process.env.BILI_DSH_AUTHORED_COMPACTION; + // #3901 flip: authored compaction is ON by default since 0.1.203's + // dogfood window — the off leg must pin the explicit escape hatch. + process.env.BILI_DSH_AUTHORED_COMPACTION = "0"; const r = await turn(h, "s-off", [ { role: "user", content: `${DSH_COMPACTION_INSTRUCTION_PREFIX} Condense.` }, ]); diff --git a/tests/inc-canonical-pointers.test.ts b/tests/inc-canonical-pointers.test.ts new file mode 100644 index 0000000000..8b8bb370d2 --- /dev/null +++ b/tests/inc-canonical-pointers.test.ts @@ -0,0 +1,165 @@ +import assert from "node:assert/strict"; +import http from "node:http"; +import { once } from "node:events"; +import test from "node:test"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { defaultConfig } from "acp-kernel"; +import { startServer } from "../src/server.ts"; +import type { ProxyOptions } from "../src/config.ts"; +import { SessionStore, _setStoreForTest, getStore } from "../src/persist.ts"; +import { _resetPluginStateForTest } from "../src/plugin.ts"; +import { _resetSessionsForTest } from "../src/session.ts"; +import { _setForTest } from "../src/registry.ts"; +import { resetToolRingForTest } from "../src/tool-ring.ts"; +import { __resetIncrementalLaneForTests, __incrementalPtrChainForTests } from "../src/incremental-lane.ts"; + +/** + * #3901 / #3798 — incremental canonical row pointers. + * + * readCanonicalFromStore used the base-seq COUNT cutoff, but the unified + * messages table reuses rows for identical (msg_id, content_hash) + * (remapPlanToAbsolute) — a resync that shares history with the rejected + * branch maps onto rows BELOW the cutoff, so a cold rebuild lost the shared + * prefix and served serverSeq 0/1 → 409 reseed loop. Fix: persist the + * ordered canonical row POINTERS (inc:canonical-ptrs) after every raw + * ingest; cold reads fetch rows by pointer; malformed pointers force an + * honest resync instead of a partial history. + */ + +process.env.NODE_ENV = "test"; +const testRoot = mkdtempSync(join(tmpdir(), "bili-inc-ptrs-")); +test.after(() => rmSync(testRoot, { recursive: true, force: true })); + +async function harness(): Promise<{ + proxyUrl: string; + turn: (t: { sinceSeq: number; messages: unknown[]; resync?: boolean }) => Promise<{ status: number; body: unknown; appliedSeq?: number }>; + forwarded: Record[]; + close: () => Promise; +}> { + const dir = mkdtempSync(join(testRoot, "run-")); + for (const key of ["XDG_STATE_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME", "XDG_CONFIG_HOME"]) process.env[key] = dir; + process.env.BILI_STORAGE_UNIFIED = "full"; + _resetSessionsForTest(); + _resetPluginStateForTest(); + resetToolRingForTest(); + __resetIncrementalLaneForTests(); + const store = new SessionStore({ dir: dir + "/sessions", enabled: true, debounceMs: 0 }); + _setStoreForTest(store); + _setForTest({}); + const forwarded: Record[] = []; + const upstream = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (chunk: Buffer) => chunks.push(chunk)); + req.on("end", () => { + forwarded.push(JSON.parse(Buffer.concat(chunks).toString("utf8")) as Record); + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ id: "msg_x", role: "assistant", content: [{ type: "text", text: "answer" }], usage: { input_tokens: 100, output_tokens: 1 } })); + }); + }); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const upstreamUrl = `http://127.0.0.1:${upstream.address() instanceof Object ? (upstream.address() as { port: number }).port : 0}`; + const proxy = await startServer({ port: 0, host: "127.0.0.1", upstream: upstreamUrl, routes: { [upstreamUrl]: { models: { "claude-test": { context: 400000 } } } }, modelContextLimit: 400000, kernelConfig: defaultConfig(400000), compress: { injectTool: true, injectNudge: false }, promptCache: { routing: "auto" }, sessionHeader: "x-acp-session", log: false, debug: false, passthrough: false, autoUpdate: false, mitm: { enabled: false, domains: [] } } as ProxyOptions); + if (!(proxy as unknown as { listening?: boolean }).listening) await once(proxy, "listening"); + const proxyUrl = `http://127.0.0.1:${(proxy.address() as { port: number }).port}`; + const turn = async (t: { sinceSeq: number; messages: unknown[]; resync?: boolean }) => { + const response = await fetch(`${proxyUrl}/__bili/plugin/turn`, { + method: "POST", + headers: { "content-type": "application/json", "x-bili-plugin": "dsh", "x-bili-plugin-conversation": "conv-ptrs" }, + body: JSON.stringify({ conversationId: "conv-ptrs", sinceSeq: t.sinceSeq, ...(t.resync ? { resync: true } : {}), path: "/v1/messages", request: { model: "claude-test", max_tokens: 1024 }, messages: t.messages }), + }); + const body: unknown = await response.json().catch(() => null); + const applied = response.headers.get("x-bili-applied-seq"); + return { status: response.status, body, appliedSeq: applied !== null && /^\d+$/.test(applied) ? Number(applied) : undefined }; + }; + return { + proxyUrl, + turn, + forwarded, + close: async () => { + await new Promise((resolve) => proxy.close(() => resolve())); + store.closeQuietly(); + await new Promise((resolve) => upstream.close(() => resolve())); + }, + }; +} + +const core = (id: string, text: string): Record => ({ id, role: "user", contentType: "text", content: text.repeat(200) }); + +test("cold canonical rebuild: identical resync keeps the shared prefix; delta assembles the FULL history", async () => { + const h = await harness(); + try { + const seed = await h.turn({ sinceSeq: 0, resync: true, messages: [core("u1", "one"), core("u2", "two"), core("u3", "three")] }); + assert.equal(seed.status, 200); + assert.equal(seed.appliedSeq, 3); + + // Identical-history resync (#3798's killer shape): the branch shares + // ALL rows with the previous one — remap reuses them below any count + // cutoff. + const resync = await h.turn({ sinceSeq: 0, resync: true, messages: [core("u1", "one"), core("u2", "two"), core("u3", "three")] }); + assert.equal(resync.status, 200); + assert.equal(resync.appliedSeq, 3); + await __incrementalPtrChainForTests("conv-ptrs"); + + // Cold: drop the resident caches, keep SQLite. + __resetIncrementalLaneForTests(); + + const delta = await h.turn({ sinceSeq: 3, messages: [core("u4", "four")] }); + assert.equal(delta.status, 200, "cold rebuild must serve the shared prefix instead of 409-reseed"); + assert.equal(delta.appliedSeq, 4); + const upstreamBody = h.forwarded.at(-1) as { messages: { content?: { text?: string }[] }[] }; + const texts = JSON.stringify(upstreamBody.messages); + let cursor = -1; + for (const expected of ["one", "two", "three", "four"]) { + const at = texts.indexOf(expected.repeat(200), cursor + 1); + assert.ok(at > cursor, `assembled history = shared prefix + delta in order (${expected} missing or out of order)`); + cursor = at; + } + } finally { + await h.close(); + } +}); + +test("cold canonical rebuild: edited-tail resync keeps the shared prefix, drops the replaced branch", async () => { + const h = await harness(); + try { + await h.turn({ sinceSeq: 0, resync: true, messages: [core("u1", "one"), core("u2", "two")] }); + // Edit the last message → resync [u1, u2']. Let the turn's pointer + // chain settle first — a real client's next turn would await it + // server-side; only a crash inside the window could observe the + // one-turn-stale pointers. + await h.turn({ sinceSeq: 0, resync: true, messages: [core("u1", "one"), core("u2", "two-EDITED")] }); + await __incrementalPtrChainForTests("conv-ptrs"); + __resetIncrementalLaneForTests(); + const delta = await h.turn({ sinceSeq: 2, messages: [core("u3", "three")] }); + assert.equal(delta.status, 200); + const upstreamBody = h.forwarded.at(-1) as { messages: { id?: string; content?: unknown }[] }; + const texts = JSON.stringify(upstreamBody.messages); + assert.ok(texts.includes("two-EDITED"), "current branch served"); + assert.ok(!texts.includes("two-EDITED".slice(0, 3) + "two"), "sanity"); + } finally { + await h.close(); + } +}); + +test("malformed pointers: honest resync, never a partial history", async () => { + const h = await harness(); + try { + await h.turn({ sinceSeq: 0, resync: true, messages: [core("u1", "one"), core("u2", "two")] }); + // Corrupt the pointer list to reference a row that does not exist. + const store = getStore(); + store.setSessionExt("conv-ptrs", "inc:canonical-ptrs", [424242]); + assert.deepEqual(store.getSessionExt("conv-ptrs", "inc:canonical-ptrs"), [424242]); + __resetIncrementalLaneForTests(); + const before = h.forwarded.length; + const r = await h.turn({ sinceSeq: 2, messages: [core("u3", "three")] }); + // Either an honest 409 (forcing client resync) or a full resync — + // but the upstream must NEVER see a truncated [u3]-only history. + const served = h.forwarded.length > before ? (h.forwarded.at(-1) as { messages: unknown[] }).messages.length : 0; + assert.ok(r.status === 409 || served >= 3, `no partial history (status=${r.status}, served=${served})`); + } finally { + await h.close(); + } +}); diff --git a/tests/incremental-lane.test.ts b/tests/incremental-lane.test.ts index cf71b9edf2..4a48b5bcdd 100644 --- a/tests/incremental-lane.test.ts +++ b/tests/incremental-lane.test.ts @@ -201,7 +201,7 @@ test("incremental lane: divergent resync survives a cold restart without a 409 l _resetSessionsForTest(); __resetIncrementalLaneForTests(); const delta = await h.turn({ sinceSeq: 1, messages: [assistantTurn(1), userTurn(2, "new branch tail")] }); - assert.equal(delta.status, 200); + assert.equal(delta.status, 200, `delta body: ${JSON.stringify(delta.body)}`); assert.equal(delta.headers.get("x-bili-applied-seq"), "3"); const upstream = h.forwarded.at(-1) as { messages: { role: string; content: unknown }[] }; assert.equal(upstream.messages.length, 3, "served history is the edited branch, not the row union"); diff --git a/tests/media-cas.test.ts b/tests/media-cas.test.ts new file mode 100644 index 0000000000..dea8486d45 --- /dev/null +++ b/tests/media-cas.test.ts @@ -0,0 +1,145 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { planIngest, loadLegacyView, type LegacySessionLike } from "../src/storage/ingest.ts"; +import { UnifiedStore } from "../src/storage/store.ts"; + +/** + * #3901 / #3762 — media-level content addressing. + * + * The same image posted into two conversations (or twice into one) is stored + * once: ingest splits media sidecars (imageBase64) out of the message JSON + * into content-CAS media rows; the stored message content keeps the key + * POSITION and carries a media reference value; getMessages() hydrates the + * bytes back so every reader sees the ORIGINAL verbatim JSON (byte-identical + * round-trip). Deleting the last referencing session releases the media row. + */ + +process.env.BILI_PERSIST_ZSTD = "0"; +process.env.BILI_STORAGE_UNIFIED = "full"; +process.env.BILI_CONFIG_FILE = path.join( + fs.mkdtempSync(path.join(os.tmpdir(), "bili-media-cas-")), + "no-config.json", +); + +const IMG_B64 = "a".repeat(4096); // stand-in for base64 image bytes + +interface CoreMsg { id: string; role: string; contentType: string; text: string; imageBase64?: string; imageMediaType?: string } + +const mkMsg = (id: string, text: string, image?: string): CoreMsg => ({ + id, + role: "user", + contentType: image !== undefined ? "image" : "text", + text, + ...(image !== undefined ? { imageBase64: image, imageMediaType: "image/png" } : {}), +}); + +const mkSession = (id: string, msgs: CoreMsg[]): LegacySessionLike => ({ + id, + createdAt: 1_000, + lastSeen: 2_000, + meta: { protocol: "openai" }, + stats: {}, + metadata: {}, + state: { blocks: [], nextBlockId: 1, nextRunId: 1, messageRefs: { byRaw: {}, byRef: {} } }, + blockContents: new Map(), + lastMessages: msgs as unknown as LegacySessionLike["lastMessages"], + lastMessagesFolded: false, + pluginSnapshot: undefined, + contentStore: undefined, +} as unknown as LegacySessionLike); + +test("media CAS: same image in two sessions stored once; round-trip verbatim; release on delete", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "storage-media-cas-")); + try { + const store = UnifiedStore.open(path.join(dir, "index.db")); + const mA = mkMsg("m-a", "look at this in session A", IMG_B64); + const mB = mkMsg("m-b", "same screenshot in session B", IMG_B64); + const mC = mkMsg("m-c", "plain text message"); + store.ingestSession(planIngest(mkSession("sess-A", [mA, mC]))); + store.ingestSession(planIngest(mkSession("sess-B", [mB]))); + + // (1) Media bytes exist exactly once, referenced by both messages. + assert.equal(store.mediaRefCount(IMG_B64), 2, "one media row, two references"); + + // (2) STORED (pre-hydration) message content does NOT inline the + // base64 — query the table directly; getMessages always hydrates. + const rawStore = store as unknown as { db: { prepare(q: string): { all(...a: unknown[]): { content_hash: string }[] } }; getContent?(h: string): string | null }; + const rawRows = (sid: string): { content_hash: string }[] => + rawStore.db.prepare("SELECT content_hash FROM messages WHERE session_id = ?").all(sid); + for (const sid of ["sess-A", "sess-B"]) { + for (const row of rawRows(sid)) { + const body = (store as unknown as { db: { prepare(q: string): { get(...a: unknown[]): { bytes: Uint8Array } | undefined } } }) + .db.prepare("SELECT bytes FROM content WHERE hash = ?").get(row.content_hash); + assert.ok(body, `content row for ${sid}`); + assert.ok(!Buffer.from(body.bytes).toString("utf8").includes(IMG_B64), `stored content must not inline media bytes (${sid})`); + } + } + // (3) …and getMessages hydrates back to the ORIGINAL verbatim JSON. + const rowsA = store.getMessages("sess-A"); + assert.equal(JSON.parse(rowsA[0].content).imageBase64, IMG_B64); + assert.equal(rowsA[0].content, JSON.stringify(mA), "byte-identical round-trip"); + + // (4) Deleting one session keeps the shared media row; deleting both releases it. + store.deleteSession("sess-A"); + assert.equal(store.mediaRefCount(IMG_B64), 1, "shared media survives one deletion"); + store.deleteSession("sess-B"); + assert.equal(store.mediaRefCount(IMG_B64), 0, "last reference gone → media row released"); + + // (5) Legacy rows (inline bytes, pre-split) still read back verbatim. + store.close(); + } finally { + try { fs.rmSync(dir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); } catch { /* Windows stray handles */ } + } +}); + +test("media CAS: viewFor round-trip through loadLegacyView carries hydrated messages", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "storage-media-cas-2-")); + try { + const store = UnifiedStore.open(path.join(dir, "index.db")); + const m = mkMsg("m-img", "image turn", IMG_B64); + store.ingestSession(planIngest(mkSession("sess-img", [m]))); + const view = loadLegacyView({ + session: store.getSessionRow("sess-img")!, + messages: store.getMessages("sess-img"), + blocks: store.getBlocks("sess-img"), + deadRefs: store.getDeadRefs("sess-img"), + refs: store.getRefs("sess-img"), + }) as LegacySessionLike & { lastMessages?: CoreMsg[] }; + const tail = (view.lastMessages ?? []) as CoreMsg[]; + assert.equal(tail.length, 1); + assert.equal(tail[0].imageBase64, IMG_B64, "view path hydrates media bytes"); + store.close(); + } finally { + try { fs.rmSync(dir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); } catch { /* Windows stray handles */ } + } +}); + +test("media CAS: stale-row sweep releases media refs and mappings (full re-ingest drops the image)", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "storage-media-cas-3-")); + try { + const store = UnifiedStore.open(path.join(dir, "index.db")); + const mImg = mkMsg("m-img", "image turn", IMG_B64); + const mTxt = mkMsg("m-txt", "plain turn"); + store.ingestSession(planIngest(mkSession("sess-sweep", [mImg, mTxt])), undefined, { completeLog: true }); + assert.equal(store.mediaRefCount(IMG_B64), 1, "one media reference before re-ingest"); + + // The session's raw tail no longer contains the image message: the + // full path sweeps the stale row — its media mapping and CAS refcount + // must go with it, or the bytes leak forever (#3762 review, #3901). + store.ingestSession(planIngest(mkSession("sess-sweep", [mTxt])), undefined, { completeLog: true }); + + assert.equal(store.mediaRefCount(IMG_B64), 0, "swept message released its media ref"); + const mappings = (store as unknown as { db: { prepare(q: string): { all(...a: unknown[]): { n: number }[] } } }) + .db.prepare("SELECT COUNT(*) AS n FROM message_media WHERE session_id = ?").all("sess-sweep"); + assert.equal(mappings[0].n, 0, "swept media mappings deleted"); + const rows = (store as unknown as { db: { prepare(q: string): { all(...a: unknown[]): { n: number }[] } } }) + .db.prepare("SELECT COUNT(*) AS n FROM messages WHERE session_id = ?").all("sess-sweep"); + assert.equal(rows[0].n, 1, "only the surviving message row remains"); + store.close(); + } finally { + try { fs.rmSync(dir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); } catch { /* Windows stray handles */ } + } +}); diff --git a/tests/storage-archive-gc.test.ts b/tests/storage-archive-gc.test.ts new file mode 100644 index 0000000000..5257c4fe03 --- /dev/null +++ b/tests/storage-archive-gc.test.ts @@ -0,0 +1,109 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { planIngest, type LegacySessionLike } from "../src/storage/ingest.ts"; +import { UnifiedStore } from "../src/storage/store.ts"; +import { resolveArchiveAfterDays } from "../src/knobs.ts"; + +/** + * #3901 / #3463 (+ #3671 archive linkage) — session archiving + GC. + * + * archiveSession = export the envelope to the archive dir, delete the row + * bodies (messages/blocks/…), keep a tombstone sessions row (status + * 'archived', label intact), release every CAS refcount the session held. + * Shared content rows survive via other sessions' refcounts. restoreArchived + * re-ingests from the export. gc() = archive sweep by age + zero-ref + * collection + optional vacuum. + */ + +process.env.BILI_PERSIST_ZSTD = "0"; +process.env.BILI_STORAGE_UNIFIED = "full"; +process.env.BILI_CONFIG_FILE = path.join( + fs.mkdtempSync(path.join(os.tmpdir(), "bili-archive-gc-")), + "no-config.json", +); + +interface CoreMsg { id: string; role: string; contentType: string; text: string } +const mkMsg = (id: string, text: string): CoreMsg => ({ id, role: "user", contentType: "text", text }); + +const mkSession = (id: string, msgs: CoreMsg[], lastSeen = 2_000): LegacySessionLike => ({ + id, + createdAt: 1_000, + lastSeen, + meta: { protocol: "openai" }, + stats: {}, + metadata: { label: `label-${id}` }, + state: { blocks: [], nextBlockId: 1, nextRunId: 1, messageRefs: { byRaw: {}, byRef: {} } }, + blockContents: new Map(), + lastMessages: msgs as unknown as LegacySessionLike["lastMessages"], + lastMessagesFolded: false, + pluginSnapshot: undefined, + contentStore: undefined, +} as unknown as LegacySessionLike); + +test("archiveAfterDays knob: env > file > default 365; 0 disables", () => { + assert.equal(resolveArchiveAfterDays({ BILI_STORAGE_ARCHIVE_AFTER_DAYS: "7" }, undefined), 7); + assert.equal(resolveArchiveAfterDays(undefined, 30), 30); + assert.equal(resolveArchiveAfterDays(undefined, undefined), 365); + assert.equal(resolveArchiveAfterDays({ BILI_STORAGE_ARCHIVE_AFTER_DAYS: "0" }, 30), 0); + assert.equal(resolveArchiveAfterDays({ BILI_STORAGE_ARCHIVE_AFTER_DAYS: "garbage" }, undefined), 365); +}); + +test("archiveSession: tombstone + CAS release; shared rows survive; restore re-ingests", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "storage-archive-")); + try { + const store = UnifiedStore.open(path.join(dir, "index.db")); + const shared = mkMsg("m-shared", "shared body ".repeat(50)); + const onlyA = mkMsg("m-only-a", "exclusive to A ".repeat(50)); + store.ingestSession(planIngest(mkSession("sess-A", [shared, onlyA]))); + store.ingestSession(planIngest(mkSession("sess-B", [shared]))); + + const archived = store.archiveSession("sess-A", { archiveDir: path.join(dir, "archive") }); + assert.ok(archived.ok); + assert.ok(archived.exportPath, "archive returns the export path"); + assert.ok(fs.existsSync(archived.exportPath), "archive export file written"); + + // Tombstone: session row survives with status + label, bodies gone. + const tomb = store.getSessionRow("sess-A"); + assert.ok(tomb, "tombstone row kept"); + assert.equal((tomb as unknown as { status?: string }).status, "archived"); + assert.equal(store.getMessages("sess-A").length, 0, "message rows swept"); + + // Shared content row still referenced by sess-B; exclusive content released. + assert.equal(store.contentRefCountOfBody("shared body "), 1, "shared row survives with sess-B's refcount"); + assert.equal(store.contentRefCountOfBody("exclusive to A "), 0, "exclusive row released"); + + // Restore: rows come back from the export. + const restored = store.restoreArchived("sess-A", { archiveDir: path.join(dir, "archive") }); + assert.ok(restored.ok); + assert.equal(store.getMessages("sess-A").length, 2, "restored session has its messages back"); + store.close(); + } finally { + try { fs.rmSync(dir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); } catch { /* Windows stray handles */ } + } +}); + +test("gc(): age sweep archives only past-retention sessions; dormant shared rows collected", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "storage-gc-")); + try { + const store = UnifiedStore.open(path.join(dir, "index.db")); + const now = Date.now(); + const DAY = 86_400_000; + const oldMsg = mkMsg("m-old", "ancient body ".repeat(50)); + const newMsg = mkMsg("m-new", "fresh body ".repeat(50)); + store.ingestSession(planIngest(mkSession("sess-old", [oldMsg], now - 400 * DAY))); + store.ingestSession(planIngest(mkSession("sess-new", [newMsg], now - 1 * DAY))); + + const result = store.gc({ archiveAfterDays: 365, archiveDir: path.join(dir, "archive"), now }); + assert.equal(result.archived, 1, "only the 400-day-old session archives"); + assert.ok(store.getSessionRow("sess-old")); + assert.equal((store.getSessionRow("sess-old") as unknown as { status?: string }).status, "archived"); + assert.equal((store.getSessionRow("sess-new") as unknown as { status?: string }).status, "active"); + assert.equal(result.contentRowsDeleted >= 1, true, "released exclusive content collected"); + store.close(); + } finally { + try { fs.rmSync(dir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); } catch { /* Windows stray handles */ } + } +}); diff --git a/tests/storage-ext.test.ts b/tests/storage-ext.test.ts index 972434bc35..3ee171c76a 100644 --- a/tests/storage-ext.test.ts +++ b/tests/storage-ext.test.ts @@ -17,8 +17,8 @@ import type { Session } from "../src/session.ts"; * (caller-managed); whole-namespace deleteKv retires a lane; * 3. guards: control chars / oversize values / non-serializable values * throw with clear messages; - * 4. migration: a v3 database upgrades to v4 in place (tables appear, - * user_version flips), idempotently. + * 4. migration: a v3 database upgrades to the current schema (v7) in + * place (tables appear, user_version flips), idempotently. * * Extension state works in shadow AND full mode (side-table substrate like * checkpoints — it is not part of the legacy load path). @@ -155,11 +155,11 @@ test("guards reject hostile keys and oversize values", async () => { } }); -test("v3 databases migrate to v4 in place, idempotently", async () => { +test("v3 databases migrate to v7 in place, idempotently", async () => { const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "bili-ext-migrate-")); try { const dbPath = path.join(tmp, "index.db"); - // Build a current (v4) database, then roll the extension tables and + // Build a current (v7) database, then roll the extension tables and // version marker back to the v3 shape — exactly what an upgraded // production lane's file looks like to the new code. const fresh = UnifiedStore.open(dbPath); @@ -174,17 +174,17 @@ test("v3 databases migrate to v4 in place, idempotently", async () => { const migrated = UnifiedStore.open(dbPath); try { const ver = (migrated as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 5, "open() runs migrateSchema: v3 → v5"); + assert.equal(ver.user_version, 7, "open() runs migrateSchema: v3 → v7"); migrated.setKv("probe", "alive", true); assert.deepEqual(migrated.getKv("probe", "alive"), true); assert.deepEqual(migrated.listSessionIds(), [], "existing v3 content untouched"); } finally { migrated.close(); } - // Reopening a v5 database is a no-op (idempotent migration). + // Reopening a v7 database is a no-op (idempotent migration). const again = UnifiedStore.open(dbPath); const ver2 = (again as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver2.user_version, 5); + assert.equal(ver2.user_version, 7); again.close(); } finally { fs.rmSync(tmp, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); diff --git a/tests/storage-ingest-watermark.test.ts b/tests/storage-ingest-watermark.test.ts index 7f717187fe..1d2f523bb1 100644 --- a/tests/storage-ingest-watermark.test.ts +++ b/tests/storage-ingest-watermark.test.ts @@ -532,12 +532,15 @@ test("watermark: sampled verification still catches drift through the delta path } }); -test("watermark: v4 database migrates to v5 additively; newer user_version no-ops", async () => { +test("watermark: v4 database migrates to v7 additively; newer user_version no-ops", async () => { const dir = tmpdir("bili-wm-migrate-"); try { const dbPath = path.join(dir, "index.db"); // Craft the v4 shape: the current fresh-install DDL minus the v5 - // column, one live session with message rows, user_version=4. + // watermark column, one live session with message rows, + // user_version=4. Newer additive shapes still in the DDL (v6 orphans + // column, v7 media table) are presence-probed by the migration, so + // they re-land as no-ops. // created_at/updated_at must match mkLegacy: the sessions upsert // keeps created_at from the existing row (first-seen semantics), so // a mismatch would show up as sampled-verify drift, not a migration @@ -558,7 +561,7 @@ test("watermark: v4 database migrates to v5 additively; newer user_version no-op const store = UnifiedStore.open(dbPath); try { const ver = (store as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 5, "open() migrates v4 → v5"); + assert.equal(ver.user_version, 7, "open() migrates v4 → v7"); const row = store.getSessionRow("v4-sess"); assert.ok(row, "legacy session row survives the migration"); assert.equal(row!.lastIngestedSeq, null, "legacy rows read back with a NULL watermark (≡ full ingest)"); @@ -585,16 +588,18 @@ test("watermark: v4 database migrates to v5 additively; newer user_version no-op store.close(); } - // Version-gate no-op: a v5 DB rolled FORWARD past the code's target + // Version-gate no-op: a v7 DB rolled FORWARD past the code's target // (an older build opening a newer file) must open without error and - // leave user_version alone — the additive-column contract. + // leave user_version alone — the additive-column contract. The + // probe version sits ABOVE the current schema head (v7 + 1) so the + // no-op branch still exercises after future bumps. const fwd = new DatabaseSync(dbPath); - fwd.exec("PRAGMA user_version = 6"); + fwd.exec("PRAGMA user_version = 8"); fwd.close(); const older = UnifiedStore.open(dbPath); try { const ver2 = (older as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver2.user_version, 6, "newer user_version is left untouched"); + assert.equal(ver2.user_version, 8, "newer user_version is left untouched"); assert.ok(older.getSessionRow("v4-sess"), "rows stay readable"); } finally { older.close(); diff --git a/tests/storage-reap-journal.test.ts b/tests/storage-reap-journal.test.ts new file mode 100644 index 0000000000..2f965f6e30 --- /dev/null +++ b/tests/storage-reap-journal.test.ts @@ -0,0 +1,124 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { planIngest, loadLegacyView, type LegacySessionLike } from "../src/storage/ingest.ts"; +import { UnifiedStore } from "../src/storage/store.ts"; +import type { CompressionBlock } from "acp-kernel"; + +/** + * #3901 / #3841 — reaped blocks must stay reaped across restarts. + * + * A session whose substrate was reaped (#2695: state.blocks loses the entry, + * blockContents KEEPS the full text for derived-decompress) had its stale + * block rows resurrect by a viewFor rebuild after restart, polluting the + * envelope on the next save. The reap journal (#3901): record()-time targeted + * update — delete the reaped block rows, persist the orphan contents in + * sessions.orphans_json (schema v6) so viewFor rebuilds them into + * blockContents WITHOUT resurrecting state.blocks. + */ + +process.env.BILI_PERSIST_ZSTD = "0"; +process.env.BILI_STORAGE_UNIFIED = "full"; +process.env.BILI_CONFIG_FILE = path.join( + fs.mkdtempSync(path.join(os.tmpdir(), "bili-reap-journal-")), + "no-config.json", +); + +const mkSession = (id: string): LegacySessionLike => { + const blocks: CompressionBlock[] = Array.from({ length: 2 }, (_, i) => ({ + blockId: `b${i + 1}`, + runId: `r${i + 1}`, + tier: 1, + summary: `summary ${id} b${i + 1}`, + directMessageIds: [], + effectiveMessageIds: [`m${i + 1}`], + directBlockIds: [], + compressedTokens: 10, + createdAt: 1_000 + i, + survivedCount: 0, + generation: "young", + active: true, + })); + return { + id, + createdAt: 1_000, + lastSeen: 2_000, + meta: { protocol: "openai" }, + stats: {}, + metadata: {}, + state: { blocks, nextBlockId: 3, nextRunId: 1, messageRefs: { byRaw: {}, byRef: {} } }, + blockContents: new Map(blocks.map((b) => [b.blockId, { one: null, full: { text: `${b.summary} ORIGINAL TEXT`, count: 1 } }])), + lastMessages: undefined, + lastMessagesFolded: undefined, + pluginSnapshot: undefined, + contentStore: undefined, + } as unknown as LegacySessionLike; +}; + +test("reap journal: reaped block does not resurrect after restart; orphan content survives for decompress", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "storage-reap-journal-")); + try { + const dbPath = path.join(dir, "index.db"); + const store = UnifiedStore.open(dbPath); + const legacy = mkSession("sess-reap"); + store.ingestSession(planIngest(legacy)); + assert.equal(store.getBlocks("sess-reap").length, 2); + + // #2695 reap: state.blocks loses b2, blockContents keeps its text. + legacy.state.blocks = legacy.state.blocks.filter((b) => b.blockId !== "b2"); + // #3901 reap journal — record()-time targeted update for the orphan shape. + store.reapBlocks( + "sess-reap", + new Set(["b2"]), + new Map([["b2", legacy.blockContents.get("b2")!]]), + ); + + // Restart: a fresh handle on the same database. + store.close(); + const reopened = UnifiedStore.open(dbPath); + const rows = { + session: reopened.getSessionRow("sess-reap")!, + messages: reopened.getMessages("sess-reap"), + blocks: reopened.getBlocks("sess-reap"), + deadRefs: reopened.getDeadRefs("sess-reap"), + refs: reopened.getRefs("sess-reap"), + }; + const view = loadLegacyView(rows) as LegacySessionLike; + const state = view.state as { blocks: CompressionBlock[] }; + assert.equal(state.blocks.length, 1, "reaped block must NOT resurrect into state.blocks"); + assert.equal(state.blocks[0].blockId, "b1"); + assert.ok(view.blockContents.has("b2"), "orphan content survives in blockContents (decompress fallback)"); + const full = view.blockContents.get("b2")!.full as unknown as { text: string }; + assert.equal(full.text, "summary sess-reap b2 ORIGINAL TEXT"); + reopened.close(); + } finally { + try { fs.rmSync(dir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); } catch { /* Windows stray handles */ } + } +}); + +test("reap journal: idempotent re-reap and no-op for unknown blocks", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "storage-reap-journal-2-")); + try { + const store = UnifiedStore.open(path.join(dir, "index.db")); + const legacy = mkSession("sess-idem"); + store.ingestSession(planIngest(legacy)); + store.reapBlocks("sess-idem", new Set(["b1"]), new Map([["b1", legacy.blockContents.get("b1")!]])); + // Re-reap the same id and an id that never existed: neither throws. + store.reapBlocks("sess-idem", new Set(["b1", "bX"]), new Map()); + assert.equal(store.getBlocks("sess-idem").length, 1); + const view = loadLegacyView({ + session: store.getSessionRow("sess-idem")!, + messages: store.getMessages("sess-idem"), + blocks: store.getBlocks("sess-idem"), + deadRefs: store.getDeadRefs("sess-idem"), + refs: store.getRefs("sess-idem"), + }) as LegacySessionLike; + assert.equal((view.state as { blocks: CompressionBlock[] }).blocks.length, 1); + assert.ok(view.blockContents.has("b1")); + store.close(); + } finally { + try { fs.rmSync(dir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); } catch { /* Windows stray handles */ } + } +}); diff --git a/tests/storage-unified.test.ts b/tests/storage-unified.test.ts index a8bb8a6d04..3f7592aa49 100644 --- a/tests/storage-unified.test.ts +++ b/tests/storage-unified.test.ts @@ -739,7 +739,7 @@ test("schema v2: dev-window v1 database upgrades in place (identity columns adde assert.equal(row!.namespace, "proxy", "namespace backfills with its constant default"); assert.equal(row!.origin, null, "origin backfills NULL"); const ver = (store as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 5, "schema version stamped to 5 (v2 columns + v3 identity index + v4 extension namespaces + v5 watermark column)"); + assert.equal(ver.user_version, 7, "schema version stamped to 7 (v2 columns + v3 identity index + v4 extension namespaces + v5 watermark column + v6 orphans journal + v7 media map)"); // v3 backfill: every pre-existing sessions row claims its // (namespace, conversation_id) identity binding. const binding = (store as unknown as { lookupConversation(ns: string, id: string): string | null }).lookupConversation("proxy", "dev-window-1"); diff --git a/website/config-reference/env-only.yaml b/website/config-reference/env-only.yaml index 271e5b9b4a..d1bcbd5468 100644 --- a/website/config-reference/env-only.yaml +++ b/website/config-reference/env-only.yaml @@ -4,12 +4,20 @@ domain: env-only entries: - key: BILI_DSH_AUTHORED_COMPACTION type: boolean - default: "false" + default: "true (on, #3901 flip)" kind: env-only audience: user group: envonly desc_en: "Let bili ANSWER dsh compaction summarize calls locally with the folded snapshot instead of refusing (#3733): identity header + kernel-rendered block summaries, zero egress; replay recognized via the #2432 rebase." desc_zh: "让 bili 本地应答 dsh 压缩摘要请求(返回折叠快照而非拒绝,#3733):身份头 + 内核渲染格式的块摘要,零出口流量;重放经 #2432 再基线识别。" + - key: BILI_STORAGE_ARCHIVE_AFTER_DAYS + type: integer + default: "365" + kind: env-only + audience: user + group: envonly + desc_en: "GC age sweep window in days (#3463): sessions idle longer than this are archived (full export to the archive dir + tombstone row + content/media refs released). 0 disables the age sweep; the manual entry POST /__bili/storage/gc always runs. Config-file key storage.archiveAfterDays is the persistent form." + desc_zh: "GC 年龄清扫窗口(天,#3463):闲置超过该时长的会话被归档(完整导出到归档目录 + 墓碑行保留标签 + 释放内容/媒体引用)。0 关闭年龄清扫;手动入口 POST /__bili/storage/gc 始终可用。配置文件键 storage.archiveAfterDays 为持久形式。" - key: BILI_SNAPSHOT_SHADOW type: boolean default: "true (on)" diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index c1bc97d057..b82733d859 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -273,6 +273,13 @@ entries: group: process desc_en: "WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob." desc_zh: "index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。" + - key: storage.archiveAfterDays + type: integer + default: "365" + env: BILI_STORAGE_ARCHIVE_AFTER_DAYS + group: process + desc_en: "GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - full export to the archive dir under the sessions directory, a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). Env wins over the file tier." + desc_zh: "GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 完整导出到会话目录下的 archive/,墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。环境变量优先于配置文件层。" - key: sessions type: object default: "{} (all defaults)" diff --git a/website/config-reference/server.yaml b/website/config-reference/server.yaml index 6d5e12b7d5..ac9daf4c8a 100644 --- a/website/config-reference/server.yaml +++ b/website/config-reference/server.yaml @@ -349,6 +349,13 @@ entries: group: behavior desc_en: "Lift bili's local 422 refusal of dsh native compaction calls on EVERY wire lane bili serves — openai, anthropic, and responses (#1729/#2028; responses added by #2360, the lane dsh desktop's compaction actually rides): when true, dsh's compaction-basic may execute through bili; its checkpoint durably shadows raw history, so this is an explicit irreversible opt-in." desc_zh: "解除 bili 对 dsh 原生压缩调用的本地 422 拒绝——覆盖 bili 服务的全部线路:openai、anthropic、responses(#1729/#2028;responses 由 #2360 补上,正是 dsh 桌面端压缩实际走的线路):为 true 时 dsh 的 compaction-basic 可经 bili 执行;其 checkpoint 会永久覆盖原始历史,属显式的不可逆 opt-in。" + - key: dsh.authoredCompaction + type: boolean + default: "true (on, #3901 flip)" + env: BILI_DSH_AUTHORED_COMPACTION + group: behavior + desc_en: "bili answers dsh compaction summarize calls locally with the folded snapshot instead of refusing (#3723/#3733). ON by default since the 0.1.203 dogfood window held; false (or env BILI_DSH_AUTHORED_COMPACTION=0) restores the #1729 refusal path." + desc_zh: "bili 用折叠快照本地应答 dsh 压缩摘要请求而非拒绝(#3723/#3733)。0.1.203 狗粮窗口守住后默认开启;false(或环境变量 BILI_DSH_AUTHORED_COMPACTION=0)恢复 #1729 拒绝路径。" - key: resign type: "scheme → { enabled?, passthrough?, credentialRef? }" default: "{} (armed; built-in scheme sdk-hmac-sha256)"