Repository navigation
Description
[storage] 0.1.202: existing install carries 12.6 GiB of refcount=0 orphans (87% of content bytes) and still grows on a long session — how to reclaim before the GC/CLI release?
TL;DR
On a stock DSH desktop install (storage.unified unset → shadow, nothing tuned), a two-month-old Windows machine has:
index.db= 15.6 GB (14.54 GiB), content CAS = 14.46 GiB in 58,453 rows;- 12.62 GiB (87.3%) of content bytes are
refcount = 0orphans — 27,289 rows that nomessagesrow references; - the live payload is only 1.83 GiB (31,164 rows);
- growth is still observably live on 0.1.202: +37 MB/min measured over a 60 s window, content rows +5,727 in ~15 min, orphan rows +1,708 in the same window;
- the same logical payload is still stored many times: the largest exact
byte_lenfamilies have 39 identical-size copies, and 1,390 rows are >2 MB images totalling 3.03 GiB.
I already read #3748, #3463, #3762, #3671, #3830, #3928, #3911, #3958 and #3919 and I understand the write-amplification defect was fixed by #3767 (payload-addressed image ids, in 0.1.202) and that the GC/archive/CLI surface (#3911 / #3958) has not reached a release yet. My questions are about the existing install and the reclaim path, not about re-reporting the bug.
Environment
| billion-context | 0.1.202 (dist/build-info.json → commit 6deb4547), installed as a DSH desktop profile plugin (~/.dsh/profiles/desktop/node_modules/billion-context), started as bili start --host 127.0.0.1 --port <port> |
storage.unified |
unset → shadow (no BILI_* env overrides present in user or machine env) |
| host | DSH desktop (Windows 11 x64), Node v24.15.0 (built-in node:sqlite) |
| DB | %USERPROFILE%\.local\share\billion-context\sessions\index.db (+ -wal 20.6 MB, -shm) |
| schema | PRAGMA user_version = 5, tables: sessions, content, messages, blocks, dead_refs, refs, checkpoints, ccr_entries, conversations, session_ext, kv_store — i.e. no message_media, SCHEMA_VERSION=5 in the bundled dist |
| sessions | 26 sessions / 31,045 messages rows / 187 blocks rows |
| cache health | [acp-usage] input=112385 cached=112128 (cache hit 100%) — proxy economics are fine, this is not runaway traffic |
The machine lost ~50 GB of C: free space in two months; this sidecar is 14.5 GiB of it.
Measurements (read-only node:sqlite / python sqlite3, mode=ro)
db file 15,616,643,072 B (14.54 GiB)
content rows 58,453 total 14.46 GiB
refcount > 0 31,164 total 1.83 GiB
refcount = 0 27,289 total 12.62 GiB ← 87.3 % of content bytes
rows > 2 MB 1,390 total 3.03 GiB
of which orphan 963 total 2.10 GiB
largest byte_len families (identical byte_len only): 39, 39, 38, 31, 31 copies
Live growth sample, 60 s apart, nothing else running except one DSH conversation:
22:08:39 15,573,843,968 B 26,422 refcount=0 rows
22:09:39 15,611,084,800 B 27,289 refcount=0 rows → +37.2 MB / +867 rows in 60 s
Content-row growth over a longer window: 52,703 → 58,453 rows / ~20 min, while messages rows stayed at ~31k.
Mechanism evidence from ~/.local/state/billion-context/bili.log (v=0.1.202)
- The sidecar is confirmed active:
[storage] unified shadow mode ENABLED. - One very long DSH session re-sends a huge body every turn:
[...] request: 9373 msgs, inbound=32.1MiB, outbound=18.8MiB, local=1200ms. - Every such turn logs a fold-reconcile drift, which is the [Bug][storage] Shadow digest drift clears ingest watermark and forces full-tail re-ingest (v0.1.201) #3919 signature:
[fold-reconcile] resent history drifted: 62 covered id(s) missing with no anchor match
followed by[storage] unified shadow DIGEST DRIFT …(5 occurrences today), i.e. the ingest watermark is dropped and the next save re-ingests the full tail — 32 MB of history rewritten into the CAS per turn. - In the bundled 0.1.202 code, the only
DELETE FROM content WHERE refcount <= 0statements run insidedeleteSession()/ checkpoint deletion.putContent()inserts withrefcount = 1and re-ingest releases the previous hashes viareleaseContent()→ the rows becomerefcount = 0and nothing sweeps them until a session is deleted — which matches the measured 27,289 orphan rows.
So the picture I get is: #3767 stopped the image-id re-minting, but (a) the long-session full-tail re-ingest after drift still replaces rows (orphaning them), and (b) without GC the orphans never go away, so the file keeps growing.
Questions
- Existing installs: when the GC/CLI release lands, will
bili storage gc --vacuum fulldelete pre-existingrefcount = 0rows (the 12.62 GiB above), or is the sweep scoped to sessions it archives during that run? Master'sgc()returnscontentRowsDeletedand the CLI prints "zero-ref content rows" in the sweep plan, which reads like a full sweep — I want to confirm before I recommend waiting to the machine's owner. - Interim reclaim:
PR #3965states "legacy JSON session files remain the source of truth in both shadow and full mode … nothing is lost". Is deletingindex.db(+-wal/-shm) and theretrieve/imgcache while bili is stopped the supported interim reclaim, or would you rather users disablestorage.unified(legacy) first? Any consistency concern (e.g. a stale shadow digest/watermark after the file is gone)? retrieve/img: this one session's cache dir holds 15,058 files / 4.63 GB, with many per-ref variants of the same screenshot (samem<ref>-*prefix, different hashes/sizes). Is that expected (re-extraction per turn) or another side effect of the same re-keying? Is deleting it safe/regenerated-on-demand?- ETA/advice: is 0.1.203 the release that carries [PR] Storage wrap-up, nine items in one batch: GC/archive + reap ledger + media CAS + incremental pointers + dsh generation c | 存储收尾九项一车:GC/archive + reap 台账 + 媒体 CAS + 增量指针 + dsh 代压缩默认开 + 多实例文档 (#3901) #3911 + [PR] feat: CLI surface for storage maintenance — bili sessions ls/rm/archive/restore + bili storage gc (#3928) #3958? If it is days away I would rather leave the DB untouched; if it is not, the owner wants the manual path now.
Reproduction (read-only, no writes)
import sqlite3, os
p = os.path.expanduser("~/.local/share/billion-context/sessions/index.db")
con = sqlite3.connect("file:" + p + "?mode=ro", uri=True)
c = con.cursor()
print("db GiB", round(os.path.getsize(p) / 2**30, 2))
print(c.execute("select count(*), sum(byte_len) from content").fetchone())
print(c.execute("select count(*), sum(byte_len) from content where refcount=0").fetchone())
print(c.execute("select byte_len, count(*) n from content group by byte_len order by count(*) desc limit 5").fetchall())查重
Searched open+closed: index.db, storage growth, refcount=0, orphan content, retrieve/img, disk space, cleanup existing installs. The bloat family is tracked by #3748 (closed, fixed by #3767) / #3463 (closed, GC) / #3762 (closed) / #3830 / #3928 (closed, CLI). I found no issue covering an already-bloated install on 0.1.202 and the interim reclaim path, which is what this report is about. Happy to fold it into #3463 or #3830 as a comment if you prefer — just say which.
Reported from a user machine by an agent at the owner's request; all numbers above were measured read-only, no data was deleted or modified.
Activity
ranxianglei commentedon Oct 20, 2026
ework
ranxianglei commentedon Oct 21, 2026
(ai: vllm-qwen/qwen3.8-27b)
查重: 不重复。 机制侧问题都已有条目且大多已修(#3463 GC 机制 → #3911 已合入 master;#3841 reap 台账;#3762 媒体 CAS;#3919 drift → 已由 #3948 关闭);本 issue 问的是「存量膨胀 install 在新版发布前如何回收 + 发版时机」这个操作面问题,没有其他 issue 覆盖。
标签: bug / dsh / scope/storage / p1(缺陷类;DSH desktop 车道;统一存储域;按 owner 2026-10-19 修订,存储只增不减的失控增长属缓存泄露类 → p1)。
四个问题都已对照当前 master 代码(即下一版本将带的形态)逐条核实:
Q1:GC 会删掉存量 refcount=0 的行吗?
会,全部。
gc()不限于本次运行归档的会话:先归档闲置超期会话(storage.archiveAfterDays,缺省 365 天,0=关),然后做全局清扫DELETE FROM content WHERE refcount <= 0(src/storage/store.ts:1317-1321)——你现存的 12.62 GiB 孤儿行一次清完。你的库早于 auto_vacuum=INCREMENTAL 常开机制,所以还要配一次 full VACUUM 让文件物理缩小:CLI 与 HTTP 入口(
POST /__bili/storage/gc?vacuum=full)走同一个 facade(store.unifiedGc),效果一致。Q2:停掉 bili、删 index.db 作为临时回收,可行吗?
在你当前配置下(0.1.202 + shadow)机械上安全——但如实说明:今天这不是一个有文档的受支持流程,是代码分析结论。依据:
建议步骤(在升级前做,原因见下方行为变更披露):
index.db、index.db-wal、index.db-shm改名留底(加.bak-日期后缀),重启后观察几天无异常再彻底删除;retrieve/img/,见 Q3,安全);更保守的变体:先设
BILI_STORAGE_UNIFIED=legacy彻底停掉统一存储写入,再删库。注意 legacy 计划移除且下一版默认是 full(见下),这只是过渡手段,不是归宿。Q3:retrieve/img 同一截图多个变体,预期吗?能删吗?
预期形态,安全删除。 该目录是客户端图片 restore 的物化缓存:索引时图片落盘为
<状态目录>/retrieve/img/<会话目录>/m<ref>-<内容哈希前缀>.<ext>,读取「文件优先」,缺失时从内容存储重新生成(writeRestoredImage 不存在即写入)。两个性质:BILI_SESSION_GC=1还会随过期会话一起清 img 目录。你 15,058 个文件 / 4.63 GB 说明近一周 churn 很重;直接清掉即可,最坏情况是下次访问时按需重新生成。Q4:0.1.203 是否就是带这些修复的版本?什么时候?
规划上是,但 0.1.203 尚不存在:npm latest 仍是 0.1.202(= 你现在装的)。所需内容已全部在 master:存储收尾九项(#3911:GC/归档、reap 台账 v6、媒体 CAS v7、增量指针、dsh 代压缩默认开)、MySQL 后端(#3388/#3931)、drift 取证 + 头槽探针(#3948,关闭了 #3919)、维护 CLI(#3958);CHANGELOG [Unreleased] 已写明「0.1.203 狗粮窗口守住」。发版需 owner 批准切分支——Agent 只能提议、不能动手。
向 owner 的发版提议:建议尽快切 0.1.203。等待方有两处:本 issue 的磁盘回收需求,以及 #3919 报告者需要在含 #3948 的构建上跑下一轮 drift 取证采集。批准后切 release PR → 人工合并 → CI 自动发布,无人工 publish 环节。
行为变更披露(升级到下一版本时会发生什么)
storage.unified,当前 effective=shadow(纯写镜像),升级后自动变 full——index.db 开始被读路径使用。JSON 仍是 source of truth,无数据风险;想留在 shadow 就显式设"storage.unified": "shadow"。这也是为什么 Q2 的删库操作建议在升级前做——full 模式下删库丢掉的 DB-only 状态更多。BILI_DSH_AUTHORED_COMPACTION=0恢复旧行为。storage.archiveAfterDays(缺省 365,envBILI_STORAGE_ARCHIVE_AFTER_DAYS)、storage.backend(缺省 sqlite,envBILI_STORAGE_BACKEND)。一个遗留缺口(决策项,请 owner 拍板)
升级后的回收路径完整,但没有周期性自动清扫(README 明示 "No scheduler: these run on demand"):zero-ref 收集只发生在会话删除/归档/checkpoint 删除/手动 gc 时。你这台机器的客户端持续改写历史(drift 签名),孤儿行会持续产生,两次手动 gc 之间库会缓慢回涨。选项:
bili storage gc(incremental vacuum 很便宜,可常跑)。影响面:零行为变更;代价:依赖用户纪律,你这台机器数月后会再次膨胀。storage.autoGcEveryHours,缺省关):进程内定期跑同一个安全谓词(zero-ref 收集 + incremental vacuum),永不自动 full VACUUM。回归风险:低——谓词与既有自动收集路径(deleteSession/archiveSession)逐字相同,增量 vacuum I/O 小;成本:新增配置键,按配置面纪律须由 owner 设计。结构化八问
bili storage gc --dry-run预览后--vacuum full即回收全部存量孤儿;[PR] storage: shadow DIGEST DRIFT now names the diverged dimension and slot; prefix rewrites fail soft at the delta boundary #3948 把 drift 收敛窗口从 ≤20 次保存缩到同次保存;当下止血 = Q2 手工步骤。遗留缺口 = 无周期自动清扫(上方决策项)。storage.archiveAfterDays(365)/storage.backend(sqlite);选项 B 若通过再加 1 个(待 owner 设计)。收工判定:本 issue 无需我另开 PR——修复机制均已合入(#3911/#3948/#3958),本 issue 交付物 = 上述四问答案 + 临时止血步骤 + 两项 owner 请求:(a) 批准切 0.1.203;(b) 对周期自动 gc 选项 A/B 拍板。你现在就可以在 0.1.202 上执行 Q2 的止血步骤。
storage correctness wave 3: derived hydration fields, delete-on-zero …
ranxianglei commentedon Oct 24, 2026
(ai: zhipuai-lb/glm-5.3)
孤儿 issue 清理:修复已由已合并 PR #4177, #4249, #4414 落地(含今日五列合并列车),但本 issue 未随之关闭。现补关;若该修复仍有残留问题请 reopen 并描述复现。
maozhuoshushu commentedon Nov 9, 2026
这台机器有几乎一样的症状,补一份同版本区间的对照数据,可能能帮你确认是「孤儿内容」还是「写放大」。
症状对照(你是 0.1.202,我们是 0.1.213
133f3525c)dead_refs8,779 行;content715,995 行 / 3,728 MBindex.dbpage_count1,379,580 × 4096Data Units Written33,713,640 → 75,017,044)ccr_entriesretrieve/关键差别:我们的事不是「文件变多」,是「盘一直被写」
C 盘是 QLC(P3 Plus 1TB,标称 TBW 220 TB),31 天写掉 21.15 TB = 682.6 GB/天,是该盘历史均速(4.19 GB/h)的 8.25 倍,
Percentage Used4% → 7%。但磁盘占用只涨了一点点:
index.db5.4 GiB + 信封 717 MB + TEMP 若干。也就是说 50 GB 那类「占空间」的问题和我们的「烧盘」是两个不同的 bug:剩下的全在
index.db的页级重写与 WAL 循环(主库 5.83 GB + 恒定 8,388,608 B 的-wal),文件系统只看到「mtime 变了、size 没变」。三层对账排除了 SSD 内部原因(400 s 纯计数器窗口):
⇒ 100% 是 host 发起的写,不是 GC 或 QLC 折叠。
如果你也在查「盘为什么一直响」,可以直接量这个
两个坑:① 窗口必须 ≥300 s(短窗口读到的是 0,写缓存/SLC 会推迟计入:我们写 1.2 GB 时 SMART 只涨了 1.2 MB,静止 600 s 又补涨 1,326 MB);② 千万别用逐进程的
IOWriteBytesPersec,它把 loopback 网络算进「写」,加起来能到LogicalDisk的 1160%。我们做的四道治理(每步都留了开关,可回退)
storage.jsonWriter: "off"—— 289 个信封自 2026-11-08 23:26 起零写入(与 test: pin persistence write-amplification shape and mode cost ordering (#3049) #3275 一致)。需重启 bili 才生效(POST /__bili/shutdown→ 宿主 40 s 内自动重新 spawn,不用重启整个 DSH)。28 B5 2F FD判别,旧未压缩文件原样可读)——session-order2.64×、prefix-affinity2.17×、plugin-conversations3.99×。var,只能改代码默认值;env 覆盖在本机无效,因为 bili 是 DSH host 的子进程,Windows 子进程继承父进程环境块,host 起来后再setx收不到)。busy != 0次数 = 0,从不阻塞;took p50 17 ms / max 30 ms)。结果:636 GiB/天 → 190 GiB/天(−70%)。功能、压缩效果、缓存命中、成本、CCR 取回、内存口径全部不变;唯一代价是崩溃时最多丢 180 秒的快照状态(三快照都是可重算缓存,
index.db里的会话正文不丢)。如果你那边
index.db是 15.6 GB,建议先做这件事PRAGMA quick_check在我们的 5.4 GB 库上耗时 7.0 s、返回ok。15.6 GB 大概 20 s 级,值得在动 GC 之前先跑一次,确认是不是真的损坏 vs 只是体积大。另外ccr_entries0 行的话,index.db的体积基本全是content表(715,995 行 / 3,728 MB BLOB),和孤儿数量的关系比和行数更直接。完整报告(测量方法学、ETW 归因、四道治理的收益与代价、复现与回滚脚本):https://gist.github.com/maozhuoshushu/11331b6e9c3388d431365f2fffd1df92