Repository navigation
[PR] storage: move six JSON sidecar states onto the unified sqlite store (#5753) - #5757
Conversation
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.8842-pr.5757.a8525a4This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5757.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…ew fixes (#5753) Addresses the self-review findings on #5757 and the windows-CI failure it caused (tests/snapshot-render-chain.test.ts installs a PARTIAL store stub — tail-view methods only — and drove every proxy-mode render through rolloutInstallId, which hit setKvAsync on the stub and crashed). - persist.ts: new kvStore() probe — returns the singleton only when it is non-legacy AND actually exposes getKv/setKv/listKv; partial test stubs now degrade to the legacy file path instead of throwing - host-plugin-notice.ts: pre-hydration dismissals no longer block the one-time file→kv migration (hydrate unions into the existing lazy cache and latches its own dismissHydrated flag); mid-hydration dismissals survive the final cache assignment - apig-resign.ts: legacy-gate direction restores master parity (no sticky pendingCache when not riding kv — sibling-lane/manual file edits stay visible); failed unified writes demote back to the FILE path (the old discipline) instead of looping retries on sqlite - order/affinity: same demote-on-failure on RMW errors; order adopts the presence-based hadRow check so an empty-but-present row is not re-uploaded on every boot - plugin.ts: failed unified conversation-table writes demote to the legacy file path instead of retrying into the same dead chain every 300ms - restart.ts/server.ts: withExitDeadline() bounds every graceful-exit storage flush at 5s — a wedged volume can no longer hold SIGTERM/SIGINT drain, eaddrinuse die paths or restart hand-over hostage (pre-migration flushes were sync and always returned) Tests: two new regression cases (pre-hydration dismissal still migrates; partial stub degrades without throwing). Typecheck clean; snapshot-render-chain + all six unified suites + affected regressions green; unused-export and test-import-graph gates clean.
ranxianglei
left a comment
There was a problem hiding this comment.
Self-review pass over c275b1e before merge. Findings and their resolution (fixes all in 4d74613):
Fixed
- [MED] Partial store stubs crashed every unified-mode caller.
tests/snapshot-render-chain.test.tsinstalls a stub with only tail-view methods;rolloutInstallId()(hot path viafeatureEnabled) hitsetKvAsyncon it → TypeError — this is exactly what killed both windows-22 CI shards. Fix: newpersist.ts kvStore()probe returns the store only when non-legacy AND actually exposing getKv/setKv/listKv; partial stubs now degrade to the legacy file path. All six callers switched from rawgetStore()to it, plus a regression test in tests/rollout-identity-unified.test.ts. - [MED] host-plugin-notice: a dismissal landing before the boot hydration lazily created the cache, which made the hydrate bail — blocking the one-time file→kv migration and hiding sibling rows for the process lifetime (inherited pattern flaw — see follow-up issue for the #5548 twin). Now: hydrate unions into the existing lazy cache, latches its own
dismissHydratedflag, and mid-hydration dismissals survive the final assignment. Regression test added. - [MED-LOW] apig-resign sticky cache broke the legacy-gate direction (master re-read the file on every op; sibling-lane/manual edits stayed invisible until restart after first local mutation). The legacy direction no longer caches at all = exact master parity. Also: failed unified writes now demote back to the FILE path (old discipline) instead of retry-looping on a dead chain every mutation.
- [LOW] order-persist used content-based
hadRowwhile the affinity twin used presence-based — an empty-but-present row triggered re-upload + "migrated N" log on every boot. Aligned to presence check. - [LOW] No timeout on exit drains: a wedged sqlite handle could hold SIGTERM/SIGINT drain, eaddrinuse die paths, and restart hand-over hostage forever (pre-migration flushes were sync and always returned). New
withExitDeadline(label, op)in restart.ts bounds every flush chain at 5s, warn-logged, proceeds with what landed (next boot re-hydrates). Applied to finishShutdown/finishDrain, all three die paths, and both restart hand-over routes. - [LOW] Misleading degradation warnings (order/affinity/plugin promised "legacy file path for this process" but kept retrying kv forever once the ready flag latched). The demote-on-failure change in #3 makes the file fallback real and the messages true again.
Accepted as-is (with rationale)
- Cross-instance concurrent first-boot: migration does a blind whole-row setKv; the losing lane's unique rows are transiently gone from kv until its next mutation re-writes them (self-healing, bounded, requires two instances racing first boot). RMW-under-chain would complicate six modules for a self-healing window; noted here rather than changed.
_reset*ForTest()seams swap chains without draining in-flight ops: test-only ghost-write risk, accepted.- order/affinity do read-modify-write persists while resign/plugin keep whole-row overwrite: this mirrors each module's PRE-change file-writer semantics (all four files were whole-map last-writer-wins); not an inconsistency introduced by this PR.
- Informational: early-exit paths flush conversations+sessions only (parity with master); killed in-flight migrations self-heal via re-import.
Follow-up (filed separately): the finding #2 pattern flaw also exists in #5548's src/notice-store.ts (addDismissedKeyInMemory + if (cache !== null || hydrating) return) — same pre-hydration-blocked-migration window for upgrade-notice receipts. Not touched in this PR to keep scope clean.
|
Round-2 CI triage found a branch regression (not an infra flake): two
Fix in 4732e77: |
|
Round-3 result: green. The lone round-2 failure was State of the PR at 4732e77 (three commits: c275b1e migration · 4d74613 review fixes + partial-store-stub hardening · 4732e77 import-graph fix): all CI green, review comments addressed above, follow-up filed at #5761 for the notice-store twin inherited from #5548. Ready for human merge whenever. |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) CI 车道收尾确认(本轮):独立复核 round-3 判定——head 4732e77 全绿,零代码改动。证据链(本轮逐条重查过,不是转述):
八问:
披露:我先前那条收尾评论因编号解析错误误发到了姊妹线程(GitHub PR #5755,增量 refs diff,镜像 #5757),已在那边补了误发说明楼层,请忽略那边那条。 判定:head 4732e77 全 CI 绿,本车道收工,等待人工合并。 |
(ai: vllm-qwen/qwen3.8-27b) 查重:不重复。mirror 上无其他 [PR] 指向 #5753「六个进程级 JSON sidecar 迁统一存储」主题——这是该 issue 的唯一 PR。(注:mirror↔GitHub 编号错位已确认,mirror #5757 是另一个无关 PR;对外一律引 GitHub 号 5757。) 标签: 结论先行:可合。 当前 head ① 是什么问题/需求:把六个进程级 JSON sidecar(session-order / prefix-affinity / rollout-id / resign-pending / plugin-conversations / host-plugin-notices)从各自独立文件迁到统一 sqlite(kv_store 行),沿用 #5548 notice-receipt 的载体规则。是存储收敛重构,不是修 bug。 ② 什么原因导致的(为何改):六个文件各一套 load/save/debounce/torn-write 防护,与 #5548 已建好的统一存储载体规则重复;其中 plugin-conversations.json 还是唯一 in-place 写的文件(#406 才有 pid-tmp+fsync+rename 防 torn-write)。迁 sqlite 后原子提交直接消掉那一类,并统一「kv 主 / 文件仅 pre-hydration fallback + legacy-only 载体 / 一次性导入」纪律。 ③ 影响范围与严重程度:
④ 怎么修复的(机制):
⑤ 是否有回归问题(逐项审过既有行为):
⑥ 是否破坏缓存:不涉及模型 prefix-cache 语义。这六个是持久化载体不是 KV/prefix cache;迁移只换存储后端,内容(entries/table/id/map)原样搬运 + 校验函数守畸形数据。prefix-affinity/session-order 仍是同一份 snapshot,只是落点变了。 ⑦ 是否改了设计方向:延续而非转向——把 #5548 已确立的「统一存储 + 载体规则」推广到剩余六个进程级状态。数据流清晰:boot hydrate(文件 warm-up→kv overlay→一次性上迁)→ steady-state kv 主 → exit flush。边界干净,未污染请求热路径(best-effort warn-once)。 ⑧ 是否引入了新配置:无新 config/env/CLI flag(config-surface-guard 绿)。复用既有 行为变更披露(Deliverables 第 4 条):
代码质量:六模块模式一致;校验函数( 测试有效性:6 个 hermetic 套件断言有牙——migration-once、restart-survival-via-kv(memory+file 双清)、pre-hydrate-fallback→late-import、idempotent re-hydration(no row churn)、LRU-cap-during-merge(1030→1024,oldest 从 front evict,newest 存活)。CI storage-baseline preload 把大语料钉 legacy,既有文件断言套件继续盖另一方向,gate 双向都有行。非假测试。 diff 整洁度:干净。17 文件全目的相关(6 模块 + cli/server wiring + persist plumbing + 新 exit-deadline.ts + 6 测试),真实 diff 无 lockfile churn、无空白噪音、无跑题文件;3 commit 逻辑分组(migration / hardening / test-isolation 修复)。 证据说明:CI 当前 head 30/30 绿为权威编译+测试判定(§7.2)。behind-master 风险已查:PR 落后 origin/master 37 commit,唯一两侧都碰的热文件是 判定:可合。 建议合并前 rebase 到当前 master(落后 37,虽可干净合并)后让 CI 在 rebased head 重跑一遍即发。我不动 merge。 |
|
Round-4 triage (run on fb65337): single red was Status at fb65337: all CI green again, otherwise unchanged from the round-3 note. |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Round-4 独立复核(head fb65337):分类成立——msgid lane 套件的既有 CI flake,不是本分支回归。本轮逐条独立验证过(不是转述):
八问:
跟踪问题处置:按 CI 协调政策(2026-10-24)不为该 flake 开新 issue——结论已按政策补评到 #4599(CI 总控);它属于既有 #4161 时序窗口家族,单独立项只会触发重复调度。若后续复发到影响合并,修法方向如正文所述: 判定:head fb65337 全 CI 绿,本车道收工,等待人工合并。 |
fb65337 to
56aeb4c
Compare
…ew fixes (#5753) Addresses the self-review findings on #5757 and the windows-CI failure it caused (tests/snapshot-render-chain.test.ts installs a PARTIAL store stub — tail-view methods only — and drove every proxy-mode render through rolloutInstallId, which hit setKvAsync on the stub and crashed). - persist.ts: new kvStore() probe — returns the singleton only when it is non-legacy AND actually exposes getKv/setKv/listKv; partial test stubs now degrade to the legacy file path instead of throwing - host-plugin-notice.ts: pre-hydration dismissals no longer block the one-time file→kv migration (hydrate unions into the existing lazy cache and latches its own dismissHydrated flag); mid-hydration dismissals survive the final cache assignment - apig-resign.ts: legacy-gate direction restores master parity (no sticky pendingCache when not riding kv — sibling-lane/manual file edits stay visible); failed unified writes demote back to the FILE path (the old discipline) instead of looping retries on sqlite - order/affinity: same demote-on-failure on RMW errors; order adopts the presence-based hadRow check so an empty-but-present row is not re-uploaded on every boot - plugin.ts: failed unified conversation-table writes demote to the legacy file path instead of retrying into the same dead chain every 300ms - restart.ts/server.ts: withExitDeadline() bounds every graceful-exit storage flush at 5s — a wedged volume can no longer hold SIGTERM/SIGINT drain, eaddrinuse die paths or restart hand-over hostage (pre-migration flushes were sync and always returned) Tests: two new regression cases (pre-hydration dismissal still migrates; partial stub degrades without throwing). Typecheck clean; snapshot-render-chain + all six unified suites + affected regressions green; unused-export and test-import-graph gates clean.
|
已 rebase 到最新 master( rebase 解决的两处冲突(
rebase 后新增的一个修复( 本地验证(worktree,干净 XDG state 目录):
与 #5901 快照表的关系不变:两条线正交,可各合各的。 |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Rebase 轮独立复核完成(head
八问:
判定: |
|
评估性能问题 |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 性能评估完成。结论:无实质性能风险——六处状态的写频率全部有界、无一在请求关键路径上被等待、单次写成本与旧文件方案持平(本环境实测)且机制上更省。证据如下: 1. 写触发频率(逐状态)
2. 热路径判定:六处写全部 fire-and-forget(模块级 promise chain: 3. 单次写 IO 特性:kv_store 为 WITHOUT ROWID 表、(namespace,key) 主键;库开
4. 最坏情形边界:唯一理论成本 = 封顶文档每 flush 一次的 JSON 往返(order 快照上限 8MiB)——但同一 5. 诚实边界(行为 delta,已披露):resign-pending 的持久化时序从「记录时同步落盘」(旧契约:on-disk by any exit)变为「异步 enqueue + shutdown await」;SIGKILL 硬杀下的丢失窗口从 ≈0 扩大到「上次入队写落地之后」——但该事件稀有、≤32 条封顶,且持久 store 失败会立刻降级回文件纪律(#5757 review fix)。这是 PR 内的设计取舍,不是静默漂移。 八问:
判定:性能面通过,无阻塞项;叠加此前各轮 CI 全绿 + CLEAN,等待人工合并。 |
|
我们在一块 QLC(Crucial P3 Plus 1TB,标称 TBW 220 TB)上做了 31 天实地测量,结论是:这条 PR 应该提到 我们量到的(0.1.213 /
|
| 文件 | 治理前原大小 | zstd 后 | 重写间隔(debounce 500 ms) |
|---|---|---|---|
session-order.json |
6.65 MB | 2.52 MB(2.64×) | 中位 13.6 s,min 6.2 s |
prefix-affinity.json |
1.68 MB | 0.76 MB(2.17×) | 同上 |
plugin-conversations.json |
137,003 B | 34,303 B(3.99×) | 300 ms debounce,300 s 内 31 次 |
rollout-id.json |
51 B | — | 静态 |
port-zone.json |
24 B | — | 启动一次 |
proxy-origin |
364 B | — | 启动一次 |
单 session-order.json 外推 ≈188 GB/天,与实测同量级。迁移后这 188 GB/天 + 2.69 MB/s 的 affinity 直接归零。
三个会影响这个 PR 的实测细节
1. session-order 的体积几乎全是 frags,而且没有时间淘汰。
57 条 entry 里 frags 占 83.7%;最大单条 12,357 个 frag / 942,513 B。normalizeForWrite2 只按 lastSeen 降序截断 MAX_ORDER_ENTRIES=1024 / MAX_SNAPSHOT_BYTES2=8 MB,没有任何按时间丢弃——本机 prefix-affinity 满 1024 条,最早一条是 27 天前。迁库时顺手加一个「超过 N 天丢弃」能立刻削掉存量,不必等 GC。
2. 迁进 kv_store 本质上还是整行重写,别把 JSON 那个放大原封不动搬过去。
这正是 #5599 里 mariopasquali 量的那个数:28 MiB mirror + 6 MiB metadata,三次变长小字段更新产生 215,385,392 B WAL,而定长编码只要 12,392 B。session-order 是典型的「长数组 + 尾部追加」,如果按单个 kv_store.value_json 整份 JSON.stringify 再 UPDATE,会从「每 13.6 s 重写 6.65 MB 的 JSON 文件」变成「每 13.6 s 重写 6.65 MB 的 SQLite 行」,总量基本不变。建议按 session 分 key(或只存 delta/frag 尾部),让每次 UPDATE 的 B-tree 页脏化只覆盖真正变化的部分。
3. port-zone.json / proxy-origin 的排除是对的,别改。
#5753 里已经把这两个排除掉了(launcher spawn 热路径)。我们这边确认 proxy-origin 只有 364 B、启动写一次,port-zone.json 24 B——顺手迁了不会省任何写入,只会把 spawn 可用性绑到 db 上。
顺带一个部署侧的坑(如果这个 PR 改了写路径)
我们用 BILI_SNAP_DEBOUNCE_MS 覆盖 debounce 时完全无效,原因是 bili 是 DSH host 的子进程,Windows 子进程继承的是父进程环境块,不是注册表 User 环境——host 已经跑起来之后再 setx 的新变量,新 spawn 的子进程收不到。最后只能改代码默认值。任何「运行时靠 env 调参」的设计都会踩这个(PERSIST_DEBOUNCE_MS2 是硬编码 var,反而没这个问题,但它也就完全不可配)。
完整报告(测量方法、四道治理的收益与代价、复现与回滚脚本):https://gist.github.com/maozhuoshushu/11331b6e9c3388d431365f2fffd1df92
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 🔶 需要决策 收到实地测量报告。三条技术判断我逐条对着 PR head 逐条核实(代码证据)① session-order 体积以 frags 为主、无时间淘汰 —— 成立。
② 迁进 kv_store 仍是整行重写、写入量基本不变 —— 成立;这也是我上轮评估没讲透的地方。
③ port-zone.json / proxy-origin 排除是对的 —— 同意,维持现状。 部署侧坑(Windows env 继承)机制属实:bili 作为 DSH host 的子进程,继承的是父进程启动时的环境块,host 跑起来后 数据可信度算术抽查通过:41,303,404 units × 512 B ≈ 21.15 TB;POH 加权速率 21.15 TB / 612 h ≈ 34.6 GB/h vs 历史 4.19 GB/h ≈ 8.26×(与你的 8.25× 一致)。这份数据(SMART units 增量 + GB/day 前后对照 + gist 复现脚本)适合作为后续削量工作的验收基线——是否采纳由 owner 定。优先级提到 #3049 那个级别同样是 owner 的判断,我这里只能确认:底层问题真实、有量化数据支撑。 选项(🔶 需要决策)
推荐 A,理由:可用性/一致性优先于性能(#3128 原则);写放大是独立的失败域,按粒度政策归并进一个后续 issue 管理(子缺陷清单:淘汰策略 / 分片布局 / 可选压缩);现场数据直接当验收基线。未拍板时默认 A:本车道不再动这个 PR,PR 保持 green+CLEAN 等人工合并。 八问
判定:等 owner 在 A/B/C 中拍板;在此之前本 PR 保持现状(green + CLEAN @ 24ec71c),本车道停手。 |
… the sessions dir (#2960) #5988 stopped the CCR content-store sidecar write under the off switch, but the contract "the SQLite index is the ONLY persistent copy" had no mechanical guard: any future save path that starts writing JSON there would silently regress it (the sidecar itself is exactly that shape of violation). This adds the gate. tests/storage-json-writer-off-json-writes.test.ts: - snapshot-diff design: every *.json file in the store dir is hashed before/after each phase; added/modified entries are checked against the whitelist. No fs interception — whatever API a write path uses shows up in the diff, so a new writer cannot hide behind an un-intercepted call. - Phases: A first save (ZERO .json writes), B restart adopt + save (ZERO), C replica + restore (the restored id's own materialized files ONLY — envelope + CCR sidecar, named via relPathFor/ contentStoreRelPathFor; session files are hash-named, so the concrete per-run set is derived from the restored id, not a regex), plus an on-control leg proving the snapshot mechanism actually sees writes. - JSON_WRITE_WHITELIST documents the allowed category (restore materialization: the index is not a restore source and does not carry forked CCR stores); extending it requires the code path and the reason the index cannot carry the bytes. - Mutation-verified: disabling the #5988 sidecar gate makes phase A fail with the offender file listed — the gate catches exactly the regression #5988 fixed. Scope: the sessions directory (persistence layer). CLI-level state files under stateDir() are handled separately by #5757 (kv_store migration). Verification: typecheck ✓, gate suite 2/2 ✓, sibling off suites + issue5805 9/9 ✓, storage regression battery 110/110 ✓, check:unused-exports ✓, check:test-import-graph ✓, gen-config-docs check-coverage ✓, check-architecture-docs ✓
|
@ranxianglei 应你之请,把这条核实报告里的三处对着代码和算术重核了一遍。A/B/C 的判断本身站得住,但记录里有三处要更正: 1. 算术行的单位差 1000 倍(结论不变,但单位口径要落实) 2. debounce knob:键已存在,只是没接线 3. 合并状态已过期 结论维持 A 合理(迁移写入量中性——两条路径每次 flush 都写整份文档,已对代码核实,合入不新增磨损;削量是独立的 human-gated 布局变更,值得单独 issue + 现场数据做验收基线)。唯一前置:第 1 点的单位确认;若单位真是 512 B,削量 issue 的优先级应相应下调。 |
session-order.json, prefix-affinity.json, rollout-id.json, resign-pending.json, plugin-conversations.json and host-plugin-notices.json now persist through the unified storage (kv_store rows), following the #5548 notice-receipt pattern: the legacy file stays only as a pre-hydration read fallback and as the sole carrier for legacy-only instances; a one-time import lands existing file content on first hydration, after which a unified instance never writes the file again. - cli.ts boot hydrates the five process-lifetime states (no-op on legacy-only instances and before anything created the storage singleton) - server.ts hydrates the plugin conversation table right after loadConversations(); shutdown paths (SIGTERM/SIGINT drain, eaddrinuse dies, restart hand-over) await the new async flushes so a graceful exit loses nothing recorded this run - apig-resign keeps SYNCHRONOUS visibility on record/clear (callers assert the pending map immediately); only the carrier write is queued — the FILE path still writes sync, a new flushResignPending() drains the kv path on every exit route - the wx-file creation race in rollout-id is kept BY DESIGN: two fresh-install lane copies must converge on one id before either mirrors into sqlite - SQLite atomicity also removes the pid-tmp torn-write class from the plugin conversation table (the #406 workaround stays inert there) Tests: six new hermetic suites (one-time migration / restart survival via kv / pre-hydrate file fallback with late import / idempotent re-hydration / LRU cap). The CI storage-baseline preload pins the broad corpus to legacy, so existing file-assertion suites keep covering the other direction of the gate.
…ew fixes (#5753) Addresses the self-review findings on #5757 and the windows-CI failure it caused (tests/snapshot-render-chain.test.ts installs a PARTIAL store stub — tail-view methods only — and drove every proxy-mode render through rolloutInstallId, which hit setKvAsync on the stub and crashed). - persist.ts: new kvStore() probe — returns the singleton only when it is non-legacy AND actually exposes getKv/setKv/listKv; partial test stubs now degrade to the legacy file path instead of throwing - host-plugin-notice.ts: pre-hydration dismissals no longer block the one-time file→kv migration (hydrate unions into the existing lazy cache and latches its own dismissHydrated flag); mid-hydration dismissals survive the final cache assignment - apig-resign.ts: legacy-gate direction restores master parity (no sticky pendingCache when not riding kv — sibling-lane/manual file edits stay visible); failed unified writes demote back to the FILE path (the old discipline) instead of looping retries on sqlite - order/affinity: same demote-on-failure on RMW errors; order adopts the presence-based hadRow check so an empty-but-present row is not re-uploaded on every boot - plugin.ts: failed unified conversation-table writes demote to the legacy file path instead of retrying into the same dead chain every 300ms - restart.ts/server.ts: withExitDeadline() bounds every graceful-exit storage flush at 5s — a wedged volume can no longer hold SIGTERM/SIGINT drain, eaddrinuse die paths or restart hand-over hostage (pre-migration flushes were sync and always returned) Tests: two new regression cases (pre-hydration dismissal still migrates; partial stub degrades without throwing). Typecheck clean; snapshot-render-chain + all six unified suites + affected regressions green; unused-export and test-import-graph gates clean.
…port phase (#5753) Round-2 CI caught two auto-restart subtests failing on this branch while they pass at c275b1e: the held-CONNECT-tunnel repro read a stale marker, and the tolerant-marker test could not see its own writes. Both trace to one mechanism. MARKER_FILE is frozen when src/restart.ts first evaluates; the auto-restart suite re-points XDG_CACHE_HOME after its static imports and relies on importing ../src/restart.ts dynamically so the constant freezes under the suite's own sandbox. Commit 4d74613 added an "import { withExitDeadline } from './restart.js'" edge in src/server.ts — and tests/auto-restart.test.ts statically imports server.ts — which dragged restart.ts forward into the static phase, freezing MARKER_FILE under the preload root while the suite's clears/writes went to the env-dynamic path. withExitDeadline now lives in its own src/exit-deadline.js (body unchanged); server.ts imports it from there, restart.ts imports it for its two default flush hooks. The server->restart edge is gone and the dynamic-import ordering the suite documents holds again: local repro 13/13, full auto-restart file + snapshot-render-chain 22/22, the six unified-store suites + apig-resign/affinity/persistence regressions 88/88, typecheck and both repo gates clean.
The one-time file-to-kv import only fired while the kv row was empty or absent. Once the row held any other scheme, a late file entry (cross-lane drift window: a legacy-only sibling still writing the file) could never migrate. Align with the per-entry trigger the sibling modules already use (order-persist, affinity-persist), plus a regression test.
The rebase onto master picked up the new architecture-docs guard (#5930) which requires reference/architecture.md to list every tracked src/ file. exit-deadline.ts (split out in the exit-flush hardening commit of this PR) was missing.
24ec71c to
0968d50
Compare
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 三处更正我逐条独立重核过,全部成立,记录按此修正。其中第 ① 条是我上一轮的算术错误,明确认领。另外:既定顺序的前置已满足(两个叠层 storage PR 均已合入 master),本轮把最后这次 rebase 做完了——详见下文。 更正确认(逐条重核证据)① 算术单位 —— 你是对的,我错了。
报告方两个端点与 512 kB/unit 精确吻合(17.26 / 38.41 TB),即其 SMART 工具按 512 kB 每 unit 换算。交叉核对复算无误:21.15/220 = 9.6%,Percentage Used 走 3% ≈ 6.6 TB 当量,差 3.2×。补一个佐证测量方需要确认单位的旁证:若按 512 B 或 1 kB 口径,该盘 4,124 POH 的累计写入只有 17–34 GB,对一块长跑系统盘低得反常(暗示计数器重置或厂商私有换算)。不受此争议影响的部分:8.25× 是比值、与单位无关,仍成立;FS 层直测(session-order.json 6.65 MB 整文件重写、中位间隔 13.6 s ≈ 42 GB/天主机写入)不经过 SMART 换算,机制与其 FS 量级都站得住。受影响的只是总量估计、「约 10 个月磨完盘」的紧迫性框架、以及削量 issue 的定级。同意你的处置:先让测量方确认工具换算单位,再定削量 issue 优先级;若真是 512 B,优先级相应下调。 ② debounce knob —— 你是对的,我表述不准。 ③ 合并状态过期 —— 核实属实,且前置已满足。 本轮执行:最后 rebase(既定顺序的下一步)
结论与挂起项A 维持(你的裁定)。挂起两项,都不阻塞本 PR 合并:① 测量方确认 SMART 工具换算单位 → 决定削量 issue 优先级(512 B 则下调);② 削量 issue 立项时带上子缺陷清单(年龄淘汰策略 / 分 key 或增量编码布局 / persist.debounceMs 接入两个 sidecar persister),以现场数据为验收基线。needs-decision 标签保留到单位确认闭环,也可由你直接摘除。 八问
判定:PR @ 0968d50 全绿、base 最新,等人工合并;本车道完成本轮工作后停手。 |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 本轮完整报告已在上方楼层(三处更正确认 + 最终 rebase 执行记录)。一句话收口:更正全部成立并已修正记录;既定顺序前置(#5988/#5992 已合入)满足,最终 rebase 已推——新 head 0968d50,本地门禁(typecheck/build/定向 32 测试)与 CI(64 项全绿)均通过,PR 现处 base 最新、可人工合并。挂起两项不阻塞合并:① 测量方确认 SMART 工具换算单位 → 定削量 issue 优先级;② 削量 issue 子缺陷清单(年龄淘汰 / 分 key 或增量布局 / persist.debounceMs 接入两个 sidecar persister)。本车道停手待命。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.1135-master.082143e # this exact build
npm install -g billion-context@master # always the newest merged build
|
… legacy→kv migration (#5761) A dismissal landing before the cli-boot hydration lazily created the receipts cache, which made hydrateNoticeReceipts bail on `cache !== null` — skipping the one-time legacy JSON→kv migration for that process's lifetime (re-import churn on every boot) and hiding sibling lanes' kv rows until restart; a dismissal landing inside the hydrate's await gap was clobbered by the final cache assignment. Twin of the host-plugin-notice fix that landed in the #5757 line (commit 4d74613), ported onto master's API surface (unifiedKvActive/getStore; the kvStore() probe from that line is not merged yet): - hydrate unions into the existing lazy cache instead of bailing; a new hydrated latch (distinct from lazy-init) gates re-entry - the live-cache snapshot moves to just before the publish with no awaits in between, so mid-hydration dismissals survive the final assignment - publish only on success: a failed listKvAsync now leaves the cache null (legacy-file fallback stays live, next call retries) instead of clobbering it with an empty/partial map for the process lifetime Tests: two regression cases mirroring the #5757 shapes — both fail on the pre-fix code and pass with it; adjacent notice suites green. (cherry picked from commit cf85588)
… legacy→kv migration (#5761) A dismissal landing before the cli-boot hydration lazily created the receipts cache, which made hydrateNoticeReceipts bail on `cache !== null` — skipping the one-time legacy JSON→kv migration for that process's lifetime (re-import churn on every boot) and hiding sibling lanes' kv rows until restart; a dismissal landing inside the hydrate's await gap was clobbered by the final cache assignment. Twin of the host-plugin-notice fix that landed in the #5757 line (commit 4d74613), ported onto master's API surface (unifiedKvActive/getStore; the kvStore() probe from that line is not merged yet): - hydrate unions into the existing lazy cache instead of bailing; a new hydrated latch (distinct from lazy-init) gates re-entry - the live-cache snapshot moves to just before the publish with no awaits in between, so mid-hydration dismissals survive the final assignment - publish only on success: a failed listKvAsync now leaves the cache null (legacy-file fallback stays live, next call retries) instead of clobbering it with an empty/partial map for the process lifetime Tests: two regression cases mirroring the #5757 shapes — both fail on the pre-fix code and pass with it; adjacent notice suites green. (cherry picked from commit cf85588)
What
The six remaining process-lifetime JSON state files now ride the unified
storage (
kv_storerows), following the #5548 notice-receipt pattern:Carrier rules everywhere: kv row is primary once hydrated; the legacy
file stays only as a pre-hydration read fallback and as the sole carrier for
legacy-only instances (
BILI_ENCRYPTION_KEYopt-out — no unified sidecar);a one-time import lands existing file content on first hydration, after
which a unified instance never writes the file again. Persistence stays
best-effort warn-once (never throws into a request path), mirroring #5548.
How it is wired
cli.tshydrates the five process-lifetime states right next tohydrateNoticeReceipts()(no-op on legacy-only instances and before thestorage singleton exists);
server.tshydrates the plugin conversationtable immediately after
loadConversations().restart hand-over routes (
restart.ts) now await the async flushes(
flushOrderSkeletons/flushPrefixAffinitybecamePromise<void>; newflushResignPending()), so a graceful exit loses nothing recorded this run.Deliberate non-changes
map right after recording; only the carrier write is queued, and the FILE
write path remains sync (a refusal on the hot path must already be durable
by any exit).
lane copies must converge on ONE id through the cross-process file race
before either mirrors into sqlite; the file becomes a once-per-lifetime
bootstrap artifact, all steady-state reads/writes go to kv.
conversation table (the minCompressRange: user docs specify tokens, kernel executes in chars (4× unit mismatch); note appended: registryLookup conflicts silently fall back. | minCompressRange 用户文档写 tokens、内核按 chars 执行(4× 单位缺口);附 registryLookup 冲突即静默回落的备注 #406 workaround stops being needed there).
summary-credentials file, update-lane self-update caches, host-owned
protocol files, and pre-DB-open sidecars all keep their files for the
recorded reasons.
Tests
Six new hermetic suites (full mode pinned in-suite, mirroring
tests/notice-unified-store.test.ts): one-time migration exactly once,restart survival via kv with memory + file wiped, pre-hydrate fallback to
the file with late import, idempotent re-hydration (no row churn), plus the
plugin LRU cap enforced during the merge. The CI storage-baseline preload
pins the broad corpus to legacy, so existing file-assertion suites
(
apig-resign,affinity-persist,rollout, persona-restart-stability, …)keep covering the other direction of the gate untouched.
Pre-flight: typecheck clean; targeted regression batches green
(71 + 98 + 71 tests across the touched entry points); unused-export gate
and test-import-graph clean. No kernel/pi-subagents changes.