Skip to content

[PR] storage: make orphan-content GC actually finish (refcount index + batched sweep) - #4800

Merged
ranxianglei merged 5 commits into
masterfrom
2026-10-26_4796_gc-batched-collection
Oct 30, 2026
Merged

ranxianglei merged 5 commits into
masterfrom
2026-10-26_4796_gc-batched-collection

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Fixes #4796

What this changes

The zero-ref content sweep in gc() ran as one uncommitted statement over an unindexed scan of the CAS table. On bloated stores it never commits: the field report (#4796) shows 8 h / 28,131 s CPU → 0 rows changed, WAL pinned around 221 MB, the exclusive write lock held until process death, and live proxies starved into the "degraded" panel state. Because nothing ever collected, orphan content accumulated forever — the v8 identity backfill (migrateV8Identities) releases the refs of dropped duplicate rows without delete-on-zero and explicitly relies on "the next gc run collects those content rows", which could not happen.

One mechanism family (uncollectable zero-ref CAS content), five sub-defects:

  • ① schema v9: idx_content_refcount on content(refcount) — candidate selection becomes an index range scan instead of a full-table scan; the one-time build cost is what open-time quick_check (Storage Reliability Trio: open-time self-check (quick_check) + online backup (VACUUM INTO) + second-process probe on the | 存储可靠性三件套:开库自检 quick_check + 在线备份 VACUUM INTO + 同库第二进程探测 #4175) already pays on every open
  • ② sqlite gc(): autocommit batches of 2000 (GC_ZERO_REF_BATCH_SIZE) — select candidates by hash, delete re-filtered under each statement's own snapshot; best-effort wal_checkpoint(TRUNCATE) between batches; progress logs at start / every 10 batches / completion
  • ③ mysql backend mirrored — DDL KEY idx_content_refcount (refcount) for fresh installs, idempotent probe + ALTER TABLE … ADD INDEX in init() for existing databases (CREATE TABLE IF NOT EXISTS does not upgrade), DELETE … LIMIT ? batches in gc(); the first-write-wins bili_meta stamp is intentionally NOT bumped (v8 precedent)
  • ④ regression suite tests/issue4796-gc-batched-sweep.test.ts — fresh-install v9 + EXPLAIN index usage, v8→v9 upgrade path, 2100-orphan multi-batch collection with 5 MiB blobs (WAL stays ≤ the 8 MiB journal_size_limit), exact-multiple edge (4000 = 2×batch), crash-resume convergence, and an end-to-end repro of the actual field path: v8 backfill drops a wireIndex-split twin message row, leaves its content row zero-ref, gc collects exactly that residue with foreign_key_check clean
  • ⑤ lifecycle-scoped collection (PR [PR] storage: make orphan-content GC actually finish (refcount index + batched sweep) #4800 review round): deleteSession / archiveSession / deleteCheckpoint (sqlite + mysql, 6 sites) no longer finish with a full-table zero-ref DELETE inside their lifecycle transaction — same giant-commit pathology as ② on a bloated store. Their own releases already go through releaseContent's delete-on-zero, so the trailing sweep only ever collected OTHER sessions' pre-existing orphans incidentally. Replaced with collectZeroRefIn(): chunked (ZERO_REF_SWEEP_CHUNK = 500) DELETEs re-filtering refcount <= 0 over just the operation's released-hash set — bounded work, transaction atomicity kept. Regression tests pin that orphans present at call time survive these ops and gc remains their collector of record.

Persistence surface (§7.4 flag)

  • sqlite: STORAGE_SCHEMA_VERSION 8 → 9. Additive only — one new index object created inside the existing migration transaction on first open after upgrade. No column/table/format change, no data movement. Older builds opening a v9 file ignore the extra index (SQLite indexes are invisible to readers that don't know them); newer builds opening older files create it on migrate. Version-pin assertions in four existing suites updated to the new head.
  • mysql: existing tables gain the index via an idempotent probe + ALTER at open; the stamped MYSQL_SCHEMA_VERSION meta value is unchanged by design (first-write-wins, same treatment as the v8 columns).
  • No new config/env/CLI surface. The batch size is a named module constant in each backend (2000 — the size proven workable by the field experiment recorded in the issue report); no knob added, per config-surface discipline.

Behavior change disclosure (old → new)

Aspect Old New Why
zero-ref sweep commit model one giant statement, commits only at process death autocommit batches of 2000, each independently committed a half-finished sweep survives crashes; re-runs converge to empty
candidate selection full scan of content index range scan on refcount O(orphans) instead of O(table)
WAL during sweep grows without bound (uncommitted-tx frames cannot checkpoint) truncated between batches, bounded by journal_size_limit (8 MiB) write lock held per batch only
observability silent progress logs at start / every 10 batches / completion #4612 lesson: no-progress runs are unacceptable UX
db file contents no refcount index one extra index object enables all of the above
lifecycle ops (deleteSession / archiveSession / deleteCheckpoint, both backends) whole-table zero-ref census + delete inside the op's transaction (unbounded on bloated stores; also silently collected other sessions' orphans) scoped DELETE over the op's own released-hash set only (bounded, atomicity kept) pre-existing orphans now persist until the next gc run — gc is the collector of record; no user-visible contract relied on the incidental over-collection
archiveSession().contentRowsDeleted metric whole-table zero-ref count before the sweep (included other sessions' orphans) deletions made by THIS archive's scoped sweep only (usually 0) honest per-operation accounting; global totals remain available via gc() and contentZeroRefReport()

Deliberate choice: the select + delete pair is NOT wrapped in an explicit transaction. An explicit transaction pins the read snapshot at SELECT time, so a concurrent writer re-referencing a row between the two statements would be invisible and we would delete freshly-referenced content (data loss). Autocommit statements each see the latest committed state, and the DELETE re-filters refcount <= 0 under its own snapshot — strictly narrower exposure than today's whole-table window.

The lazy ingest reaper (per-row DELETE … WHERE hash = ? AND refcount <= 0) stays unchanged — it was always single-hash and bounded. The other three lifecycle ops moved to scoped collection per sub-defect ⑤ above.

Coexistence notes

Tests

Targeted local runs green on head efedc0522: new suite 8/8 (incl. the two scoped-lifecycle tests); issue4677-v8-migration-streaming + storage-ext + storage-archive-gc 15/15; storage-archive-gc + session-gc + state-gc + storage-unified + storage-maintenance-cli + proxy-orphan-gc 61 pass / 1 gated skip; typecheck clean. Full-suite verdict = CI lanes on this head. The mysql paths (multi-batch gc + scoped collection) are covered by the contract lane's gc happy-path plus mirror-of-unit-tested logic (no local mysql in this environment).

@ranxianglei ranxianglei changed the title storage: make orphan-content GC actually finish (refcount index + batched sweep) [PR] storage: make orphan-content GC actually finish (refcount index + batched sweep) Oct 26, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 —— 本 PR 即 #4796 的修复本体;检索 open+closed PR,无其他同机制(零引用 CAS 收集)在途项。相邻的 #4619(quick_check 推迟)、#4628(VACUUM 锁预检)机制不同,正文已显式声明 out-of-scope。
标签: bug + dsh + scope/storage + p1(存储只增不减、官方清理路径不可用 = 缓存泄露类 p1)。

审核结论(独立做过:方向 / 回归清点 / 代码 / 测试 / diff)

方向正确。 我对代码与现场数据独立复核了 issue 的两层根因:content 表无 refcount 索引(每次 sweep 全表扫)+ 整表删除是一条巨型未提交语句(对其他连接不可见、WAL 帧无法 checkpoint、独占写锁持有到进程死亡)。修复打在产生侧(gc 如何 sweep),不是症状补丁;「select+delete 不包显式事务」的理由(显式事务钉快照会删掉刚被重新引用的字节)成立且已写进注释。

回归清点(diff 触碰的既有行为逐项过):

  1. gc() 提交模型:单条巨型 autocommit DELETE → 每批 2000 行 autocommit——有意变更(已披露),即本次修复本身;
  2. contentRowsDeleted 语义:「删前估算」→「实际删除数」——已披露,更诚实;
  3. lifecycle 内嵌 sweep(deleteSession/archiveSession/deleteCheckpoint × sqlite/mysql 共 6 处):逐处读上下文确认未动,仍在其自身事务内做全表零引用删除,原子性契约保持;
  4. dry-run plan(unifiedGcPlan)只读路径未动,受益于新索引;
  5. 迁移:additive-only CREATE INDEX IF NOT EXISTS 在既有迁移事务内、幂等;旧构建开新文件的 forward-probe no-op 路径由更新后的 watermark 测试覆盖;
  6. mysql init():每次 open 新增 information_schema 探测 + 条件 ALTER,幂等低成本;fresh-install DDL 加 KEY;bili_meta stamp 不动(与 v8 先例一致);
  7. 4 个更新测试文件为纯机械版本钉 8→9,无逻辑改动(diff 逐行读过)。
    未发现无意回归。

测试有效性:全是有牙的真测试。 6 个新测试:query plan 断言候选查询真的走 idx_content_refcount(负向门)、v8→v9 重开重建索引、多 MB blob 跨批 + WAL ≤8MiB 钉住(正是现场病理形状)、恰好整批倍数终止边界、崩溃恢复收敛、e2e 复现孤儿主来源(v8 backfill 释放 wireIndex twin 行 → gc 收走、keeper 存活、foreign_key_check 干净)。本地在 PR head 实测:typecheck 干净;定向 7 个套件(新套件 + 4 个更新套件 + 既有 storage-archive-gc / storage-maintenance-cli 回归面)58 pass / 0 fail / 1 env-skip。MySQL 分批路径本地无实例,仅 gated e2e lane 覆盖——知悉,不拦。

治理注记: schema 8→9 属持久化面变更(§7.4 human-gated 类)。#4796 F3 当时已向你要签核;但缺索引正是你开的 issue 自己诊断的根因,我判定这是在落实你的诊断而非 agent 自创配置面——如不同意 bump,合并前说一声即可。

结构化八问

  1. 是什么问题:bloated DB 上 bili storage gc 的零引用清理永远跑不完——单条巨型未提交 DELETE 跑了 8 小时 0 行提交,WAL 无法 checkpoint,写锁饿死活跃代理。
  2. 什么原因导致:两层叠加——content 表无 refcount 索引(每次 sweep 全表扫);整表删除一条长未提交语句(不可见、不可 checkpoint、独占锁持有到进程死)。孤儿主来源是 v8 identity backfill 释放 wireIndex 重复行后依赖「下次 gc 收走」,而那次 gc 永远完不成。
  3. 影响范围与严重程度:所有启用统一存储且积累大量零引用内容的用户(现场:766K 行 / 14.4GB 孤儿);p1——存储失控增长 + 官方清理手段不可用 + 清理运行期间活跃代理被饿死。
  4. 怎么修复的:双管——schema v9 加 idx_content_refcount(候选选择变索引范围扫描);gc() 改 2000 行一批的小 autocommit 删除,每批独立提交 + best-effort wal_checkpoint(TRUNCATE),进度抗崩溃、写锁按批持有;MySQL 镜像(DDL + 幂等探测 + LIMIT 分批)。select+delete 刻意不包显式事务(防删掉并发重新引用的字节)。
  5. 是否有回归问题:见上清点,未发现;高危模式逐项过——语义漂移(contentRowsDeleted)已披露、原子性契约保持、迁移 additive-only、无阈值/时序/输出格式漂移。一个残余暴露(非阻塞):lifecycle 内嵌 sweep 仍是事务内全表零引用删除,bloated DB 上首次成功 gc 之前删会话可能再触发同样病理,有界于「到首次 gc 的时间窗」,修法涉及收集时机语义变更,已记入 billion-context database cleanup records GC bug | billion-context 数据库清理记录 gc bug #4796 清单待你定夺。
  6. 是否破坏缓存:不涉及(CAS 身份与 refcount 纪律不变,测试断言 foreign_key_check 干净)。
  7. 是否改了设计方向:没有架构转向;提交模型变更即修复本身(已披露)。附带观察:gc() 目前仅手动触发(CLI + admin endpoint),无常驻自动 gc——稳态孤儿控制依赖运维定期跑,建议评估定时 gc 作为后续,已记入 billion-context database cleanup records GC bug | billion-context 数据库清理记录 gc bug #4796。
  8. 是否引入新配置:无——批量大小是两侧具名模块常量(GC_ZERO_REF_BATCH_SIZE / MYSQL_GC_ZERO_REF_BATCH,各 2000,现场实测定标),无新 env/config/CLI。

CI 状态: 写作时 head 0890d503 的 21 个 check-run 全部 queued(guards 先行,test lane 随后),出结果我会跟进补报。分支落后 master 33 个提交,合并前需 rebase(store.ts 是热文件,#4619/#4628 若先合入需按正文所述协调 hunk)。

判定: 审核未发现阻塞问题;待 CI 绿 + rebase 最新 master 后可合。

@ranxianglei ranxianglei added bug Something isn't working dsh p1 scope/storage labels Oct 26, 2026
ranxianglei pushed a commit that referenced this pull request Oct 26, 2026
…hashes (#4796)

PR #4800 review item 1: deleteSession/archiveSession/deleteCheckpoint
(sqlite + mysql, 6 sites) each finished with a full-table
DELETE FROM content WHERE refcount <= 0 inside their lifecycle
transaction — the same giant-commit pathology as the old gc sweep on
a bloated store. Their own releases already go through
releaseContent's delete-on-zero, so the trailing sweep only ever
collected OTHER sessions' pre-existing orphans incidentally.

Replace it with collectZeroRefIn(): a chunked DELETE re-filtering
refcount <= 0 over just the operation's released-hash set — bounded
work, transaction atomicity kept. Pre-existing orphans now persist
until the next gc run (disclosed semantics change; gc remains the
collector of record). archiveSession.contentRowsDeleted now counts
this operation's scoped deletions only.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

第二轮审核:新提交 efedc0522(lifecycle 内嵌 sweep 限定到本操作自身 hash 集) —— 即我第一轮 review item 1 的落地。旧 head 0890d503 的 CI 批次已被 re-push 取消,新 head 的 21 个 check-run 当前全部 queued(guards 先行,test lane 随后),出结果我继续跟进。

独立核验(不是照抄 commit message):

行为变更披露(owner policy 第 4 条,旧值→新值+原因):

  1. archiveSession().contentRowsDeleted:旧 = 全表零引用普查数(COUNT 后整表删),新 = 仅本归档操作的 scoped 删除数——日志行「archived X (N content rows collected)」的 N 会变小;原因:数字应反映本操作的工作而非顺带清扫,全表普查正是病理的一部分;
  2. 既有孤儿不再被 lifecycle 操作顺带收集,一律留到 gc() 收——收集职责收敛到 gc(唯一全局收集者)。注意与第一轮 finding 2 叠加:gc 目前仅手动触发,此提交后孤儿收敛完全依赖 gc 运行,定时 gc 评估的重要性上升。

结构化八问(第二轮,未变项沿用第一轮结论)

  1. 是什么问题:同第一轮(gc 零引用清理跑不完);本轮新增子项:lifecycle 操作(deleteSession/archiveSession/deleteCheckpoint × 两后端)在自身事务里做全表零引用删除,bloated DB 上是同一家族的巨型提交病理。
  2. 什么原因导致:主 lesion 同第一轮;新增子项是历史残留——releaseContent 早已内联 delete-on-zero,尾部全表 sweep 只顺带清别的会话的孤儿,收益为零、风险为 8 小时写锁。
  3. 影响范围与严重程度:p1 不变;新增子项有界于「首次成功 gc 前的时间窗」,单独看轻于主 lesion,但同机制家族一并修掉是对的。
  4. 怎么修复的:同第一轮(v9 索引 + gc 分批)+ 本轮 collectZeroRefIn:对本操作释放的 hash 集做分块(500/语句)DELETE、事务内重新过滤 refcount,工作量有界、原子性保持,两侧后端镜像。
  5. 是否有回归问题:再清点过——10 个定向套件含全部删会话/归档/checkpoint 行为面,全绿;两项有意变更已按上披露;无无意回归。
  6. 是否破坏缓存:不涉及(同第一轮,CAS/refcount 纪律不变)。
  7. 是否改了设计方向:无架构转向;收集职责更清晰地收敛到 gc()。finding 2(无常驻自动 gc)在本提交后更突出,已记入 billion-context database cleanup records GC bug | billion-context 数据库清理记录 gc bug #4796 待你评估。
  8. 是否引入新配置:无——新增模块级具名常量 ZERO_REF_SWEEP_CHUNK / MYSQL_ZERO_REF_SWEEP_CHUNK(各 500,控制单条 IN-list 参数数),非配置面。

判定:两个提交均未发现阻塞问题;待 efedc0522 CI 绿 + rebase 最新 master(仍落后 33 提交)后可合。

ci-check added 2 commits October 26, 2026 13:01
…4796)

The gc() sweep deleted zero-ref content rows with one uncommitted
statement over an unindexed scan. On bloated stores (field report:
766K rows / ~20 GB, 658K orphans) it never committed - invisible to
every other connection for hours while holding the exclusive write
lock and starving live proxies - so orphan content accumulated
forever (v8 identity backfill releases refs without delete-on-zero
and relies on gc to collect).

- schema v9: idx_content_refcount on content(refcount); candidate
  selection becomes an index range scan; one-time build cost is what
  open-time quick_check already pays on every open
- sqlite gc(): autocommit batches of 2000 (select candidates by hash,
  delete re-filtered under each statement's own snapshot - no explicit
  transaction, which would pin the read snapshot and risk deleting a
  concurrently re-referenced row), best-effort wal_checkpoint(TRUNCATE)
  between batches, progress logs
- mysql backend mirrored: DDL KEY + probe/ALTER in init(), DELETE ...
  LIMIT ? batches in gc()
- regression suite incl. an end-to-end repro of the v8-backfill
  residue path; version-pin ripples in four existing suites
…hashes (#4796)

PR #4800 review item 1: deleteSession/archiveSession/deleteCheckpoint
(sqlite + mysql, 6 sites) each finished with a full-table
DELETE FROM content WHERE refcount <= 0 inside their lifecycle
transaction — the same giant-commit pathology as the old gc sweep on
a bloated store. Their own releases already go through
releaseContent's delete-on-zero, so the trailing sweep only ever
collected OTHER sessions' pre-existing orphans incidentally.

Replace it with collectZeroRefIn(): a chunked DELETE re-filtering
refcount <= 0 over just the operation's released-hash set — bounded
work, transaction atomicity kept. Pre-existing orphans now persist
until the next gc run (disclosed semantics change; gc remains the
collector of record). archiveSession.contentRowsDeleted now counts
this operation's scoped deletions only.
@ranxianglei
ranxianglei force-pushed the 2026-10-26_4796_gc-batched-collection branch from 752982a to b37310a Compare October 26, 2026 05:15
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

第三轮:rebase 完成 + master 继承红定位

  1. rebase 完成:分支落后 master(tip 931c66a2f)57 个提交,已自行 rebase 并强推。新 head = b37310a92(两提交:4435f126b refcount 索引+分批 GC → b37310a92 lifecycle 扫描收敛)。冲突仅两处,都在常量声明点——master 新加的 SEQ_SPAN_QUERY_BATCH([PR] chore: empty PR (no changes) #4725 N3)与本 PR 的 GC_ZERO_REF_BATCH_SIZE / MYSQL_GC_ZERO_REF_BATCH 同位——union 解法保留双方,无语义改动。
  2. rebase 后本地验证:typecheck 干净;storage 定向 10 套件 = 93 pass / 2 fail / 1 env-skip。两条 fail 均在 tests/storage-unified.test.ts(FULL 模式读路径 digest 一致性),非本 PR 引入:干净 master checkout(931c66a2f)复现同样两条失败(确定性);bisect 钉死引入提交为 4e0c773e2("Merge branch 'master' into 2026-10-25_empty-pr",即 [PR] chore: empty PR (no changes) #4725 的分支内 merge)——它相对第一父只动 src/storage/mysql.ts(+24)、src/storage/store.ts(+36)与新增 tests/unified-remap-collision.test.ts(+186),tests/storage-unified.test.ts 本身未变。**master 在本 PR 合入前就已经红了。**完整证据已按 CI 协调政策留到 CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599:CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 (comment)
  3. CI 队列仍全仓卡死:master tip 自身 11/15 runs 停在 queued 无 runner 启动,本 PR 新 head 也在同一队列。属基建问题,非代码问题(CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 有两条状态评论)。

八问更新(rebase 后逐项复核,变化项加粗):

  1. 是什么问题:不变——孤儿 CAS 内容收不了尾(giant DELETE 挂 8 小时、WAL 钉死、面板 degraded)。
  2. 什么原因导致的:不变——缺 refcount 索引的全表扫 + 单条巨型自治提交。
  3. 影响范围与严重程度:不变——p1,存储只增不减失控类。
  4. 怎么修复的:不变——v9 索引 + 2000 行分批 select→delete→checkpoint + lifecycle 扫描收敛到本操作 hash 集。
  5. 是否有回归问题:rebase 后重新清点,结论不变——既有行为清单(lifecycle 原子性、dry-run 只读、releaseContent 零引用即删、迁移事务内建索引)全部保持;披露的行为变更仍只有 gc 提交模型与 archiveSession.contentRowsDeleted 语义两项,PR 描述已写明旧值→新值+原因。
  6. 是否破坏缓存:不涉及——不动消息身份/前缀缓存面。
  7. 是否改了设计方向:不涉及——GC 内部实现策略变更,接口与调用链不变。
  8. 是否引入了新的配置:不涉及——批大小为具名模块常量(2000/500),无新 env/config/CLI。

判定:代码面无阻塞问题。待 CI 队列恢复且除上述两条 master 继承红外全绿(或哨兵先把 master 修绿)即可合并;合并权在你。

ranxianglei added a commit that referenced this pull request Oct 26, 2026
The UNIQUE-collision repair deleted byte-identical blocker rows and their
media mappings WITHOUT releasing their content refcounts — a permanent
over-count, so those blobs could never be GC'd (#4800 delete-on-zero
only reclaims exact accounting). Same for the MySQL twin.

Found in the #4801 branch review (P1). Tests: unified-remap-collision.
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-26_4796_gc-batched-collection (22a7735)

Latest build — install this exact version

npm install -g billion-context@0.0.6279-pr.4800.4e5cde7

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
22a7735 billion-context@0.0.6279-pr.4800.4e5cde7 build
29c04d8 billion-context@0.0.6136-pr.4800.1f8f549 build
77e0b49 billion-context@0.0.6051-pr.4800.0ec3dc5 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4800.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

上游冲突已解(自动修复车道 attempt 1/2) —— 分支已合并最新 master 并推送,新 head 22a7735a9,无文本冲突。

冲突来源:上次 rebase 的基点(cdac1c099)之后,master 又合入 4 个 PR(#4807 ref-chokepoint、#4802 degenerate-429-governor、#4832 zstd-fork-gate、#4815 compress-breaker-enforcement),tip 到 05c379efd。这 4 个变更只动 retry/stream/preflight/dsh-native/kernel-sync 面,与本 PR 的 storage hunk 零文件重叠。

解法:merge origin/master 进 PR 分支(不用 rebase——分支内已有 ci-fixes 的 merge commit,rebase 会重写历史,无必要)。合并零冲突,PR 自身 diff 对 master 保持原样:仍是原两个提交的内容(storage/mysql.ts、storage/schema.ts、storage/store.ts + 5 个测试文件)。

新 head 本地验证(定向,全量交 CI):

结构化八问(冲突修复轮,未变项沿用第三轮结论)

  1. 是什么问题:本 PR 分支与目标分支产生合并冲突(dirty),需 rebase/merge 后恢复可合状态。
  2. 什么原因导致的:master 在分支上次 rebase 后又前进了 4 个 PR(重试治理/zstd fork 门/压缩熔断),基线漂移触发系统冲突告警;非代码语义冲突。
  3. 影响范围与严重程度:仅本 PR 的可合性受阻;对用户无影响(p3 级流程事件)。
  4. 怎么修复的:merge origin/master 进分支(保留既有 ci-fixes merge 历史,不做 rebase 重写),推送;合并前确认两侧文件零重叠,合并后确认 PR diff 未被污染。
  5. 是否有回归问题:无——零文本冲突意味着无语义合并决策;新 head 上重跑定向验证全绿,且顺带确认了第三轮遗留的两条 master 继承红已被 ci-fixes 修掉(该修复属 ci-fixes 哨兵职责,非本 PR 引入)。
  6. 是否破坏缓存:不涉及(同前几轮,CAS/refcount 纪律不变)。
  7. 是否改了设计方向:不涉及(同前几轮)。
  8. 是否引入了新的配置:不涉及(同前几轮,批大小仍为具名模块常量)。

判定:冲突已清,代码面无阻塞;待 CI lanes on 22a7735a9 全绿即可合并,合并权在你。

yanhuojiyusheng pushed a commit to yanhuojiyusheng/billion-context that referenced this pull request Oct 26, 2026
… declaration for lockfile-only churn

kernel/package-lock.json is generated state, not kernel source. Agents' test
prep (npm install inside kernel/, sweep-bot merges of stale lockfiles) churns
it without any intent to change the kernel — 8 PRs (ranxianglei#4823 ranxianglei#4808 ranxianglei#4802 ranxianglei#4800
ranxianglei#4799 ranxianglei#4794 ranxianglei#4790 ranxianglei#4785) tripped the human gate on lockfile-only diffs, and
the previous stopgap (bots appending kernel-change: declarations to PR
bodies) was correctly rejected by the owner as bypassing the approval step.

The human gate exists to protect kernel SOURCE. Rule 1 now ignores
kernel/package-lock.json when deciding whether a PR 'touches kernel/'; real
source/build-config changes still require the declaration + owner approval,
and lockfile CONSISTENCY stays enforced by the separate lock-sync job (which
fails on genuine drift — the ranxianglei#4725 merge red being today's live proof).
@ranxianglei
ranxianglei merged commit 6d6c7b1 into master Oct 30, 2026
44 of 47 checks passed
ranxianglei pushed a commit to elight/billion-context that referenced this pull request Oct 30, 2026
Conflict resolution notes:
- store.ts constructor: keep master's signature (dbPath last param, ranxianglei#4852
  integrity work); ranxianglei#4628's extra path param drops in favour of this.dbPath.
- gc() zero-ref sweep: keep ranxianglei#4800's index-driven batched collection
  (collectZeroRefContent, fixed 2000-row batches) instead of ranxianglei#4628's
  rowid-IN-subquery adaptive batches — with idx_content_refcount present,
  candidate selection is an index range scan and per-batch TRUNCATE
  checkpoints bound WAL growth. ranxianglei#4612's test asserts the actual log format.
- live-files.mjs: union of both sides' entries.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p1 scope/storage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

billion-context database cleanup records GC bug | billion-context 数据库清理记录 gc bug

1 participant