Repository navigation
[PR] Index content's FK child columns — releaseContent DELETE stops full-scanning children - #5540
Conversation
…er full-scans (#5539) With foreign_keys=ON, deleting a parent content row must prove no child row still references it; SQLite proves that by scanning each child table BY ITS REF COLUMN — an index lookup when indexed, a full table scan when not. Six columns reference content(hash) (messages.content_hash, blocks.one_hash, blocks.summary_hash, ccr_entries.content_hash, message_media.media_hash, checkpoints.state_hash) and none carried an index, so every releaseContent DELETE paid six full scans. Busy sessions release ~3K hashes per turn — field report #5539: 20-35 s/turn at ~18K messages, 4-5 s after adding the keys. Schema v11 (sqlite + mysql): - STORAGE_DDL_V1 / MYSQL_DDL carry the six new indexes for fresh installs - migrateSchema from<11 block: idempotent CREATE INDEX x6 (v9 discipline) - mysql init(): STATISTICS probe widened to all six tables + probe-first ALTER ADD INDEX x6 - version-pin assertions in seven existing suites moved to the new head Local repro (20K messages, one transaction, exact releaseContent shape): 0.210 ms/delete before (plan: SCAN x6) vs 0.012 ms/delete after (plan: SEARCH USING INDEX x6); FK enforcement itself unchanged (a referenced parent still throws FOREIGN KEY constraint failed).
The field report measured per-call db.prepare() at ~7x the cost of the statement itself (0.063 vs 0.009 ms/call) — releaseContent runs once per released hash (~3K/turn on busy sessions), so the prepare overhead is a real share of the hot path even after the index fix removes the scans. Statements are now prepared lazily once per UnifiedStore instance and reused. Safe because each store owns exactly one connection for its whole life (private readonly db, closed in close(), never re-opened on the same instance). Behavior unchanged: same SQL, same ordering, same FK semantics.
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.8245-pr.5540.e77458aThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5540.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。 这是 #5539(字段报告:releaseContent 每回合 ~3K 次父行 DELETE × 6 张无索引子表全表扫描,~18K 消息会话 20–35 s/turn)的修复 PR。扫了全部 open PR:唯一触碰同一机制(content 的 FK 子列建索引 / schema v11)的就是本 PR;#5250(second-writer degrade)是写者争用域,机制不同。无并列在途 PR。 标签: 现在做三件事,随后按八问体例给完整 review 结论:① 逐段读 diff(schema/store/mysql/测试)并核对既有行为;② 本地定向跑新回归测试 + 相关 storage 套件;③ 查 CI 状态。 |
Review nit (#5542): the releaseContent statement-cache hunk accidentally re-indented the untouched class field lastIntegrityOk from 4 to 9 spaces. Revert to 4 so the diff carries no incidental whitespace changes.
Review nit (#5542): the v10 -> v11 bump updated asserts but left three comments claiming v10 state that no longer holds — a fresh install now runs the v11 DDL, and the idempotence arms reopen v11-stamped databases.
(ai: vllm-qwen/qwen3.8-27b) Review 结论:可合(等最终 head 结构化八问
审中我直接推了两个小修到 PR 分支(独立标注 commit,小问题直接修):
测试有效性:新回归套件带双向门(#3617 纪律)——先断言未加索引的 v10 计划确实全扫(负向对照),再断言 EXPLAIN QUERY PLAN 命中全部六个索引且无 CI 状态:写作时对 pre-nit head 判定:可合,等 |
(ai: vllm-qwen/qwen3.8-27b) 收工小结(完整八问 review 见上条评论):
判定维持:可合,等 test shard 全绿后人工合并(schema 版本 bump,§7.4 人工档)。 |
…ap case (#5539) The six FK-child indexes added by schema v11 append index-maintenance frames to every insert on messages/blocks/ccr_entries/message_media/checkpoints. In the #3250 custom-cap sequence the post-reset tail therefore grew from 54.3 KiB locally / 66.4 KiB on CI (pre-v11) to 130.8-146.9 KiB (1182472-1198952; identical 1198952 on ubuntu-24 shard 3), past the 128 KiB budget. Same drift class as #4506 and #4796/#5193 (documented in-test): every checkpoint reset still truncates exactly to the configured cap (verified per-ingest on both trees), so the inter-reset budget widens 128 -> 256 KiB with provenance; the bound stays far below the ~4.4 MiB default-limit plateau the assertion discriminates against.
…ual (#4599) 01fcc5d (#5656/#5584, merged via #5687) diverts tool-less dsh single-shot requests onto the verbatim side lane (kernel no-touch); the residual for main turn 1 is documented in the commit's own comment. Five pre-existing pins assumed turn 1 rides the full pipeline and went born-red at merge (master red since; every PR head inherits it). Test-only realignment codifying the documented behavior: - issue2419/2446/2447/2481 guard-stamp subtests: two turns; the BLOCKED ref is witnessed on the first FULL-PIPELINE turn (turn 2). No fold can happen before turn 2 (nothing compressible, no action surface), so protection against every possible fold is unchanged. - issue2817: stats.requests off by one (turn 1 uncounted); the persona-fold teeth are intact, assertions shifted -1. - issue5403 A/B golden: turn 0 rides the side lane in BOTH phases (the gate ignores the rollout env), so rb[0] is unstamped by design; stamps resume at turn 1. Byte equality holds at i=0 (both phases forward verbatim). No src/, kernel/, config, or version changes. Locally green: all six files + side-lane-invariants + issue5656-single-shot-sidecar + dsh-persona-fingerprint.
(ai: vllm-qwen/qwen3.8-27b) 结论先行:本次红是继承性失败(master 自身红),收敛修复已在分支上(另一车道先于本车道完成合并);本轮工作 = 核实修复正确性 + 确认新 head 全绿,未重复推送。 核实链(证据):
七问:
判定:CI blocker 已清除,PR 就绪待 owner 合并。备注:镜像 check 缓存截至我读取时仍显示旧 head(fetched_at 10:50Z),若后续再有指向旧 sha 的「仍红」唤醒,按陈旧缓存处理即可。 |
(ai: vllm-qwen/qwen3.8-27b) Review verdict: APPROVE(可合) — human merge(§7.4:schema 版本 bump 属持久化格式层,不 auto-merge)。Head 独立审核说明:未照抄前轮结论——自己重读了全 diff(schema.ts / store.ts / mysql.ts / 新测试 / 全部 pin 更新)、枚举了 DDL 里所有 结构化八问:
测试有效性:新测试有牙——前置条件先断言未索引计划确实 SCAN(模拟不复现 bug 就红),再断言迁移后 EXPLAIN QUERY PLAN 命中全部六个索引且无任何 行为变更清单(披露):
未发现需要顺手修的小问题(storage-unified.test.ts:843 标题「schema v10」命名的是被测功能代际,与「schema v2」等 house style 一致,断言已正确指向 v11,保留)。 判定:APPROVE / 可合,人工合并(persistence-version 层)。 |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Review lane 收工确认:PR #5540(head f502174)已 end-to-end 独立审核完毕,完整八问审核结论见上方评论——verdict:APPROVE(可合),人工合并(§7.4 persistence-version 层)。要点:六个 FK 子列索引覆盖完整(枚举 DDL 验证)、v10→v11 纯增量迁移幂等、releaseContent SQL 逐字不变且语句缓存安全、新测试双向有牙、回归清点无破坏、行为变更已披露(6 全表扫描→6 索引查找;20–35s→4–5s/turn)。CI 镜像缓存全绿(30 checks,0 failed,mysql-contract success);本地定向套件 + typecheck 绿。无顺手修项,无 kernel/ 触碰。协调行已报 #4599。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.1044-master.bfabcd1 # this exact build
npm install -g billion-context@master # always the newest merged build
|
…anxianglei#4542/etc (conflict resolution for ranxianglei#5743) # Conflicts: # tests/opencode-rolling-cache-marks.test.ts
…dexes and the v11 stamp sweep master advanced 9a2d51d -> bfabd1ca with six commits from ranxianglei#5539/ranxianglei#5540: the releaseContent FK child indexes, cached prepared statements, the v11 stamp re-sweep, and a WAL-tail budget for the ranxianglei#3250 custom-cap case. One textual conflict, in tests/issue4666-migration-dedup.test.ts: this lane pins the mysql stamp to STORAGE_SCHEMA_VERSION (added when ranxianglei#4796 left one pin behind), while master re-swept the same lines to the literal "11". Kept the derived form - it survives the next version bump, which is the whole reason this lane introduced it - and refreshed the stale ranxianglei#4796 comment to mention ranxianglei#5539 as well.
Closes #5539
What and why
With
foreign_keys=ON, deleting a parentcontentrow requires SQLite to prove no child row still references it — it does that by scanning each child table BY ITS REF COLUMN: an index lookup when indexed, a full scan when not. Six columns referencecontent(hash)—messages.content_hash,blocks.one_hash,blocks.summary_hash,ccr_entries.content_hash,message_media.media_hash,checkpoints.state_hash— and none carried an index, so everyreleaseContentDELETE paid six full scans. Busy sessions release ~3K hashes per turn (field report #5539: 20–35 s/turn at ~18K messages).This adds all six indexes as schema v11, following the #4800 precedent (
content(refcount), v9): idempotent creates in the existing migration transaction for existing libraries, DDL lines for fresh installs, plus the mysql-side equivalent.Sixth index beyond the field audit: #5539 enumerates five child columns;
checkpoints.state_hashis the same mechanism missed there (it is also scanned on every parent delete). Same failure domain, so it ships in this fix instead of a split.Persistence surface (§7.4 flag)
from < 11step (idempotentCREATE INDEX IF NOT EXISTS).ALTER TABLE … ADD INDEXinside the existingcurrent < MYSQL_SCHEMA_VERSIONgate; stamp usesGREATEST(…)so it never downgrades. Fresh installs get theKEYlines from the DDL.MySQL honesty note: the mysql DDL declares NO
FOREIGN KEYconstraints, so InnoDB performs no FK-presence checks on delete — mysql users do not exhibit this pathology today. The mysql keys are parity + future-proofing (if FKs are ever declared), costing nothing and keeping the two backends' index sets aligned.Behavior change disclosure (old → new + why)
releaseContentper-parent-delete plandb.prepare()calls inreleaseContentLocal repro (node:sqlite, 20K content rows + six child tables, exact
releaseContentshape, single transaction): BEFORE 0.210 ms/delete, plan =SCAN×6; AFTER 0.012 ms/delete, plan =SEARCH USING INDEX×6 (≈17×). FK enforcement verified unchanged both directions (referenced parent still throws; after the child is gone the delete succeeds).Tests
tests/issue5539-fk-content-indexes.test.ts(gate discipline — both directions): fresh install has v11 + all six indexes and the DELETE plan showsSEARCH USING <index>per child and noSCAN; a library stamped v10 WITHOUT the indexes (precondition asserts the plan actually scans) migrates to v11 with the improved plan on reopen; FK enforcement still rejects deleting a referenced parent.Coexistence
src/storage/store.tsis a hot file (#3987/#4242 lineage touchesreleaseContent); the second commit is a self-contained lazy-cache inside that one method and stands independently of the schema commits.