Skip to content

[PR] fix: tolerate JSON null content during storage identity backfill - #6055

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-09_fix-storage-null-identity
Nov 10, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-09_fix-storage-null-identity

Conversation

@Afloat16

@Afloat16 Afloat16 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Problem

Identity backfill treats parsed JSON null as an object, throwing TypeError and aborting SQLite migration.

Changes

Hash null/scalar/array content as its unchanged raw bytes while retaining normal message metadata stripping.

Regression coverage

Raw JSON null/scalar/array identities, normal positional-metadata stripping, and real SQLite v7→v11 migration with unchanged content bytes and stable reopening.

Validation

Passed against exact fork commit 6b0cb61a6ce69241faff00164e9195c85bb4e9fc in an isolated GitHub Actions checkout: validation job. The targeted regression and related suites, npm run typecheck, and npm run build all passed on Node 22. This is targeted verification, not the full upstream CI matrix.

node --import ./tests/preload-storage-baseline.mjs --import ./tests/preload-test-env.mjs --import tsx --test tests/storage-null-identity.test.ts tests/issue4677-v8-migration-streaming.test.ts
npm run typecheck
npm run build

Related work

Related #4672, #4684 and #4888 cover archive/migration/generation work; this is the null-content failure.

Closes #6045.

@ranxianglei ranxianglei changed the title fix: tolerate JSON null content during storage identity backfill [PR] fix: tolerate JSON null content during storage identity backfill Nov 9, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Dedup check: no duplicate. Searched open+closed issues and PRs for this failure signature (identity backfill + JSON null + storage): the only entries are bug report #6045 and this PR itself. The related work the author cites (#4672 archive/restore identity carry, #4684 streaming backfill, #4888 generation retirement) covers different mechanisms; none touches null-content tolerance.

Labels (judgment): bug · scope/storage · p1 — rationale: deterministic store-open failure for any user on schema ≤ 7 whose store holds even one "null" content row: the v8 backfill rethrows on every open and resumes into the same row forever (session-unrecoverable class). Note: the mirror's label API is not implemented in this environment, so they could not be applied programmatically; the judgment stands as recorded here.

Initial verification (targeted, local):

  • Reproduced on current master (= PR base 4be21b08ed): identityOfStoredContent("null") throws TypeError: Cannot read properties of null (reading 'id') at src/storage/identity.ts:93. The bug report is real, and the file is byte-identical between the reported commit and master, so the bug still exists on current master.
  • Root cause: identityInputOf assumes every parseable payload is an object; JSON.parse("null") yields null and the .id read throws. The PR's guard sits in the shared function itself — the producer of the bad behavior — so all 13 call sites (store / mysql / schema / ingest) inherit the fix. It mirrors the pattern splitMediaContent already uses one function up in the same file (if (parsed)).
  • The observable behavior delta is exactly one case: raw bytes that parse to JSON null change from throwing to hashing as unchanged raw bytes. Every other input (objects with/without id/wireIndex/ledgerSeq, scalars, arrays, non-JSON strings) returns byte-identical results old vs new — no existing row changes identity, no join/cache churn.
  • On PR head 6b0cb61a6c: new tests 3/3 pass; issue4677-v8-migration-streaming 5/5; related identity/migration suites (unified-remap-collision, storage-review-p2s, issue4796-gc-batched-sweep, issue4665-archive-restore-identity, issue4666-migration-dedup) 21 pass / 0 fail / 4 env-skipped; npm run typecheck clean; npm run build clean.
  • One out-of-scope observation: the live ingest path cannot itself produce a "null" row — serializeForStore/intern would crash earlier at the role read (src/storage/ingest.ts:193). Such rows must arrive via legacy/import data or direct DB writes, which makes consumer-side tolerance in the shared identity helper the right and sufficient fix for this thread.

CI (read from the mirror cache, fetched 15:20 UTC): 5 lanes green (gate ×2, docs-coverage, lane-guard, unused-exports), 14 test/static shards queued, 0 failures so far. Full-suite verdict pending — my final review conclusion follows once the shards finish.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI is now fully green (mirror cache, fetched 15:55 UTC): 30 checks — 18 success, 12 skipped (e2e lanes correctly not triggered by this diff), 0 failures. Master has since moved 111 commits past the PR base (4be21b08ed → 4446301708) but touches none of src/storage/identity.ts, src/storage/schema.ts, or the new test file — the branch applies cleanly, no conflicts, no concurrent semantic churn on these paths.

Review verdict: pass. Direction is right (the reported failure is real and reproduced on master), the fix lands at the root-cause layer (the shared identity function, not one call site), the behavioral delta is exactly the one crashing case, the tests have teeth (the migration e2e test fails red pre-fix), and the diff is minimal and clean (one commit, 2 files, +67/−2, no version/lockfile churn). Per §7.4 this stays in the human review tier because it touches storage identity (id logic) — no auto-merge.


结构化八问

  1. 是什么问题:schema ≤ 7 的旧版 SQLite 库在升级到 v8–v11 时,v8 identity 回填对每条消息的存储字节做身份计算;只要库里存在一条内容字节为字面量 "null"(JSON null)的消息,identityInputOf 就会在 null 上读 .id 抛 TypeError,迁移 catch 后 rethrow——该库此后每次打开都失败(可恢复设计会一直重试到同一行,永久卡死)。
  2. 什么原因导致的:identityInputOf(src/storage/identity.ts:86)假设「JSON.parse 成功 ⇒ 一定是对象」;JSON.parse("null") 返回 null,属性访问直接抛错。是函数自身的防御缺失。另查证:live ingest 路径自己产不出这种行(会在 src/storage/ingest.ts:193 更早崩),此类行只能来自 legacy/import 数据或直接写库——消费侧容错即完整修复,本线程内无生产者 bug 需修。
  3. 影响范围与严重程度:仅命中「从 schema ≤ 7 升级」且「库里至少有一条 null-content 行」的用户;一旦命中,统一存储整体无法打开、该库全部会话不可用 → p1。新装库与已迁完库不受影响(回填只跑一次)。
  4. 怎么修复的:在共享函数 identityInputOf 内加守卫——解析结果非对象(null/标量/数组)时直接按原始 raw bytes 哈希,不做元数据剥离(这些类型本来就没有 id/wireIndex/ledgerSeq 可剥)。13 个调用点(store/mysql/schema/ingest)同时受益;与同文件 splitMediaContent 既有的 if (parsed) 模式一致。修在函数层(行为的生产者),不在迁移调用点打补丁。
  5. 是否有回归问题:逐项审过该函数承载的全部既有行为——非 JSON 内容→raw 哈希(不变)、无位置元数据的对象→raw 哈希(不变)、含 id/wireIndex/ledgerSeq 的对象→剥离+重序列化(不变)、标量/数组 JSON→raw 哈希(不变);唯一变化是 JSON null 从抛错变为 raw 哈希。无语义漂移、无 fail-fast 变吞错、无阈值/时序/输出格式变化。本地 typecheck/build/定向套件全绿 + CI 全矩阵 18 绿/12 skip/0 红。结论:无回归。
  6. 是否破坏缓存:否。所有原本正常输入的 identity 值前后逐字节一致,存量行的 join key 不动;新写入的 null 行按既有约定取 sha256(raw)。
  7. 是否改了设计方向:不涉及。schema 版本、wire 协议、配置面、kernel 均未触碰。
  8. 是否引入了新的配置:不涉及。

测试有效性:三条新测试均为真测试——单元级断言 9 种非对象输入的 raw-bytes 哈希 + 元数据剥离契约的双向钉死(同内容不同位置戳同 identity、改内容必变);e2e 级构造真实 SQLite v7 库插入 null 行走完 migrateSchema 到 v11,断言 identity=sha256(raw)、内容字节不变、refcount 稳定、重开幂等。该测试在修复前必然红(迁移抛 TypeError)→ 合格的回归测试。

代码整洁度:3 行守卫 + 64 行测试,命名/引号/缩进与仓库一致,无死代码、无魔法值、无跑题改动。

收工判定:审核通过,可合并(human 档,请 owner 自行 merge:https://github.com/ranxianglei/billion-context/pull/6055)。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Closing summary (the substance is already on floor 3 and floor 5 above):

@ranxianglei
ranxianglei merged commit 3d35880 into ranxianglei:master Nov 10, 2026
48 checks passed
This was referenced Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Tolerate JSON null content during storage identity backfill

2 participants