Skip to content

[PR] fix: preserve null historical rows in full JSONL exports - #6725

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_jsonl-historian-null
Nov 14, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_jsonl-historian-null

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Closes #6724

Problem

A stored historical message can contain valid JSON null. The identity/migration path deliberately preserves such raw content, and a plain JSONL export already retains it. With --full, the table renderer assumes every parsed value is an object and reads message.text.

That throws on a null row. The public export path catches the exception as an unreadable unified index and emits the shorter legacy-tail fallback. A readable SQLite index therefore loses its historical rows from the generated export, including retired rows that the historian export is meant to retain.

Change

Keep parsed content typed as unknown and inspect its text only when it is a non-null object. Nulls, scalars, arrays, and the existing invalid-JSON {raw: ...} representation remain in their original output records. Actual object placeholders keep their existing CCR dereferencing behavior. The database and stored content are unchanged.

Verification

Eleven new tests use the real SQLite store. The public-path case persists a live session, retains an older null row in the historian, then calls exportSession() with JSONL/full/output-file options. It verifies the unified source, both historical/current rows, exact retiredAt, the written output, and unchanged database rows.

Other cases cover padded null, booleans, numbers, strings, arrays, objects without string text, malformed JSON, the non-full control, and a subsequent valid CCR placeholder.

  • Unchanged base: 4 failures / 7 controls passed. The public-path failure really changes source to legacy-tail; the renderer failures are Cannot read properties of null (reading 'text').
  • Fixed: 30/30 tests passed across the six related JSONL/export/identity modules, including the existing JSON-null migration regression.
  • Full npm run typecheck, npm run build, and git diff --check passed.
  • Independent read-only review covered the complete production diff, all eleven cases, and the upper-level fallback, with no blocking finding. The reviewer did not rerun tests.
node --import ./tests/preload-test-env.mjs --import ./tests/preload-storage-baseline.mjs --import tsx --test --test-concurrency=1 tests/export-jsonl-content.test.ts tests/export-jsonl.test.ts tests/export-jsonl-output.test.ts tests/storage-null-identity.test.ts tests/export-block-summaries.test.ts tests/export-uncompressed.test.ts

Tests ran with TZ=UTC and an isolated writable temporary directory on Node 24.22.1. No live MySQL server was used; the production change stays inside the shared backend-neutral renderer.

History check

Checked Afloat16's 127-issue index and actual 111-PR patch index plus current submissions. #6230 concerns scanning archive files during restore, and #6209 only makes JSONL honor output paths. Actual upstream #2768 handles an unreadable index, #3432 introduces the backend seam, #4670 includes retired rows, and #5076/#5084 optimize session selection. None guards this full-export content access. Focused all-state searches found no prior report or patch for this failure.

Base: current master 3858b6956abab52da781c2436c0245586ce18358, bili 0.1.214. Full-suite results remain the hosted CI on the exact pushed head.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Duplicate screening: Not a duplicate. #6725 is the only open PR fixing #6724's symptom (a JSON-null historical row degrading the full JSONL export to the legacy-tail fallback). Same-author #6727 ("export CCR originals instead of reference metadata") is a different mechanism — it changes what a CCR placeholder resolves to, not how a non-object row is handled. No other open issue/PR targets this access.

Direction (①): Correct, and it fixes the root cause at the decision point. The producer of the offending access is renderSessionJsonlFromTables (src/export.ts:308-320), which JSON.parses each stored row's m.content. A valid JSON null parses to null, and the old message.text read threw Cannot read properties of null (reading 'text'). The outer export (src/export.ts:465-481) catches that as an "unreadable unified index" and degrades to the shorter legacy-tail renderer — silently dropping the historical/retired rows the historian export exists to keep. Guarding the text access before dereferencing removes the throw at the source; it is not a mask (the parsed value is still emitted verbatim; only the .text probe is type-guarded).

Regression (②): I inventoried every existing behavior of the touched path. Only one changes, and it is the intended fix:

  • non-full: full false → guard skipped → unchanged.
  • full, object with string text → guard true → CCR dereference as before → unchanged.
  • full, object with no / non-string text, plain scalars, arrays, {raw:…} invalid-JSON → guard false → skip → unchanged (the old code also skipped these, since their .text was undefined).
  • full, null row → OLD threw → degraded to legacy-tail (data loss); NEW preserves the row as null. This is the fix.
  • The sibling .text read at src/export.ts:379 (renderSessionJsonlFromSession) iterates s.lastMessages (CoreMessage[], in-memory typed objects), not JSON.parsed strings — it cannot be null, so it is correctly out of scope.

Code (③): The type change Record<string, unknown> → unknown is more honest (JSON.parse can return any JSON value). The guard message !== null && typeof message === "object" && "text" in message && typeof message.text === "string" narrows correctly; the ph! assertion and the { ...message, … } spread are unchanged from the original and typecheck clean.

Tests (④): 11 real-SQLite tests (no mocks). I verified teeth both ways: on the base src/export.ts the new file fails exactly as the PR claims (4 failures / 7 controls — 2× Cannot read properties of null (reading 'text'), 1× public-path source flipping to legacy-tail, 1× CCR-after-null), and on the PR head all 30 tests across the six related modules pass. The public-path case asserts source === "unified" (the exact degradation signal), exact retiredAt, and unchanged DB rows; the CCR case asserts object placeholders still dereference after a non-object row (the "does NOT trigger" direction). Both gate directions are covered.

Diff (⑤/⑤b): 2 files, +140/−3, every line on-purpose; no unrelated changes, whitespace churn, or lock-file churn. External-submission strict standard passes: honest naming, no dead code, no duplication, error handling preserved, no magic numbers, style matches the file, no new dependencies, single-responsibility, clean tests.

CI: Green on the pushed head f7d64eaf — 36 success, 12 skipped (the gated e2e lanes), all test shards (ubuntu/windows × node 22/24), static-gates, mysql-contract, and every guard (kernel-guard, version-guard, lock-sync, unused-exports, config-surface-guard, architecture-docs, …) pass. PR is on current master 3858b695 (not stale), mergeable: true.


结构化八问

  1. 是什么问题:可读的 SQLite 索引里有一条历史消息内容是 JSON null,跑 --jsonl --full 导出时渲染器读 message.text 抛 TypeError,被外层当成"索引不可读"降级成更短的 legacy-tail,把本应保留的历史/retired 行丢掉了。
  2. 什么原因导致的:renderSessionJsonlFromTables 把每行 m.content 做 JSON.parse,合法 JSON null 解析成 null,旧代码无类型守卫直接读 .text 抛错;错误被上层 catch 误判为索引损坏 → 降级。(根因在渲染器的类型假设,不在存储。)
  3. 影响范围与严重程度:仅影响"可读索引 + 历史行内容为 JSON null + --full JSONL 导出"这一组合;触发面窄(需 rewind 产生 retired 的 null 行),但触发时该会话的历史行从导出中静默丢失。属特定条件下的功能异常,非不可用 → p2。
  4. 怎么修复的:把解析结果类型从 Record<string, unknown> 改成 unknown,只在"非 null 对象且带 string text"时才做 CCR 解引用探针;null/标量/数组/坏 JSON 一律原样输出。不动存储、不动真正不可读索引的降级行为。
  5. 是否有回归问题:无。逐条清点触碰路径既有行为:非 full、带 string text 的对象、无 text 的对象、标量、数组、坏 JSON 全部保持原行为;唯一变化是 null 行不再抛错/不再降级——正是本 PR 目的。src/export.ts:379 的同类 .text 读的是内存 CoreMessage[](非 JSON.parse),无此 bug,正确不纳入。
  6. 是否破坏缓存:不涉及。导出是只读生成物,不碰缓存/重试/游标/去重键。
  7. 是否改了设计方向:不涉及。无架构/边界/数据流转向,仅修正渲染器一处类型守卫。
  8. 是否引入了新的配置:不涉及。无新 env/config/默认值。

判定:可合并。这是一处根因定位准确、范围恰当、测试有牙(双向)、CI 全绿、diff 干净的高质量外部贡献,无阻塞项。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Full JSONL export falls back to the live tail when a historical message is JSON null

2 participants