Skip to content

[PR] fix: export CCR originals instead of reference metadata - #6727

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_jsonl-ccr-originals
Nov 14, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_jsonl-ccr-originals

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Closes #6726

Problem

A full JSONL export should replace a CCR placeholder with its stored original. The table renderer currently uses getCcrEntries() as if each entry's content were that original. planIngest() actually stores serialized reference metadata there: hash, rawId, kind, token/character counts, and a short head. The original text lives under MessageContentStore.byHash.

As a result, a real persisted session exports metadata JSON as the message text and adds dereferencedFrom, even when the actual original is available in both the unified mirror and its content-store sidecar. The incorrect indexed entry also prevents the correct sidecar fallback from filling the reference. This affects full JSONL handoffs; the legacy-tail renderer already resolves the hash correctly.

Change

Resolve indexed originals through the existing backend getCcr() contract: look up each reference's hash in byHash, and include only actual string payloads. An empty original remains a valid empty string. Missing, corrupt, or null reference data leaves the placeholder intact unless the sidecar supplies a real original.

Use the same extraction for the sidecar, so a missing hash cannot be mistaken for an empty original. Preserve indexed-original priority, fill only gaps from the disk copy, and keep plain exports/non-placeholder text verbatim. This changes the generated export only; it adds no storage format or setting and does not rewrite persisted data.

Verification

  • 14 new tests use real SQLite ingestion, including public SessionStore → exportSession() calls with the sidecar present and after it is removed.
  • Unchanged base: 12 failed / 2 controls passed. Actual output contains the reference metadata JSON instead of the expected original bytes.
  • Tests verify exact Unicode/newline/quoted bytes, valid empty text, missing hashes, missing/null refs, a corrupt mirror, disk fallback and indexed priority, a missing-byte sidecar, non-full exports, noncanonical placeholder-like text, the existing legacy path, and unchanged stored CCR data.
  • 34/34 related tests pass across the new module, JSONL exports, output-file handling, CCR storage, and the MySQL backend seam.
  • Full npm run typecheck, npm run build, and git diff --check pass.
  • Independent read-only review checked the complete diff, all fourteen cases, planIngest()'s actual entry format, and getCcr() validation; no blocker. The reviewer did not rerun tests.
node --import ./tests/preload-test-env.mjs --import ./tests/preload-storage-baseline.mjs --import tsx --test --test-concurrency=1 tests/export-jsonl-ccr-originals.test.ts tests/export-jsonl.test.ts tests/export-jsonl-output.test.ts tests/storage-ccr.test.ts tests/storage-mysql-seam.test.ts

Validation used Node 24.22.1, TZ=UTC, and an isolated temporary directory. SQLite/public export paths ran locally. The existing MySQL seam tests use a typed injected backend; no live MySQL server was tested.

History check

Checked the complete Afloat16 issue index and actual related PR patches, including a fresh read of #6209's current files. That change only implements output destinations. Actual #3010 adds the unified CCR mirror and getCcr() for retrieval but does not change this exporter; #3946 adds MySQL export selection. Focused all-state searches found no previous correction of the metadata-versus-original lookup.

This is independent of the separately submitted historical-null guard: that fixes an exception and whole-export fallback. This patch fixes the payload source used for healthy object placeholders.

Base: master 3858b6956abab52da781c2436c0245586ce18358, bili 0.1.214. Full-suite status is the hosted CI on the exact pushed head.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 Searched open+closed issues/PRs for the export/CCR family: #3010 (closed) added the unified CCR mirror + getCcr() but never touched this exporter; #6209 only added output destinations; #6725 (open) is a different mechanism — it fixes the renderer throwing on JSON null rows (exception → legacy-tail fallback), while this PR fixes the payload source for healthy placeholders. No prior correction of the metadata-vs-original lookup exists.

Independent review — verified, not taken on faith:

  • Bug is real, reproduced on base. planIngest() (src/storage/ingest.ts:460-465) stores JSON.stringify(byRef[key]) — the reference metadata — as each ccr_entries content, and the old renderer (src/export.ts:304) fed that straight into message.text. I ran the 14 new tests against base 3858b6956a: 12 failed / 2 controls passed — exactly the PR's claim. The two passing controls (verbatim non-placeholder text; legacy-tail path) are the right ones.
  • Fix verified on head. Same 5-file targeted group on 24f6d073: 34/34 pass. npm run typecheck clean on the head tree. CI on the exact head sha: 48/48 green (36 success + 12 skipped, 0 failed) — confirmed live via check-runs.
  • Code. ccrOriginals() is correct and matches the kernel's own retrieval semantics (store.byHash[store.byRef[ref].hash], kernel/src/content-store.ts:113-115). The entry?.hash + typeof payload === "string" guards keep valid empty strings and drop missing/null refs. Both call sites receive shape-validated stores (getCcr in src/storage/store.ts:3568-3584 and src/storage/mysql.ts:1733-1750; sidecar via src/persist.ts:814-816). Switching to getCcr stays inside the StorageBackend contract, so the MySQL export path (src/export.ts:425) gets the same fix. ccr is only read under the full guard (src/export.ts:317), so skipping the query for non-full exports is safe. The ccr_entries table is still written and still used for content dedup/GC refcounts — nothing orphaned.
  • Tests are real. Actual SQLite ingestion, exact-byte assertions (Unicode/CRLF/quotes), both gate directions (dereference when present; placeholder preserved across missing-hash / missing-ref / null-ref / corrupt-mirror), indexed-over-disk priority in both directions, public SessionStore → exportSession() paths with the sidecar present and removed. This is a true regression test — the pre-existing test (tests/export-jsonl.test.ts:207) only asserted "non-empty", which is why the metadata JSON slipped through for so long.
  • Coordination note (not a blocker): [PR] fix: preserve null historical rows in full JSONL exports #6725 touches adjacent lines in the same function; whichever merges second needs a trivial rebase.
  • State: PR is 35 commits behind master, but master has not touched src/export.ts or these test files since the base — no conflict expected (mergeable: true).

8问 (PR review):

  1. 是什么问题: 全量 JSONL 导出时,CCR 占位符被替换成了引用元数据 JSON(hash/rawId/kind/tokens/chars/head),而不是原始文本,且仍打上 dereferencedFrom 标记。
  2. 什么原因导致的: 渲染器把 getCcrEntries() 的 content 当成了原文,但 planIngest() 写入该表的是 JSON.stringify(byRef[key])(引用元数据);原文实际存放在 MessageContentStore.byHash 里。错误的索引条目还挡住了 sidecar 兜底。
  3. 影响范围与严重程度: 所有走统一索引的 full JSONL 导出(会话交接)受影响——导出的消息文本是元数据 JSON,交接方拿到的是错误内容;会话本体、存储、在线压缩均不受影响,legacy-tail 路径本来就正确。属于导出产物错误,非数据丢失,定 p2 合理。
  4. 怎么修复的: 改走既有 getCcr() 契约取完整 store,按 byRef[ref].hash → byHash 解析出真正的原文(与 kernel 自身 retrieve 语义一致);sidecar 用同一提取逻辑;只收 string 载荷(空串合法),缺失/损坏/null 一律保留占位符;索引优先、磁盘补缺的次序不变。
  5. 是否有回归问题: 审过的既有行为:①非 full 导出——输出逐字节不变,少一次无用的 getCcrEntries 查询(改进);②full 导出正常占位符——由元数据 JSON 变为原文(本修复);③原文缺失时——由导出元数据 JSON 变为保留占位符(改进);④sidecar 缺失 hash——行为变更:旧代码 ?? "" 会导出空字符串,新代码保留占位符(PR 正文已披露,方向正确:空串是假原文,保留占位符才是诚实输出);⑤ccr_entries 表与 getCcrEntries 方法——仍被写入、仍用于内容去重/GC 引用计数,未孤儿化;⑥legacy-tail 渲染器未动。无无意回归。
  6. 是否破坏缓存: 不涉及——纯导出渲染路径,不触碰缓存/重试/增量游标/去重键;存储数据不重写(测试断言导出前后 getCcr 逐字节不变)。
  7. 是否改了设计方向: 不涉及——没有新存储格式、新配置、新依赖;只是把导出器对准既有的 getCcr() 契约,与 legacy-tail 路径语义对齐。
  8. 是否引入了新的配置: 不涉及。

判定: 可合入。 单模块纯 fix,回归测试在 base 上真实复现(12 红),head 上 34/34 绿,CI 全绿,diff 干净(2 文件 1 commit,每行都与目的相关),Closes #6726 在位,§7.4 自动合并门禁 8 条全过。合并动作留给人工。

@ranxianglei
ranxianglei merged commit 90b019c into ranxianglei:master Nov 14, 2026
48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Full JSONL exports replace CCR placeholders with reference metadata instead of original text

2 participants