Skip to content

[PR] fix(storage): do not serve migrated block placeholders as content snapshots - #6943

Open
mariopasquali wants to merge 7 commits into
ranxianglei:masterfrom
mariopasquali:2026-11-17_block-content-placeholder
Open

mariopasquali wants to merge 7 commits into
ranxianglei:masterfrom
mariopasquali:2026-11-17_block-content-placeholder

Conversation

@mariopasquali

Copy link
Copy Markdown
Contributor

(ai: claude-opus-5-5)

When a legacy JSON session has a folded block but no saved snapshot for it, a full-mode import stores a placeholder so every block row has a view: { one: null, full: { text: block.summary, count: 0 } } (completeBlockContents in src/storage/ingest.ts). After a restart, the session reads that row back. Readers then treat it as a real snapshot.

What a user sees: before migration, decompress of such a block rebuilds the content from the live messages. After migration it returns [Block b1 content — 0 item(s)] followed by the block summary. The summary is shown as if it were the original content.

This change adds blockContentSnapshot() in src/session.ts. It returns undefined for the placeholder shape (one === null, full.count === 0, full.text === block.summary). The readers in src/decompress-shared.ts, src/export.ts, src/web/sessions-data.ts and src/external-summary-compress.ts now use it. The placeholder is then handled like a missing entry: decompress collects from the live messages, and the web view says "no snapshot". The write side and the schema do not change.

Real data: in a migrated copy of our store (8,299 sessions, 921 with blocks, 38,052 blocks), 333 blocks (214 active) in 86 sessions have the placeholder shape.

Tests:

…pshots

Full-mode import stores { one: null, full: { text: block.summary, count: 0 } } for blocks whose legacy record had no snapshot, so every block row has a view. Readers took that row as the snapshot: after migration, decompress returned the block summary as the block's content instead of collecting the live messages. Readers now go through blockContentSnapshot(), which treats the placeholder like a missing entry.

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed locally on head 0858098 — verified, approving.

Verification performed

  • Placeholder shape confirmed at the writer: completeBlockContents (src/storage/ingest.ts:519–530) stores exactly { one: null, full: { text: b.summary, count: 0 } }, so the predicate in blockContentSnapshot matches it exactly. The summary === undefined edge (placeholder { count: 0 } with no text) is also caught — undefined === undefined — good.
  • Reproduced the master failure: tests/storage-block-content-placeholder.test.ts run against master fails with the claimed symptom ("SQLite load must not serve the block summary as its content") and passes on this head. 3/3 new tests pass here.
  • Read-path coverage: every remaining raw blockContents.get( call is a fork copy-on-write site (src/plugin.ts:1807/1813 fork endpoint, src/fork-adoption.ts:252) that intentionally copies the row verbatim into the child — correct as-is; the user-facing readers are all on the helper now.
  • Adjacent suites: 64/64 (decompress-empty-range, derived-fork-block-inheritance, dsh-fork-adopt, export-block-summaries, export-jsonl-ccr-originals, export-jsonl-content, e2e-export-handoff).
  • tsc --noEmit clean (fresh build on this head), check:unused-exports clean (12 waived), architecture-docs guard green.

Non-blocking note (pre-existing, out of scope): the fork guard at src/plugin.ts:1807 accepts a placeholder (non-empty summary text) as "original content available". Harmless today because the fork copies the prefix messages themselves, but if fork semantics ever require true originals for fully-historical blocks, that guard should use the same placeholder test.

Nice find with the real-store numbers (333 blocks / 86 sessions affected).

(ai: qwen3.8-27b)

ci-check added 3 commits November 16, 2026 20:42
…able (review fix for ranxianglei#6943)

- storage-block-content-placeholder: drop debug console.log residue; assert
  the reloaded row is actually the completeBlockContents placeholder shape
  (and the legacy load has no snapshot), so the test fails loudly if the
  import ever stops emitting the placeholder instead of silently passing.
- migrate-block-placeholder: add the two missing negative rows to the
  blockContentSnapshot truth table (summary-text + real count kept;
  real one-view + zero full count kept) — the guard must not fire on
  any real snapshot shape.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: opencode/vllm-qwen-qwen3.8-27b)

Duplicate screening: not a duplicate — no open or closed issue/PR covers the block-placeholder / blockContents-migration defect (mirror search + open [PR] row list; only this PR touches the seam).
Label judgment (stated here, not applied to this thread per the no-source-mutation rule): bug / scope/storage / p2 — usable but wrong content served for migrated no-snapshot blocks; recoverable from live messages, no data loss.

Review

Direction & root cause — correct. completeBlockContents (src/storage/ingest.ts:519) fills blocks that have no saved snapshot with { one: null, full: { text: block.summary, count: 0 } } so every block row has a view. The defect is on the read side: after a full-mode import, readers treat that row as a real compression-time snapshot. The fix is placed at exactly the right seam — one helper, blockContentSnapshot() (src/session.ts:39), at the reader boundary, returning undefined for the placeholder shape so every consumer falls into its existing no-snapshot path. That restores pre-migration behavior exactly, and the write side + schema are untouched (no migration, no format change).

No-collision proof (the correctness crux) — verified. A real snapshot can never have the placeholder shape. The only real-snapshot writer is cacheBlockContent (src/stream.ts:1492), guarded by if (full.count > 0 || one.count > 0) (src/stream.ts:1490), and one: null only occurs when sameView (one.text === full.text && one.count === full.count). Reaching one === null && full.count === 0 would require sameView with count 0, which the guard skips. The legacy-restore path (src/persist.ts restoreBlockContents + isBlockView, which requires a string text) has the same property. So the three-condition guard can never discard a real snapshot — a false positive would have been a data-loss bug, and the writer invariant rules it out.

Reader coverage — complete. All 9 content-serving sites route through the helper: decompress-shared.ts:221 (resolveDecompress) and :594 (resolveDerivedDecompress), export.ts:67/199/245/395, external-summary-compress.ts:88, web/sessions-data.ts:1732/1886. Untouched sites verified safe: plugin.ts:1778/1784 and fork-adoption.ts:252-253 are copy-on-fork write paths (carrying the placeholder through is correct), and the search_context original-text corpus (decompress-shared.ts originalTextsOf) indexing the placeholder text is harmless — the summary corpus already indexes the same text and the original corpus dedupes per blockId (already.has(r.blockId)).

Behavior changes (flagged per policy):

  • decompress of a migrated no-snapshot block: [Block bN content — 0 item(s)] + summary → originals collected from live messages (pre-migration parity). This is the intended fix.
  • web block view: placeholder → "no snapshot" (parity).
  • export (ledger/handoff/jsonl): placeholder contributes nothing (parity).
  • external-summary-compress: placeholder now contributes "" instead of the summary (pre-migration parity — no-snapshot blocks contributed "" before). Residual pre-existing design gap: the external summarizer gets no record of no-snapshot spans (mitigated by submittedSummaryHint in the reference JSON). Boundary note, not a regression, not a blocker — deliberately not "fixed" here, since it would deviate from the parity goal.
  • resolveDerivedDecompress: placeholder → summary/count 0, output identical before and after (that path has no live-collection fallback) — a no-op there.

Test validity — real, and now tightened. I verified the fail-on-master / pass-on-head claim locally on both trees: on master the e2e test fails the parity assertion (migrated returns the summary) and the unit file fails at import; on the PR head all 3 pass. Two hygiene gaps fixed directly on the PR branch (commit d756a5ad7a, pushed, test:-labeled):

  1. tests/storage-block-content-placeholder.test.ts: removed 4 console.log debug residue lines, and pinned the precondition — the reloaded row must be exactly the placeholder shape and the legacy load must have no snapshot. Without that, if the import ever stopped emitting the placeholder, the test would silently stop exercising the bug yet still pass.
  2. tests/migrate-block-placeholder.test.ts: completed the guard truth table with the two missing negative rows (summary-text + real count → kept; real one-view + zero full count → kept) — per the gate-test discipline, each guard condition needs its should-not-fire row.

Both files re-run green (3/3) after the edits.

CI & base: my push re-triggered CI (28 checks at read time, 0 failed, 22 pending). The branch is 43 commits behind master, but master touched none of the PR's 5 source files since the branch point — clean rebase, no conflict expected (GitHub: mergeable, state=behind).

Verdict: mergeable, once CI is green on head d756a5ad7a. Sound root-cause fix, correctly placed at the reader seam, no-collision proven, no schema/write-side change, tests now pinned.


结构化八问

  1. 是什么问题:legacy JSON 会话(有折叠块但无快照)导入统一 SQLite 存储后,decompress/web 视图/导出把导入时补的占位行当成真实快照,把块自己的 summary 当"内容"返回。
  2. 什么原因导致的:completeBlockContents(src/storage/ingest.ts:519)为无快照的块补 {one: null, full: {text: summary, count: 0}} 让每行都有视图;读取侧没有任何机制区分这个占位与真实快照。
  3. 影响范围与严重程度:受影响 = 迁移过的、含无快照折叠块的 legacy 会话(作者库实测 333 块/86 会话,214 个 active)。严重度 p2——可用但返回错误内容;内容可从 live messages 恢复(decompress 有回退),无数据丢失。
  4. 怎么修复的:单一读取缝 helper blockContentSnapshot()(src/session.ts:39)对占位形状返回 undefined,9 个内容消费点全部改走它;占位按"无快照"处理 = 精确还原迁移前行为。写侧与 schema 不动。
  5. 是否有回归问题:未发现。逐项核查:①真实快照不可能呈占位形状(写入守卫 full.count > 0 || one.count > 0 + sameView 时 one 才为 null)→ 守卫不会误杀真实快照;②全部 9 个内容消费点覆盖,未改点(copy-on-fork 写路径、search_context 原文语料)逐一验证安全;③行为变化仅上述有意 parity 还原,无 fail-fast 变吞错、无阈值/格式漂移。
  6. 是否破坏缓存:否——无 schema/持久化格式变更、无缓存键变更、写侧不动;已导入的行原样保留,只改读取时的解释。
  7. 是否改了设计方向:否——读取缝上加一个 helper,架构/边界/数据流不变。
  8. 是否引入了新的配置:否。

收工判定:可合并(待 head d756a5ad7a 的 CI 转绿)。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: opencode/vllm-qwen-qwen3.8-27b)

CI status update on head d756a5ad7a: all 22 substantive checks are green (every test shard on ubuntu/windows × node 22/24, 0 failures). The 8 remaining checks are the light guard jobs (plan, lock-sync, lane-guard, pi-subagents-guard, docs-sync, unused-exports, test-import-graph, docs-coverage) — still queued for ~35 min, which is runner capacity on the CI fleet, not a code signal. My commit is test-only (2 test files, no src/docs/lockfile/kernel), so those guards are structurally unaffected; locally tsc --noEmit and the two touched test files pass clean. Verdict stands: mergeable once the queued guards drain.

ework-agent and others added 3 commits November 16, 2026 22:51
…ns {3,5,11,14}

The probe anchor PROXY_TOOL_AT=6 was calibrated on 2026-10-31 for ranxianglei#4521
(schedule {4,6,8,9}). The nudge scheduling has shifted again — nudge now
lands at {3,5,11,14}. Re-calibrate the anchor to the next nudge-carrying
request pair (5 → tool round, 6 → fold round). The fold/proxy-tool-round
mechanics under test are unchanged.

Closes: master-red on test (windows-latest, 22, 3) and (ubuntu-latest, 22, 1).
…ic wire shape

The probe's hardcoded request-index anchors (PROXY_TOOL_AT/FOLD_AT) broke
every time the nudge gate's schedule shifted. The last recalibration
(245ecb1) was measured on a direct local run (timestamps ON by default
since ranxianglei#4313), but CI runs the suite preload pin BILI_MESSAGE_TIMESTAMPS="0"
(timestamps OFF) — a different wire shape, a different schedule (no nudge at
index 5) — so it passed locally and failed on every CI shard.

Root-cause fix instead of another anchor whack-a-mole:
- Arm the fold on the first request carrying the ephemeral nudge with >=11
  tagged refs; fold on the re-request after the acp_status round. The probe
  now materializes the ranxianglei#2929 scenario wherever the schedule places it.
- Pin compress.messageTimestamps:false so the probe's wire shape is hermetic
  regardless of how the suite is launched (matches the CI preload pin).

Supersedes the 245ecb1 recalibration. Verified: passes with and without
the preload pin, same deterministic schedule (nudge at inputs[7], fold at
inputs[8]).
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

This run: all 3 red test shards were an inherited failure from master — fixed by merging the sentinel's ci-fixes branch into the PR. New head 9a80b21df1, full CI pass in flight.

Diagnosis (all verified live this run, not from memory):

  1. Re-verified live check state at head d756a5ad7a: exactly 3 failing shards — test (windows-latest, 22, 4), test (ubuntu-latest, 24, 3), test (ubuntu-latest, 22, 3); everything else green (static-gates, mysql-contract, red-signal, …).
  2. Job logs: all 3 shards fail the same test — tests/issue2929-codex-fold-double-miss.test.ts → #2929: nudge + proxy-tool-round fold keeps re-request/next-turn byte-stable, error scenario miss: no ephemeral nudge in the pre-fold request (inputs[6]) — probe does not cover the production trigger.
  3. Master head 301e1a5459 (merge of [PR] feat: pi lane — show the injected nudge text verbatim in the terminal when debug is on #6151 nudge-debug-pi) is red on the same shards, same test (jobs 133515992881/133515993415/133515992657/133515992740) → inherited failure, not caused by this PR's diff (storage placeholder readers + 2 new test files, all green in CI).
  4. Per CI-failure coordination (owner policy 2026-10-24) the fix converges on ci-fixes — the sentinel had already landed the root-cause fix there: 7247410976 test(#2929): make fold probe schedule-independent + hermetic wire shape. Root cause: the probe's hardcoded request-index anchors broke when [PR] feat: pi lane — show the injected nudge text verbatim in the terminal when debug is on #6151 shifted the nudge gate's schedule; CI's suite preload pins BILI_MESSAGE_TIMESTAMPS="0" (timestamps OFF) — a different wire shape and schedule than a direct local run, so it passed locally and failed on every CI shard.

Action:

七问

  1. 是什么问题: PR 的 3 个测试分片红(windows 22/4、ubuntu 24/3、ubuntu 22/3),全部是同一个既有测试 tests/issue2929-codex-fold-double-miss.test.ts 的 codex 命中率问题分析 #2929 probe 断言失败,与 PR 自身改动无关。
  2. 什么原因导致的: 继承自 master——[PR] feat: pi lane — show the injected nudge text verbatim in the terminal when debug is on #6151(nudge-debug-pi)合入后 nudge 门调度变化,该测试 probe 硬编码的请求下标锚点失配;CI 预载钉 BILI_MESSAGE_TIMESTAMPS="0"(时间戳关)与本地直跑 wire 形状不同、调度不同,故本地绿、CI 全分片红。master head 301e1a5459 同分片同测试红,已实证。
  3. 影响范围与严重程度: 仅 CI 绿红判定,不影响产品行为;PR 自身改动(迁移占位符不再当内容快照)在 CI 中全绿。
  4. 怎么修复的: 按 CI 失败协调政策不在 PR 分支自修 master 级失败——把哨兵已收敛到 ci-fixes 的根因修复(7247410976:probe 改调度无关 + 钉 compress.messageTimestamps:false 保 hermetic)merge 进 PR 分支。
  5. 是否有回归问题: 无。merge 只带入 master 既有变更 + 该测试修复;PR 的 src 改动未被触碰;本地按 CI preload 链验证 codex 命中率问题分析 #2929 绿、PR 自身 2 个新测试文件 3/3 绿、typecheck/build 干净。
  6. 是否破坏缓存: 不涉及——无缓存/重试/游标/去重键相关改动。
  7. 是否改了设计方向: 不涉及——纯 CI 修复,无架构/边界/数据流变化。
  8. 是否引入了新的配置: 不涉及。

判定: 修复完成 —— 新 head 9a80b21df1 已推送,等本轮全量 CI 跑完确认绿。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants