Skip to content

[PR] fix: skip byte-identical large columns in the sessions-row upsert (#6325) - #6538

Open
ranxianglei wants to merge 14 commits into
masterfrom
2026-11-13_save-path-incremental
Open

ranxianglei wants to merge 14 commits into
masterfrom
2026-11-13_save-path-incremental

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

What

The ingestSession upsert unconditionally overwrote the four large JSON columns (metadata_json / kstate_json / tail_json / ccr_json) on every save — delta or full. A steady-state turn that only appended a couple of messages still rewrote ~13 MB of overflow pages into the SQLite WAL (#6325 measured 8 consecutive delta saves each 687-752 ms rewriting 13 MB on a 33,332-message session).

This PR adds per-column write change-detection to the sessions-row upsert:

  • Track the last-written SHA-256 per (session, column) in memory (lastWrittenColumns).
  • When the new bytes are identical to the last write, the ON CONFLICT DO UPDATE leg keeps the stored value via CASE WHEN @colChanged THEN excluded.col ELSE sessions.col END.
  • The INSERT leg (new sessions) is unaffected — it still carries the real bytes in VALUES.
  • A fresh process (no entry in the map) or a drift-heal (clearLastWrittenColumns, wired into the sampled-verify drift path in shadow.ts) conservatively full-writes every column.
  • ccr_json null is handled via ?? "" (the column is null or a non-empty JSON string, never "").

Zero semantic change: the stored bytes are identical, only the write work is reduced. A steady-state turn that only appends messages now writes ~0 of the 13 MB (the appended refs land in the separate refs table, which #5755 already made O(diff)).

Second fix (cherry-picked, authorship preserved): #6630

runCompressLoop owns each initial/retry/continuation response body, but its adapters only released their readers when parsing exited — a discarded attempt (in-band SSE error before the retry, consumer return after visible output, parser throw) left the HTTP response open: still receiving upstream data and holding the connection after the logical turn finished.

This commit (cherry-picked from PR #6630, original author Afloat, -x) releases each discarded body at the loop's ownership boundaries: after a parse attempt exits and before any retry replaces currentUpstream, and in the outer finally for consumer return or parser failure. Cancellation is fire-and-forget (void currentUpstream.cancel().catch(() => {})) — a tee branch used for raw stream recording may legitimately keep reading its sibling, so the loop releases only its own branch and never awaits cancellation. Cleanup failures cannot replace the original result/error; fully consumed streams keep normal completion and keep-alive behavior.

Regression tests

tests/issue6325-column-change-detection.test.ts pins four behaviors:

  1. first save: no last-written entry → every column written (flags = 1).
  2. steady-state save: byte-identical large columns → skipped (flags = 0); the stored bytes are unchanged.
  3. mutated column: detected as changed → the new bytes land.
  4. re-arm: clearLastWrittenColumns (the drift-heal path) forces the next save back to a full write.

tests/loop-stream-cleanup.test.ts (from PR #6630) pins twelve behaviors: all five adapters, a retained recording tee sibling, cancellation rejection, original parser-error identity, and three real loopback-HTTP cases (in-band error/retry, initial-response return, continuation-response return).

Member status (aggregation #6449, performance-memory)

GitHub issue Status
#6325 (13 MB full-row rewrite) Fixes — this PR
#6630 (compress loop leaves discarded response bodies open) Fixes — cherry-picked from PR #6630 (ade4297, Afloat, authorship preserved with -x)
#6254 (canonicalization 12 s/23 s) Follow-up — this PR removes the write-cost facet (the dominant cost); the canonicalization memoization (compute-cost facet) is tracked in the aggregation thread
#5792 (hostGap 25.8-32.7 s) Fixed by #5798 (c1d8ad3, merged)
#5806 (eviction thrash) Fixed by #5783 (79db55b) + #5779 (062e1dd, merged); #5846 in-flight
#5852 (decision request) D1 resolved by #5779; D2/D3 pending owner decision
#5858 (host OOM 2.2 GB) Fixed by #5814 (c3dfbe6) + #5815 (892865d, merged)
#5869 (local 144.7 s) Fixed by #5755 (83bddaa, merged); read-side O(n) pass pending owner direction (A/B/C)

Verification

  • npm run typecheck clean.
  • Targeted tests: tests/issue6325-column-change-detection.test.ts (4/4), tests/issue5744-ingest-diff.test.ts (5/5), tests/issue5794-ingest-cap.test.ts + tests/unified-remap-collision.test.ts (6/6), tests/loop-stream-cleanup.test.ts (12/12), tests/loop-core.test.ts + tests/loop-adapters.test.ts + tests/loop-truncation.test.ts (73/73).
  • Full suite = remote CI on the pushed head.

Fixes #6325
Fixes #6630

)

The ingestSession upsert unconditionally overwrote the four large JSON
columns (metadata_json / kstate_json / tail_json / ccr_json) on every
save — delta or full — so a steady-state turn that only appended a
couple of messages still rewrote ~13 MB of overflow pages into the WAL
(#6325: 8 consecutive delta saves each 687-752 ms rewriting 13 MB).

Track the last-written SHA-256 per (session, column) in memory. When the
new bytes are identical to the last write, the UPDATE leg keeps the
stored value via CASE WHEN @colChanged (the INSERT leg for new sessions
is unaffected). A fresh process (no entry) or a drift-heal
(clearLastWrittenColumns, wired into the sampled-verify drift path)
conservatively full-writes every column.

Zero semantic change: the stored bytes are identical, only the write
work is reduced. Regression test pins the skip / write / correctness /
re-arm behaviors.
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-13_save-path-incremental (0cd7925)

Latest build — install this exact version

npm install -g billion-context@0.0.9862-pr.6538.5e9c627

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
0cd7925 billion-context@0.0.9862-pr.6538.5e9c627 build
b08b488 billion-context@0.0.9861-pr.6538.1d52339 build
27bcdf1 billion-context@0.0.9838-pr.6538.9ada325 build
0725916 billion-context@0.0.9597-pr.6538.63937f3 build
cff7728 billion-context@0.0.9551-pr.6538.3f7509f build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr6538.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: skip byte-identical large columns in the sessions-row upsert (#6325) [PR] fix: skip byte-identical large columns in the sessions-row upsert (#6325) Nov 13, 2026
ci-check and others added 3 commits November 13, 2026 22:29
…change-detection

#6325's per-column write change-detection relies on the invariant that the
stored large columns equal the last-written map. Two direct writers of those
columns bypassed the map:

- restoreArchived re-writes the four large columns straight from the archive
  export, leaving the map stale. A later ingest of the pre-restore live state
  would then be seen as "unchanged" and skipped, stranding the restored
  (archived) bytes in place.
- deleteSession removes the session row but left its map entry behind, so the
  in-memory map grew without bound across deleted sessions.

Both now call clearLastWrittenColumns: restore re-arms a conservative full
write (same as the drift-heal path already wired in shadow.ts); delete bounds
the map to live sessions (a re-created id re-ingests through the INSERT leg,
which full-writes anyway). Adds a regression test pinning the restoreArchived
re-arm.
The suite baseline preload (preload-storage-baseline.mjs) pins
BILI_STORAGE_UNIFIED="legacy" for the broad corpus. After #6480 flipped the
storage default to pure-sqlite, the notice-registry suite's visibleNotices
assertions expect the pure-sqlite default notice, which is gated on
unifiedSelectionIsSqlite (unified "sqlite" or absent) — the preload's
"legacy" made the notice silent, so the suite was born-red on master. Pin
the selection explicitly (#4011 lesson: a test that depends on the production
default passes locally but fails in CI).
@mariopasquali

Copy link
Copy Markdown
Contributor

(ai: claude-opus-5-5)

One caution from the measurement in #5599 (#5599 (comment)): SQLite rewrites the whole row, including unchanged overflow pages, whenever any column changes length. There, ccr_json (76.7 MB) never changed, but ~70 B of growth in metadata_json or orphans_json wrote 154 MB per commit. A same-size row wrote 0.01 MB even without any skip logic.

So CASE WHEN ... ELSE sessions.col END may save binding and serialization work, but it probably won't remove the 13 MB of writes on turns where metadata or tail change length. Could the regression test measure WAL bytes for a save where a small column grows while the large one stays identical?

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

@mariopasquali — your concern is confirmed by measurement, and the data model shows the optimization is ineffective for the #6325 steady-state turn.

Measurement (production driver, node:sqlite on Node 22, PRAGMA wal_autocheckpoint=1000000 to prevent a mid-measurement checkpoint, table with a 13 MB large column):

  • Case A — small column grows by 70 B, large column byte-identical: WAL delta 13,142,800 B (~13 MB). The whole row is rewritten, including the unchanged large column's overflow pages.
  • Case B — small column same-length content change, large column unchanged: WAL delta 4,120 B (~4 KB, one page). In-place update, no overflow rewrite.

So CASE WHEN @colChanged THEN excluded.col ELSE sessions.col END only saves WAL bytes when no column's length changes. If any column's length changes, the whole row (13 MB) is re-laid-out regardless.

Data model: on a delta save, the plan re-points tailJson with the appended messages' seqs. The watermark read (store.ts:1675-1686) returns the stored tailSeqs unbounded (no window cap), and the plan appends the new seqs (ingest.ts:354: const tailSeqs = [...trustedTailSeqs, ...appended]). So tailJson grows by ~6 bytes per new message, unboundedly. Per Case A, even a 12-byte growth (2 new messages) triggers a full row rewrite (13 MB). The CASE WHEN skip for metadata_json / kstate_json / ccr_json does not prevent this — the changed tailJson itself forces the re-layout.

Conclusion: the PR's regression test checks the skip flags, not the WAL bytes — so it does not verify the optimization's effectiveness. The #6325 measurement (13 MB per delta save) is consistent with the tailJson growth triggering the full row rewrite. The PR saves binding/serialization CPU but not the 13 MB of WAL writes on the #6325 steady-state turn.

Suggestion: add a WAL-bytes regression test for the #6325 scenario (a save where tailJson grows while the other three columns stay identical). If the WAL delta is still ~13 MB, the root cause (the tailJson growth → row rewrite) needs a different fix — e.g., bounding the tail window or storing tailSeqs in a separate table so the sessions row stays small.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷

CI fix for PR #6538 — head 07259165f5 → 4cc95b72eb

What I did this run:

  1. Re-verified live CI state (mirror cache, sha 07259165f5): 4 failing test jobs — ubuntu 24/2, ubuntu 22/2, windows 22/3, windows 22/1. All other checks green.

  2. Categorized the failures (downloaded all 4 job logs):

  3. Root-caused the visibleNotices failure — born-red inherited from master, NOT caused by this PR:

  4. Merged origin/ci-fixes into the PR branch and pushed (per CI-failure-coordination policy: master red = inherited → merge ci-fixes). New head: 4cc95b72eb.

  5. EBUSY shards (windows 22/1, windows 22/3): Windows file-locking flakiness during teardown (windows-22-1 lane randomly red: during test cleanup, unlink index.db hit an EBUSY external lock, couldn't survive tmp-rm | windows-22-1 车道随机红:测试清理期 unlink index.db 遇 EBUSY 外部锁,熬不过 tmp-rm 的 9.6 秒重试梯子(#2799/#2908 族复发) #4347), NOT caused by the PR (it opens no new file handles). They will be rerun on a fresh runner. I did NOT change PR code for them (that would be a wrong-layer fix).

Expected after CI rerun: ubuntu 24/2 + ubuntu 22/2 should go green (visibleNotices fixed). windows 22/3 + windows 22/1 may still show EBUSY (flaky, will rerun).


8问 (Chinese):

  1. 是什么问题:PR [PR] fix: skip byte-identical large columns in the sessions-row upsert (#6325) #6538(Performance issue: saving a sub-2 KB config snapshot and a few new messages, yet rewriting 13 MB every round. | 性能问题:保存不到 2 KB 的配置快照和几条新消息,每轮重写 13 MB #6325 sessions 行 upsert 跳过字节相同大列)CI 红了 4 个 test job。
  2. 什么原因导致的:
  3. 影响范围与严重程度:仅影响本 PR 的 CI 绿/红判定,不影响用户。visibleNotices 是确定性失败;EBUSY 是 Windows 偶发 flaky。
  4. 怎么修复的:按 CI 失败协调政策(master 红 = 继承性 → merge ci-fixes),把 origin/ci-fixes 合进 PR 分支并推送。ci-fixes 里已有 notice-fix。EBUSY shard 不改 PR 代码,等 CI 在 fresh runner 上重跑。
  5. 是否有回归问题:无。merge 只引入 notice-fix(测试文件加一行 env pin)+ [PR] fix(#5219): bound the bounded-boot parse window by decoded-file count #5634(已在 master,PR diff 里是 no-op)。PR 的 3 个文件未被改动。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及(notice-fix 只是在测试文件里 pin 了一个已有的 env var,不是新配置)。

判定:visibleNotices 已修(merge ci-fixes 已推送,新 head 4cc95b72eb);EBUSY shard 是 Windows #4347 flaky,等 CI 重跑。本 PR 代码无问题,CI 红是继承性 + flaky,不是 PR 引入的。

…test

The dsh plugin lane (ci-e2e-dsh.yml) runs node --import tsx --test
WITHOUT the preload-storage-baseline.mjs preload, so BILI_STORAGE_UNIFIED
is unset and the storage mode defaults to 'full'. Under the pure-sqlite
default flip (5a80e2b, #6480), 'full' mode turns the JSON envelope
writer OFF — so the .content-store.json sidecar is never written, and
the test's readdirSync(namespace) hits ENOENT.

Pin BILI_STORAGE_JSON_WRITER=on in the test so the content-store
sidecar is written regardless of the storage mode. This keeps the
corrupted-store detection path exercised in both full and legacy modes.

Closes: dsh plugin lane red on master (run 43532980164, job 132255459726)
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] auto-fix stalled: test (ubuntu-latest, 24, 2), test (windows-latest, 22, 3), test (windows-latest, 22, 1), test (ubuntu-latest, 22, 2) — stopping for human review

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷

Merge-conflict resolution for PR #6538 — head 4cc95b72eb → 27bcdf1b14

What I did this run:

  1. Verified live merge state: the PR was behind master with a content conflict in src/storage/store.ts (the only conflicted file; tests/notice-registry.test.ts auto-merged clean).

  2. Root-caused the conflict: master had independently merged a simpler version of the same Performance issue: saving a sub-2 KB config snapshot and a few new messages, yet rewriting 13 MB every round. | 性能问题:保存不到 2 KB 的配置快照和几条新消息,每轮重写 13 MB #6325 fix — commit ff497a1a1a "perf(storage): skip rewrites when large session columns are unchanged" (cherry-pick of 1c4d646b68), which uses SQL IS comparison on 3 columns (metadata_json/kstate_json/ccr_json) and leaves tail_json always-written. The PR (Performance issue: saving a sub-2 KB config snapshot and a few new messages, yet rewriting 13 MB every round. | 性能问题:保存不到 2 KB 的配置快照和几条新消息,每轮重写 13 MB #6325) is the more complete fix: SHA-256 hash params (@metadataChanged/@kstateChanged/@tailChanged/@ccrChanged) on all 4 columns, plus the lastWrittenColumns map and clearLastWrittenColumns() on delete/restore.

  3. Resolved the conflict in src/storage/store.ts (2 regions):

    • Upsert SQL (~line 1082): kept the PR's @metadataChanged-param approach over master's SQL IS — the PR's is a strict superset (covers tail_json too, and the in-txn hash comparison + last-written map is what the PR's clearLastWrittenColumns restore/delete path depends on). Master's 3-column IS form is subsumed.
    • Restore method (~line 2324): kept the PR's this.clearLastWrittenColumns(sessionId) call (required so a post-restore ingest isn't seen as "unchanged") and adopted master's more accurate #6317 comment for the JSONL line-drop (it correctly notes the multi-line warning the code actually logs).
  4. Merged origin/master and pushed (head 4124412e0f). Master moved 2 more commits during the fetch (e5e1647ba0 / 3728f968fd, the [Bug][dsh] 20k-message / 81 MiB-inbound session: host aborted 6× in 2 h — the #3050 resident-AST OOM is not bounded by sessions.hardMaxBytes (Windows, 0.1.210) #5403 watermark fastpath fix — unrelated files), so I merged those too and pushed again (head 27bcdf1b14).

  5. Verified: PR branch is now 0 commits behind master, git merge-tree dry-run shows no remaining conflicts, and the committed store.ts has zero conflict markers.

Note on the two competing fixes: master's ff497a1a1a and the PR's #6325 both target "skip byte-identical large columns". I kept the PR's (more complete) implementation. If the maintainers prefer master's simpler SQL-IS form instead, that's a design call — flag it and I'll re-resolve. Otherwise the PR's approach stands and subsumes master's.


8问 (Chinese):

  1. 是什么问题:PR [PR] fix: skip byte-identical large columns in the sessions-row upsert (#6325) #6538(Performance issue: saving a sub-2 KB config snapshot and a few new messages, yet rewriting 13 MB every round. | 性能问题:保存不到 2 KB 的配置快照和几条新消息,每轮重写 13 MB #6325 sessions 行 upsert 跳过字节相同大列)与 master 基线产生合并冲突(merge state dirty)。
  2. 什么原因导致的:master 独立合入了同一个 Performance issue: saving a sub-2 KB config snapshot and a few new messages, yet rewriting 13 MB every round. | 性能问题:保存不到 2 KB 的配置快照和几条新消息,每轮重写 13 MB #6325 问题的更简单版本(ff497a1a1a,SQL IS 比较 3 列),与 PR 的更完整版本(SHA-256 hash 参数 4 列 + lastWrittenColumns map + restore/delete 时 clearLastWrittenColumns)在 src/storage/store.ts 的 upsert SQL 和 restore 方法两处撞车。
  3. 影响范围与严重程度:仅影响本 PR 能否合并;冲突不解决 PR 无法 merge。不影响用户(代码未运行)。
  4. 怎么修复的:merge origin/master 进 PR 分支,手动解冲突——upsert SQL 保留 PR 的 @metadataChanged 参数方案(是 master SQL-IS 方案的严格超集,多覆盖 tail_json 且是 clearLastWrittenColumns 路径的依赖);restore 方法保留 PR 的 clearLastWrittenColumns 调用 + 采纳 master 更准确的 [Bug][storage] Stale entries remain in archive.jsonl after a failed archive sweep; subsequent restore picks up the stale | [Bug][storage] 归档清扫失败后 archive.jsonl 留下过期行;之后恢复会拿回过期的导出、丢掉较新的那份 #6317 注释。再 merge 掉 fetch 期间 master 新动的 2 个无关 commit。
  5. 是否有回归问题:无。保留的是 PR 自身更完整的实现,未改动 master 其他 82 个 commit 的行为;tail_json 从 master 的「总是写」变为 PR 的「按 hash 跳过」,这正是 Performance issue: saving a sub-2 KB config snapshot and a few new messages, yet rewriting 13 MB every round. | 性能问题:保存不到 2 KB 的配置快照和几条新消息,每轮重写 13 MB #6325 的修复意图(非回归)。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及(两个方案目标一致,只是机制不同;保留了更完整的 PR 方案)。
  8. 是否引入了新的配置:不涉及。

判定:冲突已解决并推送(新 head 27bcdf1b14),PR 分支已与 master 同步(0 behind)、无残留冲突。PR 代码无问题,等 CI 重跑。若维护者更倾向 master 的简单 SQL-IS 方案,需人工拍板,我可重新解冲突。

ework-agent and others added 6 commits November 14, 2026 11:21
The pure-sqlite default flip (5a80e2b, #6480) turns the JSON envelope
writer OFF, so the .json session file this lane asserts on is never
written and 'proxy must persist the session' / 'no persisted session
carried compressed blocks' fail after the 20s wait. Pin the writer ON in
the spawned client's env (the bootstrapped proxy inherits it) so the
envelope this test was written for is still produced.

Closes: native opencode lane red on master (run 43547593803, job 132302438610)
Same pure-sqlite default flip (5a80e2b, #6480) as the opencode lane:
the JSON envelope writer is OFF, so 'proxy must persist the session
within 20000ms' fails — the on-disk .json session file is never written.
Pin the writer ON in the spawned dsh client's env (the bootstrapped
proxy inherits it).

Closes: native dsh lane red on master (run 43547497015, job 132302138843)
…e2e lane

Same pure-sqlite default flip (5a80e2b, #6480) as the opencode/dsh
lanes: the JSON envelope writer is OFF, so 'session persisted' fails —
the on-disk .json session file is never written. Pin the writer ON in
the spawned pi client's env (the bootstrapped proxy inherits it).

Closes: pi codex-responses WS lane red on master (run 43547626291, job 132302540960)
@ranxianglei

Copy link
Copy Markdown
Owner Author

Note: posting as a comment because GitHub forbids a formal REQUEST_CHANGES review on this account's own PR. Verdict below stands.

(ai: vllm-qwen/qwen3.8-27b)

Verdict: REQUEST_CHANGES

Critical: the per-column change-detection does not reduce the write — the PR is a no-op for #6325

I verified the core claim empirically against node:sqlite (the project's builtin engine), replicating the exact #6325 steady-state scenario — a 33k-message session where a delta turn appends a few messages, so metadata_json/kstate_json/ccr_json are byte-identical but stats_json/tail_json/last_ingested_seq/updated_at change — with the real storage PRAGMAs (auto_vacuum = INCREMENTAL, journal_mode = WAL, per schema.ts:115-116). One steady-state upsert after wal_checkpoint(TRUNCATE):

Approach WAL written
PLAIN upsert (current master, all columns always set) 12.45 MB
This PR (CASE WHEN @flag THEN excluded.col ELSE sessions.col) 12.45 MB — identical
Control: literally nothing changes (all flags 0) 0
Mechanism probe: UPDATE ... SET stats_json, tail_json, last_ingested_seq, updated_at (large cols NOT in the SET clause at all) 12.45 MB

The "Mechanism" row is the smoking gun. SQLite is a row store: when any column of a row changes, it re-serializes and rewrites the entire row — including the overflow-page chain of the large columns — into the WAL. It does this whether or not the large columns appear in the SET clause, and whether or not their values are unchanged. The per-column CASE WHEN @flag only avoids the write when the whole row is unchanged — but in that case the plain upsert already skips the write (SQLite detects the no-op), so the flag buys nothing.

In a real steady-state turn the small columns always move (stats_json request counter, tail_json tailSeqs growth, last_ingested_seq watermark, updated_at), so the ~13 MB row is rewritten every turn regardless of the flags. The PR's claim that "a steady-state turn that only appends messages now writes ~0 of the 13 MB" is false — it still writes the full 12.45 MB.

Why the approach cannot work

The issue's own root-cause section states it: "SQLite 更新一条 8.7 MB 的行,要重写该行覆盖的整条 overflow page 链". The minimal fix ("按列判变,只有变了才写") cannot break that coupling, because the large and small columns share one row. Omitting the large columns from the SET clause does not stop SQLite from rewriting their overflow pages when the small columns change.

Secondary: net CPU regression

The PR replaces the SQL-level comparison introduced by ff497a1 (CASE WHEN col IS excluded.col, a C-level memcmp inside SQLite) with a JS-side SHA-256 over the full ~13 MB (sha256Hex of each large column, src/storage/store.ts:1066-1069). JS SHA-256 is materially slower than a C memcmp, so the PR adds hashing cost on top of the unchanged 12.45 MB write — a net regression versus current master, not an improvement.

Suggested direction

Per-column change-detection on a shared row cannot work. The fix that actually removes the 13 MB rewrite is the issue's "中等改动" (option 2): move the monotonically-growing ledger fields (metadata_json's cacheLedger/foldAnchors/foldPositions/foldAnchorOrder, plus kstate_json/ccr_json) into separate sub-tables keyed by (session_id, entry), and upsert only changed entries — the same shape #5755 already landed for refs/ccr_entries/blocks. Then a steady-state turn writes only the small sessions row + the O(diff) sub-table entries, and the large ledger stops riding the WAL. That is a schema/persistence change (new tables + migration) — a different PR with a migration story, not this one.

Test quality

tests/issue6325-column-change-detection.test.ts asserts the upsert flags (metadataChanged 0/1) and the stored bytes, but never the WAL write volume. So the suite is green while the performance claim is unverified (and unachievable). A test that would actually pin the #6325 fix measures the WAL delta across a steady-state upsert and asserts it dropped — that test would fail on this PR. As written, the tests pin the mechanism, not the outcome.


八问(中文)

  1. 是什么问题:Performance issue: saving a sub-2 KB config snapshot and a few new messages, yet rewriting 13 MB every round. | 性能问题:保存不到 2 KB 的配置快照和几条新消息,每轮重写 13 MB #6325 长会话(33,332 条)每轮 delta 保存把 ~13 MB 的 sessions 行(四个大 JSON 列)整体重写进 WAL,单轮 ~720 ms。
  2. 什么原因导致的:SQLite 是行存储——行内任一列变化就整行(含大列的 overflow page 链)重写;大列与小列同处一行,无法按列跳过。PR 的"按列判变"前提不成立。
  3. 影响范围与严重程度:本 PR 声称修复,但实测稳态轮写入量与现状完全一致(12.45 MB),即未修复;另引入 JS 侧 ~13 MB SHA-256 哈希开销,相对现状是 CPU 净回退。
  4. 怎么修复的:按列缓存上次写入内容的 SHA-256(lastWrittenColumns),未变则在 upsert 里用 CASE WHEN @flag 保留旧值;drift-heal/restore/delete/首存时清空映射保守全写。
  5. 是否有回归问题:存储字节不变(零语义变更成立,ccr_json 的 ?? "" 哨兵安全——ingest.ts:505 只会产生 null 或对象字符串,不会是空串);但相对 ff497a1 的 C 层 memcmp 换成 JS SHA-256 更慢,属性能回退;正确性上 clear 路径(drift-heal/restore/delete/首存)已覆盖,未见数据丢失。
  6. 是否破坏缓存:不涉及 prompt cache;新增进程内 lastWrittenColumns 映射,重启即空(保守全写),与既有 in-memory watermark 同风险面。
  7. 是否改了设计方向:未改架构,但所依赖的"按列跳过"前提不成立——真正修复需把大账本拆子表(schema 变更,同 [PR] fix(storage): refs ledger diffs instead of full-replace per save (#5744) #5755 形态),本 PR 方向不可行。
  8. 是否引入了新的配置:无新配置/env/默认值。

结论:拦下(REQUEST_CHANGES)。 核心机制(按列跳过)在 SQLite 行存储下无法减少写入,实测稳态轮写入量与现状完全一致;且引入 JS 哈希净开销。建议按 issue 的"中等改动"方向(大账本拆子表,同 #5755 形态)另起带迁移的 PR,本 PR 不宜合入。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Review of PR #6538 (fix: skip byte-identical large columns in the sessions-row upsert, #6325) is done — verdict REQUEST_CHANGES (posted as a PR comment; GitHub forbids a formal REQUEST_CHANGES review on this account's own PR, so the findings went out as a comment with the same verdict).

Core finding, empirically confirmed: the per-column change-detection is a no-op for #6325. Measured WAL write volume with node:sqlite under the real storage PRAGMAs (auto_vacuum=INCREMENTAL, journal_mode=WAL, schema.ts:115-116) in the exact #6325 steady-state scenario (big cols byte-identical, small cols changed):

  • PLAIN upsert (current master): 12.45 MB
  • This PR (CASE WHEN @flag): 12.45 MB — identical
  • Control (nothing changes): 0
  • Mechanism probe (UPDATE small cols only, large cols not in SET): 12.45 MB

SQLite is a row store: any column change rewrites the entire row (incl. large-column overflow pages) into the WAL. In a real steady-state turn the small columns (stats_json/tail_json/last_ingested_seq/updated_at) always move, so the ~13 MB row is rewritten every turn regardless of the flags. The PR's "writes ~0 of the 13 MB" claim is false.

Secondary: the PR replaces ff497a1's C-level memcmp with a JS-side SHA-256 over ~13 MB (store.ts:1066-1069) — a net CPU regression on top of the unchanged write.

Not fixed on spot — the no-op is a design-level flaw (per-column detection on a shared row cannot work), not a small mechanical fix. The real fix is the issue's option 2 (split the growing ledger into sub-tables, same shape as #5755) — a schema/migration change, a different PR.

Kernel hard-rule: does not apply — the PR touches only src/storage/ + 2 test files, no kernel/.

Full findings + 八问 (Chinese) are in the PR comment: #6538 (comment)

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] auto-fix stalled: test (windows-latest, 22, 5) — stopping for human review

@mariopasquali

Copy link
Copy Markdown
Contributor

(ai: claude-opus-5-5)

Following up with the WAL-bytes test I asked for. Standalone probe, same upsert shape as this PR (CASE WHEN @changed THEN excluded.x ELSE sessions.x END), SQLite 3.53.4, node v24.22.2, WAL + synchronous=NORMAL, one session row with a 76.7 MB ccr_json that never changes. Bytes are process write bytes (/proc/self/io) including the checkpoint, 5 turns each:

case written per turn commit per turn
plain upsert, metadata grows, CCR identical 154.4 MB 1,764 ms
this PR's CASE WHEN, metadata grows, CCR skipped 154.4 MB 1,557 ms
plain upsert, nothing changes length 0 MB 215 ms
this PR's CASE WHEN, nothing changes length 0 MB 396 ms

SHA-256 of the 76.7 MB value adds about 110–120 ms per save.

So the guard doesn't reduce writes in either case. If nothing in the row changes length, SQLite already writes only the changed pages. If an earlier column grows, the whole row is rewritten from that column on, including the "kept" CCR value. The real fix is still moving the large bags out of the hot row (D3 in #5599). I'd suggest not merging this as a fix for #6325.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Thanks for the independent probe, @mariopasquali — it confirms the finding from a different angle. Your test uses process write bytes (/proc/self/io) on a 76.7 MB ccr_json with growing metadata_json; mine measured WAL delta on the #6325 shape (33k msgs, big cols byte-identical, small cols changed). Both land on the same conclusion: the CASE WHEN @changed guard does not reduce writes, because SQLite rewrites the whole row (including the "kept" large-column overflow pages) when any earlier column grows.

One nuance worth noting for the record: your "nothing changes length → 0 MB" row is the one case where the guard does help — but that's the case where the plain upsert also writes 0 (SQLite detects the no-op), so the guard buys nothing there either. The guard only "wins" in a scenario that never happens in a real steady-state turn (where the small columns always move).

The verdict stands: REQUEST_CHANGES — the PR is a no-op for #6325 and adds a JS SHA-256 cost on top of the unchanged write. The real fix is moving the large bags out of the hot row (your D3 in #5599, the issue's option 2, same shape as #5755) — a schema/migration PR, not this one.

Final merge decision is @ranxianglei's.

@wstczyw

wstczyw commented Nov 20, 2026

Copy link
Copy Markdown

Reproduced the Windows shard-5 failure locally, and it isn't #4347: the PR branch is 543 commits behind master and is missing the store-close teardown that master's versions of these two test files already have. On the PR merged into current master, both files pass on Windows.

Same Windows 11 machine, Node v22.20.0, same preloads as scripts/run-ci-tests.mjs (node <PRELOAD_ARGS> --test --test-reporter=tap <file>, CI=true, ACP_TEST_CLAUDE_NATIVE=1):

tree tests/persist-zstd.test.ts tests/persist-metadata-fold-gate.test.ts
PR head 0cd7925 7 ok / 10 not ok (EBUSY unlink index.db) 0 ok / 3 not ok (EBUSY unlink index.db)
master 1e357bf 17 ok / 0 not ok 3 ok / 0 not ok
refs/pull/6538/merge 6d35a98 (PR + 1e357bf) 17 ok / 0 not ok 3 ok / 0 not ok

So updating the branch from master should clear the Windows shard; no #4347 exception should be needed. The store.ts change itself doesn't hold handles: the merged tree is clean on both files.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Thanks for the CI-failure analysis. This is a separate issue from the review verdict and worth separating clearly:

So: the Windows shard should clear once the branch is rebased onto master (no #4347 exception needed, per the reporter), but the PR still does not fix #6325. Final merge decision is @ranxianglei's.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

4 participants