Skip to content

[PR] fix(storage): count active blocks without loading content - #6626

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-13_count-active-blocks
Nov 14, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-13_count-active-blocks

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Fixes #6624.

The drift-latched cold-load path compares the index and envelope active-block counts to choose the fuller compression history. Its scalar helper currently hydrates every block and decodes both CAS views before counting, including inactive block content. Large retained summaries therefore add avoidable work before the actual recovery load.

Changes

  • Count active rows directly in SQLite and use that narrow operation in UnifiedShadow.activeBlockCount().
  • Preserve exact active = 1 semantics, session isolation, orphan block rows without a session header, missing-session zero, and null when the synchronous lane is unavailable or errors.
  • Leave the cold-load choice, schema, persisted format and async backend contract unchanged.
  • Include a reproducible real SQLite benchmark for the complete public helper.

Measured effect

Node 24.22.1, built-in SQLite, real sessions ingested through planIngest with both full and one-line block views. Each workload has seven fresh baseline/candidate process pairs with alternating order. Setup, one warmup, input hashing and teardown are outside the measured interval; each interval measures complete UnifiedShadow.activeBlockCount() calls.

Workload Calls per interval Original median Updated median Faster pairs
4,000 blocks, 3,000 active 10 1,251.314 ms 6.784 ms 7 / 7
8 blocks, 6 active 50 12.857 ms 0.494 ms 7 / 7
Empty-session control 50 0.751 ms 0.414 ms 7 / 7

The large-case ranges are 1,126.274–1,349.795 ms before and 6.290–7.860 ms after. Input hashes and returned counts match across all fourteen runs of every workload. These are scalar-helper measurements, not end-to-end cold-load, memory, or inference-throughput claims. The existing session/block index is sufficient; no new index or migration is introduced.

Validation

  • Six new compatibility cases pass on both original and updated code, including active = 2, other sessions, missing headers/content, unchanged persisted rows/CAS ownership, closed SQLite and async-backend null behavior.
  • Those cases plus the existing drift-escalation and full-mode recovery suites: 22 passed on both trees, including fuller-index, fuller-envelope and tie selection controls.
  • Typecheck, complete build, unused-export guard, test-import-graph guard and git diff --check pass.
  • Independent review confirmed exact row membership, unchanged null/error fallback and unchanged persist.ts arbitration.

This is separate from #6618: that change makes necessary content reads cheaper; this change avoids content reads entirely when the caller only needs a count. Current-head remote CI remains a separate publication check.

@ranxianglei ranxianglei changed the title fix(storage): count active blocks without loading content [PR] fix(storage): count active blocks without loading content Nov 13, 2026
@ranxianglei
ranxianglei merged commit 6ceebbf into ranxianglei:master Nov 14, 2026
78 of 84 checks passed
@ranxianglei

Copy link
Copy Markdown
Owner

Superseded by #6677 (merged as 3858b6956, 2026-11-14).

Landed into master via the real merge commit 6ceebbfd594 (Merge Afloat16/2026-11-13_count-active-blocks (PR #6626)), so your original commits are preserved verbatim in master history — only hashes of the batch wrapper changed, your authorship and commit messages are intact.

The fix for #6624 is now in master. Closing this PR as superseded — thanks for the careful, well-tested fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Drift recovery hydrates all block content just to count active rows

2 participants