Skip to content

fix: report core storage byte sizes from rocksdb key ranges - #947

Closed
hakarrr wants to merge 3 commits into
holepunchto:mainfrom
hakarrr:fix-890-storage-info
Closed

hakarrr wants to merge 3 commits into
holepunchto:mainfrom
hakarrr:fix-890-storage-info

Conversation

@hakarrr

@hakarrr hakarrr commented Nov 8, 2026

Copy link
Copy Markdown

Fixes #890

Summary

core.info({ storage: true }) always returned null since the move to rocksdb-backed storage: Info.storage destructured session.core.{oplog,tree,blocks,bitfield}, which no longer exist, and the catch swallowed the error.

Fix

The four old per-file sections are mapped onto their rocksdb key ranges in the core's data namespace, and each section's byte size is read with db.approximateSize(start, end, { includeMemtables: true }):

  • blocks: DATA_BLOCK range
  • tree: DATA_TREE range
  • bitfield: DATA_BITFIELD range
  • oplog: DATA_HEAD + DATA_DEPENDENCY + DATA_HINTS - the oplog used to hold the core header, its dependencies and its hints

Keys come from hypercore-storage/lib/keys.js, so the ranges follow the storage layout exactly. Sessions without a rocksdb-backed storage (or a closed one) still return null, as before, via the existing catch.

Test

Un-skipped storage info in test/basic.js and replaced the snapshot (volatile byte counts) with structural assertions: non-null object, numeric non-negative sections, and blocks > 0 after appending. Failing-first: against the published lib the test fails with info.storage === null; with the fix it passes. Also verified info() without the flag still returns null storage, and that snapshot sessions report storage too. The full test/basic.js file OOMs on my 1 GB box, so the touched tests were run extracted with the repo's real helpers - calling that out honestly rather than claiming a full-suite run.

Re-enable the storage info test and add assertions to check storage sections.
@hakarrr
hakarrr requested a review from a team November 8, 2026 22:35
@greptile-apps

greptile-apps Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The PR appears safe to merge; the remaining test coverage gap is non-blocking.

Fix All in Claude CodeFindings

  1. P2 Wrong ranges pass this test ▶
Fix with agent prompt
### Issue 1
test/basic.js:530-536
The assertions allow `oplog`, `tree`, and `bitfield` to report zero regardless of what they contain. Swapping their ranges would also pass, leaving misleading storage estimates undetected. Add focused checks that each section measures its intended keys and excludes another core's keys. Checking the requested ranges would avoid relying on exact, changeable byte counts.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Restores core.info({ storage: true }) by measuring RocksDB key ranges instead of removed per-file storage objects.

  • Core info reports four storage sizes from RocksDB key ranges.

Reviews (3) · Last reviewed commit: "test: assert storage info ranges measure..." · Reviewed by Greptile

Comment thread test/basic.js Outdated
Comment on lines +530 to +536
for (const section of ['oplog', 'tree', 'blocks', 'bitfield']) {
t.ok(
typeof info.storage[section] === 'number' && info.storage[section] >= 0,
section + ' reports a byte size'
)
}
t.ok(info.storage.blocks > 0, 'blocks section holds the appended data')

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Wrong ranges pass this test

The assertions allow oplog, tree, and bitfield to report zero regardless of what they contain. Swapping their ranges would also pass, leaving misleading storage estimates undetected. Add focused checks that each section measures its intended keys and excludes another core's keys. Checking the requested ranges would avoid relying on exact, changeable byte counts.

Prompt To Fix With AI
This is a comment left during a code review.
Path: test/basic.js
Line: 530-536

Comment:
**Wrong ranges pass this test**

The assertions allow `oplog`, `tree`, and `bitfield` to report zero regardless of what they contain. Swapping their ranges would also pass, leaving misleading storage estimates undetected. Add focused checks that each section measures its intended keys and excludes another core's keys. Checking the requested ranges would avoid relying on exact, changeable byte counts.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch - addressed in 4e3feb5.

The test now checks that every section is nonzero after appends, that an appended block strictly grows the blocks section, and that a second core opened on the same database measures strictly less in every section - so a range that was empty, db-wide, or pointed at another core's keys would fail.

One honest limit: while the data is still in the memtable, RocksDB spreads its size estimate across a core's sections by proportion, so exact per-section byte counts stay approximate (that's why the assertions are structural, not numeric). Verified locally: both storage tests pass, 12/12 assertions (extracted run - the full test/basic.js OOMs on my small box, same as before).

This comment was marked as duplicate.

Every section must be nonzero after appends, an appended block must grow the blocks section, and a second core on the same database must measure less in every section (cross-core isolation).

Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
@hakarrr

hakarrr commented Nov 13, 2026

Copy link
Copy Markdown
Author

Could a maintainer take another look at this core-storage size reporting fix? My earlier reply records the regression checks and the memtable estimate limitation. Please let me know if there is anything further to change before review. Thanks.

@hakarrr hakarrr closed this Nov 16, 2026
@lejeunerenard

Copy link
Copy Markdown
Contributor

@hakarrr thank you for taking a look at implementing the storage info size API. Why did you decide to close it?

@hakarrr

hakarrr commented Nov 17, 2026

Copy link
Copy Markdown
Author

Thanks for checking. I closed it while cleaning up inactive PRs after my follow-up went unanswered. The changes are still preserved. Is this implementation something the project would like to take forward?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

core.info({ storage: true }) always returns storage: null

2 participants