Repository navigation
Conversation
Re-enable the storage info test and add assertions to check storage sections.
[Low impact] The PR appears safe to merge; the remaining test coverage gap is non-blocking.
|
| for (const section of ['oplog', 'tree', 'blocks', 'bitfield']) { | ||
| t.ok( | ||
| typeof info.storage[section] === 'number' && info.storage[section] >= 0, | ||
| section + ' reports a byte size' | ||
| ) | ||
| } | ||
| t.ok(info.storage.blocks > 0, 'blocks section holds the appended data') |
There was a problem hiding this comment.
The assertions allow oplog, tree, and bitfield to report zero regardless of what they contain. Swapping their ranges would also pass, leaving misleading storage estimates undetected. Add focused checks that each section measures its intended keys and excludes another core's keys. Checking the requested ranges would avoid relying on exact, changeable byte counts.
Prompt To Fix With AI
This is a comment left during a code review.
Path: test/basic.js
Line: 530-536
Comment:
**Wrong ranges pass this test**
The assertions allow `oplog`, `tree`, and `bitfield` to report zero regardless of what they contain. Swapping their ranges would also pass, leaving misleading storage estimates undetected. Add focused checks that each section measures its intended keys and excludes another core's keys. Checking the requested ranges would avoid relying on exact, changeable byte counts.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Good catch - addressed in 4e3feb5.
The test now checks that every section is nonzero after appends, that an appended block strictly grows the blocks section, and that a second core opened on the same database measures strictly less in every section - so a range that was empty, db-wide, or pointed at another core's keys would fail.
One honest limit: while the data is still in the memtable, RocksDB spreads its size estimate across a core's sections by proportion, so exact per-section byte counts stay approximate (that's why the assertions are structural, not numeric). Verified locally: both storage tests pass, 12/12 assertions (extracted run - the full test/basic.js OOMs on my small box, same as before).
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
Every section must be nonzero after appends, an appended block must grow the blocks section, and a second core on the same database must measure less in every section (cross-core isolation). Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
|
Could a maintainer take another look at this core-storage size reporting fix? My earlier reply records the regression checks and the memtable estimate limitation. Please let me know if there is anything further to change before review. Thanks. |
|
@hakarrr thank you for taking a look at implementing the storage info size API. Why did you decide to close it? |
|
Thanks for checking. I closed it while cleaning up inactive PRs after my follow-up went unanswered. The changes are still preserved. Is this implementation something the project would like to take forward? |
Fixes #890
Summary
core.info({ storage: true })always returnednullsince the move to rocksdb-backed storage:Info.storagedestructuredsession.core.{oplog,tree,blocks,bitfield}, which no longer exist, and thecatchswallowed the error.Fix
The four old per-file sections are mapped onto their rocksdb key ranges in the core's data namespace, and each section's byte size is read with
db.approximateSize(start, end, { includeMemtables: true }):blocks:DATA_BLOCKrangetree:DATA_TREErangebitfield:DATA_BITFIELDrangeoplog:DATA_HEAD+DATA_DEPENDENCY+DATA_HINTS- the oplog used to hold the core header, its dependencies and its hintsKeys come from
hypercore-storage/lib/keys.js, so the ranges follow the storage layout exactly. Sessions without a rocksdb-backed storage (or a closed one) still returnnull, as before, via the existing catch.Test
Un-skipped
storage infointest/basic.jsand replaced the snapshot (volatile byte counts) with structural assertions: non-null object, numeric non-negative sections, andblocks > 0after appending. Failing-first: against the published lib the test fails withinfo.storage === null; with the fix it passes. Also verifiedinfo()without the flag still returnsnullstorage, and that snapshot sessions report storage too. The fulltest/basic.jsfile OOMs on my 1 GB box, so the touched tests were run extracted with the repo's real helpers - calling that out honestly rather than claiming a full-suite run.