Repository navigation
adding tailContigLength and test for it - #948
mynameisdaniil wants to merge 5 commits into
Conversation
lejeunerenard
left a comment
There was a problem hiding this comment.
Great first pass! While using the bitfield as the source of truth works if accessing the property is infrequent and performance isn't an issue, but in cases where the tail is checked on append or download like blind-peer does here:
https://github.com/holepunchto/blind-peer/blob/16a5d086cc63c5f496fd9be0636ba89f25aaa3c2/index.js#L728-L733
[Low impact] The PR appears safe to merge; the latest documentation matches the shared getter behavior. SummaryAdds
Reviews (2) · Last reviewed commit: "updating readme to reflect tailContiguou..." · Reviewed by Greptile |
| if (!tx && !stats.corrupt) tx = core.state.storage.write() | ||
|
|
||
| core.bitfield.set(index, false) | ||
| core._tailContiguousStart = -1 // direct write bypasses Core._setBitfieldRanges |
There was a problem hiding this comment.
We should consider changing this bitfield set call to the core._setBitfieldRange() equivalent, but that should be in a separate PR since it also modies the skip bitfield and is outside the scope here.
| get tailContiguousLength() { | ||
| const length = this.state.length | ||
|
|
||
| if (this._tailContiguousStart === -1 || length < this._tailContiguousEnd) { |
There was a problem hiding this comment.
The _tailContiguousEnd should never be greater than the length. That would mean we didn't update the property correctly elsewhere.
Looking into this, in _setBitfieldRanges a case is missing where the updated range exceeds the current length. This should crop the tail contiguous end to the length or invalidate the cached values entirely if the length was truncated to before the previous length:
} else if (end > this.state.length && this._tailContiguousEnd > this.state.length) {
const length = this.state.length
if (this._tailContiguousStart >= length) this._tailContiguousStart = -1
else this._tailContiguousEnd = lengthI placed this before the last check that sets the cached start.
| // can then extend the cached interval without rescanning the old suffix. | ||
| if (!this.bitfield.get(length - 1)) return 0 | ||
|
|
||
| if (this.bitfield.firstUnset(this._tailContiguousEnd) >= length) { |
There was a problem hiding this comment.
Initially this seemed to indicate we were missing an update elsewhere to the cached range, but walking through it I realized that the reason this is here is for situations where the tail is filled in before extending the previous range through. In that scenario the cached range could be updated by scanning the bitfield, but this would likely be a more expensive scan.
After that rabbit trail, I realized that the .firstUnset() scan is unneccessary and can fall through to the full scan below. This is because:
- it still scans and the case that it returns early is longer than the scan below.
- It only updates the
._tailContiguousEndto thelengthanyways. Same as the full scan.
So I think it can be skipped as doing a full scan only uses one scan and also ensures the start is accurate.
Given that, the full scan also happens twice, once for the -1 start case and again as the last case. So we can flip the early execution around so it checks non-full scale scenarios first and falls through to the full scan once. It would look like this:
get tailContiguousLength() {
const length = this.state.length
if (this._tailContiguousStart !== -1) {
if (length === this._tailContiguousEnd) return length - this._tailContiguousStart
// Retain the known tail when the new head is missing. Downloading just that head
// can then extend the cached interval without rescanning the old suffix.
if (!this.bitfield.get(length - 1)) return 0
}
const start = this.bitfield.lastUnset(length - 1) + 1
this._tailContiguousStart = start
this._tailContiguousEnd = length
return length - start
}This has the nice side effect that all early returns dont update the cache and the fallback is the only place that does the most expensive scan and updates the cached values.
| const length = this.state.length | ||
|
|
||
| if (this._tailContiguousStart === -1 || length < this._tailContiguousEnd) { | ||
| const start = length === 0 ? 0 : this.bitfield.lastUnset(length - 1) + 1 |
There was a problem hiding this comment.
| const start = length === 0 ? 0 : this.bitfield.lastUnset(length - 1) + 1 | |
| const start = this.bitfield.lastUnset(length - 1) + 1 |
This is already guarded in the findLast() of the bitfield for the 0 case. It returns -1 which when + 1 is applied becomes the 0 the ternary used.
|
|
||
| // Retain the known tail when the new head is missing. Downloading just that head | ||
| // can then extend the cached interval without rescanning the old suffix. | ||
| if (!this.bitfield.get(length - 1)) return 0 |
There was a problem hiding this comment.
Leaving some thoughts on how this will be used for reference if it comes up later.
While this is correct, this also means that each append event will cause the tailContiguousLength to be 0 for the event briefly since the upgrade comes through first and then the block is downloaded.
This probably means that append is just not the right event to detect additive tailContiguousLength changes. download would be more appropriate in those cases. But it does indicate that maybe a tail-contiguous-length event might be useful. We'll see when integrating elsewhere.
| t.is(b.tailContiguousLength, 3) | ||
| t.is(scans.length, 1, 'repeated reads reuse the tail scan') |
There was a problem hiding this comment.
| t.is(b.tailContiguousLength, 3) | |
| t.is(scans.length, 1, 'repeated reads reuse the tail scan') |
Redundant lines. Covered by tailContiguousLength - repeated reads and sessions reuse the first scan test.
| t.is(b.tailContiguousLength, 3, 'an unrelated earlier download leaves the tail cached') | ||
| t.is(scans.length, 1, 'unrelated earlier blocks do not scan the known tail') | ||
|
|
||
| downloads.length = 0 |
There was a problem hiding this comment.
nit: I'd rather we kept the entries than set the length to truncate the array. If we want only the new entries, then maybe slice it in the assert.
| t.alike(scans, [{ method: 'lastUnset', position: 4 }]) | ||
|
|
||
| t.is(core.tailContiguousLength, 5) | ||
| t.is(core.tailContiguousLength, 5) |
There was a problem hiding this comment.
| t.is(core.tailContiguousLength, 5) |
Only need one more call to assert it doesn't change.
|
|
||
| start = index | ||
| dst.bitfield.set(index, true) | ||
| dst._tailContiguousStart = -1 // direct write bypasses Core._setBitfieldRanges |
There was a problem hiding this comment.
Same as audit, we should consider changing the bitfield set to dist._setBitfieldRange() equivalent, but not now.
| const roots = nodes.slice(0, batch.roots.length) | ||
| dst.state.setRoots(roots) | ||
| dst.header.tree = prologueToTree(prologue) | ||
| dst._tailContiguousStart = -1 // head upgraded outside Core._setBitfieldRanges |
There was a problem hiding this comment.
This shouldn't be necessary right?
- Its not updating the bitfield here.
- This only runs if there is no head entry so there should be no blocks. IE this should always be
-1already.
Implementing tailContigLength