Repository navigation
adding tailContigLength and test for it #948
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
22bd814
16f1d89
f989db6
c6e8158
12ff6e2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -121,6 +121,7 @@ module.exports = async function auditCore( | |
| if (!tx && !stats.corrupt) tx = core.state.storage.write() | ||
|
|
||
| core.bitfield.set(index, false) | ||
| core._tailContiguousStart = -1 // direct write bypasses Core._setBitfieldRanges | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We should consider changing this bitfield set call to the |
||
|
|
||
| const page = core.bitfield.getBitfield(index) | ||
| if (page.bitfield) tx.putBitfieldPage(page.index, page.bitfield) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -150,6 +150,7 @@ async function flushBatch(prologue, src, dst, batch) { | |
| const roots = nodes.slice(0, batch.roots.length) | ||
| dst.state.setRoots(roots) | ||
| dst.header.tree = prologueToTree(prologue) | ||
| dst._tailContiguousStart = -1 // head upgraded outside Core._setBitfieldRanges | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This shouldn't be necessary right?
|
||
| } | ||
|
|
||
| if (userData.length > 0) { | ||
|
|
@@ -176,6 +177,7 @@ async function flushBatch(prologue, src, dst, batch) { | |
|
|
||
| start = index | ||
| dst.bitfield.set(index, true) | ||
| dst._tailContiguousStart = -1 // direct write bypasses Core._setBitfieldRanges | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same as |
||
| } | ||
|
|
||
| if (length > 0) signalReplicator(dst, upgraded, start, length) | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -63,6 +63,8 @@ module.exports = class Core { | |||||
| this._verifiesFlushed = null | ||||||
| this._legacy = !!opts.legacy | ||||||
| this._repairMode = false | ||||||
| this._tailContiguousStart = -1 | ||||||
| this._tailContiguousEnd = 0 | ||||||
|
|
||||||
| this.opening = this._open(opts) | ||||||
| this.opening.catch(noop) | ||||||
|
|
@@ -869,6 +871,46 @@ module.exports = class Core { | |||||
| _setBitfieldRanges(start, end, value) { | ||||||
| this.bitfield.setRange(start, end, value) | ||||||
| if (this.skipBitfield !== null) this.skipBitfield.setRange(start, end, value) | ||||||
|
|
||||||
| if (start >= end || this._tailContiguousStart === -1) return | ||||||
|
|
||||||
| if (value) { | ||||||
| if (start <= this._tailContiguousEnd && end >= this._tailContiguousStart) { | ||||||
| if (end > this._tailContiguousEnd) this._tailContiguousEnd = end | ||||||
| if (start < this._tailContiguousStart) { | ||||||
| this._tailContiguousStart = this.bitfield.lastUnset(start - 1) + 1 | ||||||
| } | ||||||
| } | ||||||
| } else if (end > this._tailContiguousStart && start < this._tailContiguousEnd) { | ||||||
| this._tailContiguousStart = Math.min(end, this._tailContiguousEnd) | ||||||
| } | ||||||
| } | ||||||
|
|
||||||
| get tailContiguousLength() { | ||||||
| const length = this.state.length | ||||||
|
|
||||||
| if (this._tailContiguousStart === -1 || length < this._tailContiguousEnd) { | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The Looking into this, in } else if (end > this.state.length && this._tailContiguousEnd > this.state.length) {
const length = this.state.length
if (this._tailContiguousStart >= length) this._tailContiguousStart = -1
else this._tailContiguousEnd = lengthI placed this before the last check that sets the cached start. |
||||||
| const start = length === 0 ? 0 : this.bitfield.lastUnset(length - 1) + 1 | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
This is already guarded in the |
||||||
| this._tailContiguousStart = start | ||||||
| this._tailContiguousEnd = length | ||||||
| return length - start | ||||||
| } | ||||||
|
|
||||||
| if (length === this._tailContiguousEnd) return length - this._tailContiguousStart | ||||||
|
|
||||||
| // Retain the known tail when the new head is missing. Downloading just that head | ||||||
| // can then extend the cached interval without rescanning the old suffix. | ||||||
| if (!this.bitfield.get(length - 1)) return 0 | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Leaving some thoughts on how this will be used for reference if it comes up later. While this is correct, this also means that each This probably means that |
||||||
|
|
||||||
| if (this.bitfield.firstUnset(this._tailContiguousEnd) >= length) { | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Initially this seemed to indicate we were missing an update elsewhere to the cached range, but walking through it I realized that the reason this is here is for situations where the tail is filled in before extending the previous range through. In that scenario the cached range could be updated by scanning the bitfield, but this would likely be a more expensive scan. After that rabbit trail, I realized that the
Given that, the full scan also happens twice, once for the get tailContiguousLength() {
const length = this.state.length
if (this._tailContiguousStart !== -1) {
if (length === this._tailContiguousEnd) return length - this._tailContiguousStart
// Retain the known tail when the new head is missing. Downloading just that head
// can then extend the cached interval without rescanning the old suffix.
if (!this.bitfield.get(length - 1)) return 0
}
const start = this.bitfield.lastUnset(length - 1) + 1
this._tailContiguousStart = start
this._tailContiguousEnd = length
return length - start
}This has the nice side effect that all early returns dont update the cache and the fallback is the only place that does the most expensive scan and updates the cached values. |
||||||
| this._tailContiguousEnd = length | ||||||
| return length - this._tailContiguousStart | ||||||
| } | ||||||
|
|
||||||
| const start = this.bitfield.lastUnset(length - 1) + 1 | ||||||
| this._tailContiguousStart = start | ||||||
| this._tailContiguousEnd = length | ||||||
| return length - start | ||||||
| } | ||||||
|
|
||||||
| close() { | ||||||
|
|
||||||
Uh oh!
There was an error while loading. Please reload this page.