Skip to content

adding tailContigLength and test for it - #948

Open
mynameisdaniil wants to merge 5 commits into
mainfrom
add-tail-contig-length
Open

mynameisdaniil wants to merge 5 commits into
mainfrom
add-tail-contig-length

Conversation

@mynameisdaniil

Copy link
Copy Markdown

Implementing tailContigLength

Comment thread index.js Resolved

@lejeunerenard lejeunerenard left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great first pass! While using the bitfield as the source of truth works if accessing the property is infrequent and performance isn't an issue, but in cases where the tail is checked on append or download like blind-peer does here:
https://github.com/holepunchto/blind-peer/blob/16a5d086cc63c5f496fd9be0636ba89f25aaa3c2/index.js#L728-L733

Comment thread index.js Resolved
@mynameisdaniil
mynameisdaniil marked this pull request as ready for review November 13, 2026 07:00
@mynameisdaniil
mynameisdaniil requested a review from a team November 13, 2026 07:00
@greptile-apps

greptile-apps Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The PR appears safe to merge; the latest documentation matches the shared getter behavior.

Summary

Adds core.tailContiguousLength to count locally available blocks backward from the shared default core’s end.

  • Hypercore now counts local blocks backward from its shared end.

Reviews (2) · Last reviewed commit: "updating readme to reflect tailContiguou..." · Reviewed by Greptile

Comment thread index.js Resolved
Comment thread lib/audit.js
if (!tx && !stats.corrupt) tx = core.state.storage.write()

core.bitfield.set(index, false)
core._tailContiguousStart = -1 // direct write bypasses Core._setBitfieldRanges

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should consider changing this bitfield set call to the core._setBitfieldRange() equivalent, but that should be in a separate PR since it also modies the skip bitfield and is outside the scope here.

Comment thread lib/core.js
get tailContiguousLength() {
const length = this.state.length

if (this._tailContiguousStart === -1 || length < this._tailContiguousEnd) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _tailContiguousEnd should never be greater than the length. That would mean we didn't update the property correctly elsewhere.

Looking into this, in _setBitfieldRanges a case is missing where the updated range exceeds the current length. This should crop the tail contiguous end to the length or invalidate the cached values entirely if the length was truncated to before the previous length:

    } else if (end > this.state.length && this._tailContiguousEnd > this.state.length) {
      const length = this.state.length

      if (this._tailContiguousStart >= length) this._tailContiguousStart = -1
      else this._tailContiguousEnd = length

I placed this before the last check that sets the cached start.

Comment thread lib/core.js
// can then extend the cached interval without rescanning the old suffix.
if (!this.bitfield.get(length - 1)) return 0

if (this.bitfield.firstUnset(this._tailContiguousEnd) >= length) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially this seemed to indicate we were missing an update elsewhere to the cached range, but walking through it I realized that the reason this is here is for situations where the tail is filled in before extending the previous range through. In that scenario the cached range could be updated by scanning the bitfield, but this would likely be a more expensive scan.

After that rabbit trail, I realized that the .firstUnset() scan is unneccessary and can fall through to the full scan below. This is because:

  1. it still scans and the case that it returns early is longer than the scan below.
  2. It only updates the ._tailContiguousEnd to the length anyways. Same as the full scan.
    So I think it can be skipped as doing a full scan only uses one scan and also ensures the start is accurate.

Given that, the full scan also happens twice, once for the -1 start case and again as the last case. So we can flip the early execution around so it checks non-full scale scenarios first and falls through to the full scan once. It would look like this:

  get tailContiguousLength() {
    const length = this.state.length

    if (this._tailContiguousStart !== -1) {
      if (length === this._tailContiguousEnd) return length - this._tailContiguousStart

      // Retain the known tail when the new head is missing. Downloading just that head
      // can then extend the cached interval without rescanning the old suffix.
      if (!this.bitfield.get(length - 1)) return 0
    }

    const start = this.bitfield.lastUnset(length - 1) + 1
    this._tailContiguousStart = start
    this._tailContiguousEnd = length
    return length - start
  }

This has the nice side effect that all early returns dont update the cache and the fallback is the only place that does the most expensive scan and updates the cached values.

Comment thread lib/core.js
const length = this.state.length

if (this._tailContiguousStart === -1 || length < this._tailContiguousEnd) {
const start = length === 0 ? 0 : this.bitfield.lastUnset(length - 1) + 1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const start = length === 0 ? 0 : this.bitfield.lastUnset(length - 1) + 1
const start = this.bitfield.lastUnset(length - 1) + 1

This is already guarded in the findLast() of the bitfield for the 0 case. It returns -1 which when + 1 is applied becomes the 0 the ternary used.

Comment thread lib/core.js

// Retain the known tail when the new head is missing. Downloading just that head
// can then extend the cached interval without rescanning the old suffix.
if (!this.bitfield.get(length - 1)) return 0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving some thoughts on how this will be used for reference if it comes up later.

While this is correct, this also means that each append event will cause the tailContiguousLength to be 0 for the event briefly since the upgrade comes through first and then the block is downloaded.

This probably means that append is just not the right event to detect additive tailContiguousLength changes. download would be more appropriate in those cases. But it does indicate that maybe a tail-contiguous-length event might be useful. We'll see when integrating elsewhere.

Comment on lines +60 to +61
t.is(b.tailContiguousLength, 3)
t.is(scans.length, 1, 'repeated reads reuse the tail scan')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
t.is(b.tailContiguousLength, 3)
t.is(scans.length, 1, 'repeated reads reuse the tail scan')

Redundant lines. Covered by tailContiguousLength - repeated reads and sessions reuse the first scan test.

t.is(b.tailContiguousLength, 3, 'an unrelated earlier download leaves the tail cached')
t.is(scans.length, 1, 'unrelated earlier blocks do not scan the known tail')

downloads.length = 0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I'd rather we kept the entries than set the length to truncate the array. If we want only the new entries, then maybe slice it in the assert.

t.alike(scans, [{ method: 'lastUnset', position: 4 }])

t.is(core.tailContiguousLength, 5)
t.is(core.tailContiguousLength, 5)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
t.is(core.tailContiguousLength, 5)

Only need one more call to assert it doesn't change.

Comment thread lib/copy-prologue.js

start = index
dst.bitfield.set(index, true)
dst._tailContiguousStart = -1 // direct write bypasses Core._setBitfieldRanges

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as audit, we should consider changing the bitfield set to dist._setBitfieldRange() equivalent, but not now.

Comment thread lib/copy-prologue.js
const roots = nodes.slice(0, batch.roots.length)
dst.state.setRoots(roots)
dst.header.tree = prologueToTree(prologue)
dst._tailContiguousStart = -1 // head upgraded outside Core._setBitfieldRanges

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be necessary right?

  1. Its not updating the bitfield here.
  2. This only runs if there is no head entry so there should be no blocks. IE this should always be -1 already.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants