Skip to content

Tell a peer the new length even when it can't be upgraded yet - #949

Open
gusttav-lang wants to merge 2 commits into
holepunchto:mainfrom
gusttav-lang:fix/always-announce-length
Open

gusttav-lang wants to merge 2 commits into
holepunchto:mainfrom
gusttav-lang:fix/always-announce-length

Conversation

@gusttav-lang

@gusttav-lang gusttav-lang commented Nov 10, 2026 •

Copy link
Copy Markdown

A peer can go on believing a core is at an old length, and then never asks it for blocks it holds: _makeRequest skips a peer whose remoteLength is below the block. Two paths drop the sync that would have told it.

  • _updateCanUpgradeAndSync only sends when canUpgrade flips. signalUpgrade takes that path whenever the core is ahead of a peer it can't upgrade yet, so when its length grows and it still can't upgrade that peer, nothing goes out. A sparse relay hits this easily: it learns a writer's new length without the tree nodes for the length its other peer is at, so upgradeable() stays false.
  • When the peer's sync lands while that check is awaiting, the check returns and leaves it to the end of onsync, which only signals again if the core is still longer. If the peer caught up to the same length at that moment, it never hears that the core has it.

Nothing repairs it afterwards. A get without a timeout waits until the channel between the two is reopened, and one with a timeout fails on every retry. A minute of waiting, the relay growing further, and the reader moving ahead through another peer all leave the reader seeing the relay at its old length.

The sync now goes out whenever the core's length or fork, or the peer's length it acks, is news to the peer. sendSync records the three it sent as lengthSent, forkSent and remoteLengthSent; _updateCanUpgradeAndSync sends when canUpgrade flips or any of them is stale, and onsync signals again when the core's own length or fork is unsent.

Reproduction

A relay at length 1 and a reader holding only the roots of length 3, so neither can upgrade the other. The relay grows to 6 from the writer, the reader learns 6 from the writer too, and then asks the relay for block 5. On main this times out every run; with this change it prints ok.

const Hypercore = require('hypercore')
const fs = require('fs')
const os = require('os')
const path = require('path')

const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'repro-'))
const open = (name, key) => new Hypercore(path.join(tmp, name), key)

function link(a, b) {
  const s1 = a.replicate(true, { keepAlive: false })
  const s2 = b.replicate(false, { keepAlive: false })
  s1.pipe(s2).pipe(s1)
  return async () => {
    const closed = [s1, s2].map((s) => new Promise((r) => s.once('close', r)))
    s1.destroy()
    s2.destroy()
    await Promise.all(closed)
  }
}

async function main() {
  const writer = open('writer')
  await writer.ready()
  const relay = open('relay', writer.key)
  const reader = open('reader', writer.key)

  await writer.append('a')
  let unlink = link(writer, relay)
  await relay.get(0) // relay at length 1
  await unlink()

  await writer.append(['b', 'c'])
  unlink = link(writer, reader)
  await reader.update({ wait: true }) // reader at length 3, roots only: neither can upgrade the other
  await unlink()

  link(relay, reader)

  await writer.append(['d', 'e', 'f'])
  unlink = link(writer, relay)
  await relay.get(5) // relay at length 6, holds block 5
  await unlink()

  unlink = link(writer, reader)
  await reader.update({ wait: true }) // reader at length 6
  await unlink()

  await reader.get(5, { timeout: 3000 }) // main: REQUEST_TIMEOUT, the reader still sees the relay at 1
  console.log('ok')
  process.exit(0)
}

main()

Tests

  • a relay that cannot upgrade a peer still tells it its new length is the script above; it fails on main.
  • a relay that catches up with a peer at once still tells it its new length lets the relay and the reader catch up at the same moment, ten times over, for the race path; a fix for the first path alone misses about half of them.
  • a peer that cannot upgrade a shorter one still acks its length covers the ack: on main the shorter peer's update({ wait: true }) never returns.

npm test and bare test/all.js pass, 475 of 475.

Tradeoffs

  • More syncs only where main dropped them. Over 300 appends, a writer with two full readers, a full chain and a batch append send exactly the same syncs as before. A triangle with a sparse relay sends 1804 instead of 1607–1633, one per length change per direction.
  • Each sync is about 5 bytes of payload plus framing, so that triangle carries roughly 1 KB more, about 1% of its traffic. The check that decides to send is the _canUpgrade call main already makes, plus three comparisons.
  • Each peer keeps three more numbers, and the send rule is "anything in the sync is stale" rather than "canUpgrade flipped", which is a little more to hold in your head.
  • Sending on every _updateCanUpgradeAndSync would also fix both paths and is simpler, but it sends again on calls where nothing in the sync changed. The runs above never hit such a call, so both send the same count there; this keeps those calls quiet where they do happen.

I found this in a multi-device app where a phone reaches a writer only through a third device. A reply handed off through that device never arrived in about a quarter of runs; under CPU load the change took that test from hanging to 30 of 30 passes.

A peer only asks us for blocks below the length it last heard from us.
The sync went out only when canUpgrade flipped, so a relay that grew
while still unable to upgrade a peer stayed silent, and a peer that
caught up while the canUpgrade check was in flight never heard our
length either. Send the sync whenever our length, fork or the length
we ack is newer than what we last sent.
@gusttav-lang
gusttav-lang marked this pull request as ready for review November 10, 2026 19:30
@gusttav-lang
gusttav-lang requested a review from a team November 10, 2026 19:30
@greptile-apps

greptile-apps Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issues were found.

Summary

Keeps peers informed of a core’s length even when it cannot upgrade them yet.

  • Peers send fresh length information even when they cannot upgrade each other.

No new actionable issues were found in the changes since the previous review. No previous findings were supplied.

Reviews (2) · Last reviewed commit: "Cover the missed ack and keep a regressi..." · Reviewed by Greptile

@lejeunerenard lejeunerenard left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great find! Thanks for checking those edgecases.

As mentioned in the comments the change in onsync adds a good amount of sync messages. As the _lengthSent() added condition for upgrading or syncing fires whenever a peer has updated and not told the peer about it. That's the point of the edgecase where it races. I'd like to have another pair of eyes on this before approving. In the mean time I commented on some other nits I found.

Comment thread test/replicate.js Outdated
await b.update({ wait: true })
await unreplicate(s)

t.alike(await b.get(5, { timeout: 5000 }), b4a.from('f'), 'b gets the block from the relay')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should use t.execution() (docs) to not crash the tests and allow the rest to run if it regresses again.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f7c54d8. The new ack test fails cleanly on main too.

Comment thread lib/replicator.js Outdated
this.lastUpgradableFork = this.remoteFork
}

// the sync carries our length and acks theirs, so a remote we cannot upgrade still needs it

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment essentially explains why the code changes in this PR but isn't really how you'd read this function outside that context. It should probably be move up to where the ability to upgrade is checked.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in f7c54d8.

Comment thread lib/replicator.js
this.needsSync === true ||
(this.core.state.fork === this.remoteFork && this.core.state.length > this.remoteLength)
(this.core.state.fork === this.remoteFork && this.core.state.length > this.remoteLength) ||
this._lengthSent() === false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to oversend syncs in response. So far I can't figure out a way to reduce it only to the race conditions (in the second 10x try test). So it might be a matter of whether the extra syncs are worth catching that edgecase.

@marcus-pousette-hp marcus-pousette-hp Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Analysed this too. When measuring it locally, this only fires in the race, and each extra sync is correctly needed. It is about +20% syncs at zero latency (tests), 1% at 1 ms, and no difference at at 20-40 ms. And so for the low latency scenarios we can in the future add a 20 ms debounce/delay on this signal path and I think we would get back to mains behavior.

@marcus-pousette-hp marcus-pousette-hp Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now looked at slow tree reads. With a simulated 5–15 ms read in _canUpgrade, the race also shows up at 20–40 ms. In this scenario main drops 2–6% of length announcements and this PR adds 3–7% syncs but does the right thing, so my analysis above was partial here

@marcus-pousette-hp marcus-pousette-hp left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

+1 to Sean’s nits.

For perf concerns in my investigation this is not huge for common prod paths, and for low latency ones there is a follow up #949 (comment)

Comment thread lib/replicator.js
}

_syncSent() {
return this._lengthSent() && this.remoteLengthSent === this._remoteLengthToAck()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No test covers remoteLengthSent. It also seem to fix a second hang: on main, when two peers can’t upgrade each other, update({ wait: true }) on the shorter one waits forever for the longer one to ack its length. Could you add a test for that?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in f7c54d8: "a peer that cannot upgrade a shorter one still acks its length". It hangs on main and passes here.

Test that a peer which cannot upgrade a shorter one still acks its
length, so the shorter one's update({ wait: true }) returns. Use
t.execution for the relay test, and race the update against a timer,
so either one failing on a regression fails only its own test. Drop
the comment in _updateCanUpgradeAndSync.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants