Repository navigation
Tell a peer the new length even when it can't be upgraded yet - #949
gusttav-lang wants to merge 2 commits into
Conversation
A peer only asks us for blocks below the length it last heard from us. The sync went out only when canUpgrade flipped, so a relay that grew while still unable to upgrade a peer stayed silent, and a peer that caught up while the canUpgrade check was in flight never heard our length either. Send the sync whenever our length, fork or the length we ack is newer than what we last sent.
[Medium impact] The PR appears safe to merge; no new actionable issues were found. SummaryKeeps peers informed of a core’s length even when it cannot upgrade them yet.
No new actionable issues were found in the changes since the previous review. No previous findings were supplied. Reviews (2) · Last reviewed commit: "Cover the missed ack and keep a regressi..." · Reviewed by Greptile |
lejeunerenard
left a comment
There was a problem hiding this comment.
Great find! Thanks for checking those edgecases.
As mentioned in the comments the change in onsync adds a good amount of sync messages. As the _lengthSent() added condition for upgrading or syncing fires whenever a peer has updated and not told the peer about it. That's the point of the edgecase where it races. I'd like to have another pair of eyes on this before approving. In the mean time I commented on some other nits I found.
| await b.update({ wait: true }) | ||
| await unreplicate(s) | ||
|
|
||
| t.alike(await b.get(5, { timeout: 5000 }), b4a.from('f'), 'b gets the block from the relay') |
| this.lastUpgradableFork = this.remoteFork | ||
| } | ||
|
|
||
| // the sync carries our length and acks theirs, so a remote we cannot upgrade still needs it |
There was a problem hiding this comment.
This comment essentially explains why the code changes in this PR but isn't really how you'd read this function outside that context. It should probably be move up to where the ability to upgrade is checked.
| this.needsSync === true || | ||
| (this.core.state.fork === this.remoteFork && this.core.state.length > this.remoteLength) | ||
| (this.core.state.fork === this.remoteFork && this.core.state.length > this.remoteLength) || | ||
| this._lengthSent() === false |
There was a problem hiding this comment.
This seems to oversend syncs in response. So far I can't figure out a way to reduce it only to the race conditions (in the second 10x try test). So it might be a matter of whether the extra syncs are worth catching that edgecase.
There was a problem hiding this comment.
Analysed this too. When measuring it locally, this only fires in the race, and each extra sync is correctly needed. It is about +20% syncs at zero latency (tests), 1% at 1 ms, and no difference at at 20-40 ms. And so for the low latency scenarios we can in the future add a 20 ms debounce/delay on this signal path and I think we would get back to mains behavior.
There was a problem hiding this comment.
I now looked at slow tree reads. With a simulated 5–15 ms read in _canUpgrade, the race also shows up at 20–40 ms. In this scenario main drops 2–6% of length announcements and this PR adds 3–7% syncs but does the right thing, so my analysis above was partial here
There was a problem hiding this comment.
Looks good to me.
+1 to Sean’s nits.
For perf concerns in my investigation this is not huge for common prod paths, and for low latency ones there is a follow up #949 (comment)
| } | ||
|
|
||
| _syncSent() { | ||
| return this._lengthSent() && this.remoteLengthSent === this._remoteLengthToAck() |
There was a problem hiding this comment.
No test covers remoteLengthSent. It also seem to fix a second hang: on main, when two peers can’t upgrade each other, update({ wait: true }) on the shorter one waits forever for the longer one to ack its length. Could you add a test for that?
There was a problem hiding this comment.
Added in f7c54d8: "a peer that cannot upgrade a shorter one still acks its length". It hangs on main and passes here.
Test that a peer which cannot upgrade a shorter one still acks its
length, so the shorter one's update({ wait: true }) returns. Use
t.execution for the relay test, and race the update against a timer,
so either one failing on a regression fails only its own test. Drop
the comment in _updateCanUpgradeAndSync.
A peer can go on believing a core is at an old length, and then never asks it for blocks it holds:
_makeRequestskips a peer whoseremoteLengthis below the block. Two paths drop the sync that would have told it._updateCanUpgradeAndSynconly sends whencanUpgradeflips.signalUpgradetakes that path whenever the core is ahead of a peer it can't upgrade yet, so when its length grows and it still can't upgrade that peer, nothing goes out. A sparse relay hits this easily: it learns a writer's new length without the tree nodes for the length its other peer is at, soupgradeable()stays false.onsync, which only signals again if the core is still longer. If the peer caught up to the same length at that moment, it never hears that the core has it.Nothing repairs it afterwards. A
getwithout a timeout waits until the channel between the two is reopened, and one with a timeout fails on every retry. A minute of waiting, the relay growing further, and the reader moving ahead through another peer all leave the reader seeing the relay at its old length.The sync now goes out whenever the core's length or fork, or the peer's length it acks, is news to the peer.
sendSyncrecords the three it sent aslengthSent,forkSentandremoteLengthSent;_updateCanUpgradeAndSyncsends whencanUpgradeflips or any of them is stale, andonsyncsignals again when the core's own length or fork is unsent.Reproduction
A relay at length 1 and a reader holding only the roots of length 3, so neither can upgrade the other. The relay grows to 6 from the writer, the reader learns 6 from the writer too, and then asks the relay for block 5. On main this times out every run; with this change it prints
ok.Tests
a relay that cannot upgrade a peer still tells it its new lengthis the script above; it fails on main.a relay that catches up with a peer at once still tells it its new lengthlets the relay and the reader catch up at the same moment, ten times over, for the race path; a fix for the first path alone misses about half of them.a peer that cannot upgrade a shorter one still acks its lengthcovers the ack: on main the shorter peer'supdate({ wait: true })never returns.npm testandbare test/all.jspass, 475 of 475.Tradeoffs
_canUpgradecall main already makes, plus three comparisons.canUpgradeflipped", which is a little more to hold in your head._updateCanUpgradeAndSyncwould also fix both paths and is simpler, but it sends again on calls where nothing in the sync changed. The runs above never hit such a call, so both send the same count there; this keeps those calls quiet where they do happen.I found this in a multi-device app where a phone reaches a writer only through a third device. A reply handed off through that device never arrived in about a quarter of runs; under CPU load the change took that test from hanging to 30 of 30 passes.