Leios prototype: Fix PastHorizon errors in syncing - #2252
Merged
Conversation
'announcementValidity' reuses ChainSync's in-future check to date an announced EB, and threw whatever it returned. A syncing node cannot place the network's current slot on a wall clock, so every announcement from a caught-up peer raised 'PastHorizon', killed the per-peer handler and shut the peer down. With no peer surviving to serve headers, ChainSync stopped, no announcements arrived via rollforward, and chain selection froze until the node was restarted -- 148 such disconnects in one 29-minute stall. The stock ChainSync client calls the same 'judgeHeaderArrival' and reads 'PastHorizon' as "cannot judge yet", blocking for a newer ledger state. Only the Leios copy turned it into a disconnect. So the verdict is now 'VerdictIgnore', the path 'StaleOCIN' already takes: accepted from the peer, neither processed nor relayed. A /far-future/ slot is a different matter -- that one is the peer lying, and 'handleHeaderArrival' still disconnects them. 'TraceLeiosAnnouncementPastHorizon' records the slot, and only the slot: a 'PastHorizonException' renders as a couple of kilobytes of era summary, and this fires once per announcement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The in-future check was judging against the immutable tip, whose forecast horizon early in a sync is tiny: the summary in one PastHorizon carried six eras collapsed at slot 0 and the last ending at slot 21600, against an announced slot of 1971193. Nearly every announcement was undatable, and the preceding commit could only ignore them. The volatile tip forecasts as far as we have any right to, so the same announcements become datable and are accepted instead. On the testnet that is the difference between ~150 announcements per run reaching a verdict of 'VerdictIgnore' and 1126 reaching 'VerdictProcess', with a residue of 4 -- all at the true chain tip, which a node 1.4M slots behind genuinely cannot place. 'validateAnnouncementHeader' keeps the immutable tip: an opcert revocation should only count once it cannot be rolled back. The two states are read in one 'atomically' so they cannot describe different chains, and the signature now names which is which. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Merging to release and tag a node with this (for further validation on the testnet). Hope to get your feedback on the change still @nfrisby |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Should fix or at least improve the situation with input-output-hk/ouroboros-leios#1039 and input-output-hk/ouroboros-leios#998. Both issues had various reports with PastHorizon or forecast range errors.
When syncing a testnet on preparing a release I got stuck multiple times and the root cause was the connection going down because of the LeiosNotify handler
throwIOing the PastHorizon error result duringjudgeHeaderArrival. I also noticed that we use the immutable tip to do time converstions, but the volatile one would have a slightly longer range and so switched to that.I did not see nodes using this fix get stuck so far.