Skip to content

fix: skip a headers that does not start on the locator of our outstanding getheaders, so that an announcement before the answer keeps the connection (#92) - #94

Merged
lorenzolfm merged 10 commits into
mainfrom
issue-92
Nov 14, 2026

Conversation

@lorenzolfm

@lorenzolfm lorenzolfm commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Closes #92. The first option of #92: skip an announcement while our getheaders is out.

What

  • Download::locator: the locator of the getheaders for each download. SyncState::tick now uses it. The locator of a branch or a low-work sync is its last hash, then genesis.
  • Download::is_answered_by: for each download that is not Download::Chain, a headers is the answer when it is empty, or when its first header is on a hash of that locator.
  • receive_answer checks the answer before the state moves. A headers that is not the answer gives Answer::Skipped with the download. Peer::receive_headers puts it back with the same sent and gives NoReply. The state, the chain and branch_tip do not change, and the headers timeout does not restart.
  • Chain::extend_on_locator gives None for a batch that is not on our locator, and changes nothing. For Download::Chain, this search is the check. ExtendError::NotOnLocator is gone.
  • receive_on_genesis uses Chain::extend: the check found that the answer starts on genesis, which is our tip. Its match on Extended and its crash are gone.
  • SyncState::Outstanding keeps the hash of our tip when we send the getheaders. receive_headers crashes when our tip changed before the answer. Step::Done holds a Box<Peer>, for clippy::large_enum_variant.
  • The fuzz model of machines skips in the same way, with PeerModel::locator. Its locator of a branch or a low-work sync has genesis below the last hash. The check finds the height of the first hash and gives it to its extend_on_locator.
  • Seeds:
    • 5 rows have a new end. Their first header is not on our locator, so elo now skips them: headers-off-tip, headers-off-tip-after-full, headers-off-tip-after-sync-twice, sync-low-work-full-off-tip-signet and sync-low-work-regtest-second-off-last.
    • New headers-off-tip-after-sync-twice-then-answer: the steps of An announcement that arrives before the answer to our getheaders ends the connection #92. An announcement off our tip, one more off our locator, then the answer ends the sync.
    • New sync-low-work-regtest-first-on-genesis: the second batch of a low-work sync starts on genesis. elo ends the connection at once with HeadersSync(NotContinuous).
    • New sync-low-work-regtest-second-break-second: the second batch of a low-work sync breaks at its second header. HeadersSync(NotContinuous) had a seed only through a first header off the last one, and that is now a skip.
  • Tests:
    • headers_off_our_locator_while_our_getheaders_is_outstanding_are_skipped replaces headers_off_our_tip_in_answer_to_our_getheaders_end_the_connection. It has the same 2 peers, one in the first sync and one after an announcement off our tip. Each skips the headers, keeps its chain, and then takes the answer.
    • headers_off_our_locator_do_not_restart_the_headers_timeout.
    • New headers_on_a_header_in_a_gap_of_our_locator_are_skipped replaces the Chain test extend_on_locator_refuses_a_batch_on_a_header_that_is_not_in_the_locator. That Chain test is now extend_on_locator_takes_no_batch_on_a_header_that_is_not_in_the_locator: its row in the gap gives None and keeps the tip.
    • New headers_off_the_locator_of_a_low_work_sync_are_skipped: a low-work sync skips a batch off its locator and then ends. It kills is_answered_by -> true without cfg(fuzzing).
    • New low_work_sync_ends_the_connection_at_once_on_an_answer_from_genesis and branch_download_ends_the_connection_at_once_on_an_answer_from_genesis.
    • refused_answer_keeps_the_chain loses its row for NotOnLocator. low_work_sync_that_refuses_a_batch_ends_the_connection now breaks at the second header.
    • New L3 announcement_before_the_answer_to_our_getheaders_keeps_the_connection: 10 times, bitcoind drops its tip with invalidateblock and mines 2 blocks with one RPC. It announces the first block, which is off our tip, so elo asks. Its announcement of the second block can come before the answer. At the end, elo has the tip of bitcoind and the connection.
    • L3 announcement_on_a_branch_that_lost_to_ours_after_a_low_work_sync_gets_no_getheaders has no ban. settle does ping RPCs until bitcoind sends no more headers or inv and gets no more getheaders. Then the test checks that elo logged no new tip.
  • README: one sentence.
  • Proof:
    • At c06bb4a, after the illegal-states fixes (34d2926, 85fdc5e, c06bb4a): elo-test (229 tests, L3 included), elo-clippy, elo-fuzz, elo-fmt and elo-comments pass in Nix. A thermo-nuclear review of c06bb4a has no findings.
    • After the TigerStyle fixes (4083ec9, c6ffd05, 6d44b15): 205 lib tests pass, 17 L3 tests pass in a netns, and 207 tests pass with --cfg fuzzing. clippy passes with and without --cfg fuzzing, and so do cargo fmt --check and elo-comments. I did not run nix flake check, cargo mutants or the fuzz again.
    • A test without genesis in the locator: 14 tests fail, the seed table among them. A test with no skip: the 2 skip tests crash in Chain::extend_on_locator.
    • Before the fixes:
      • 225 tests pass in a netns, 17 of them L3. nix flake check passes. The 204 model and seed-table tests pass with --cfg fuzzing.
      • The new L3 test without the skip: 5 of 5 runs end the connection with headers do not start at a header of our locator. With the skip: 5 of 5 alone, 30 of 30 in 3 loops at a time, and in the Nix sandbox.
      • The L3 test with no ban passes without the skip too, here and in the Nix sandbox (3 of 3). bitcoind sent only inv in its reorg, so no announcement came before an answer. So that test does not prove An announcement that arrives before the answer to our getheaders ends the connection #92. The new L3 test does.
      • cargo mutants --in-diff: 10 mutants, 4 caught, 6 unviable, 0 missed.
      • machines fuzz from a copy of the committed corpus, 300 s: 109,240 runs, no crash.

Why

Skip, not read each headers by its content. Option 2 of #92 needs elo to process an announcement in the middle of a download. A download can be a branch or a low-work sync, so an announcement on our tip would change the chain under it. Option 1 needs no new state. The answer, or the next getheaders, gets the skipped headers again.

An answer starts on our locator. bitcoind sends the headers after the first hash of the locator on its chain (net_processing.cpp:4441, validation.cpp:127). An announcement starts on a header that the peer knows we have (net_processing.cpp:5880). That is our tip or a header that it sent us. So the check uses only the hashes that elo sent, and no state about the peer.

An announcement on the locator is read as the answer. For example, an announcement on our tip while our getheaders with our tip is out. This is the overlap of the two options. The answer of bitcoind starts on the same header or one below, so the announcement is a correct start of an answer. elo takes it, and the real answer then comes as an announcement. It is not on our new tip, so elo asks again. This costs one round trip and no error.

Compute the locator again, do not keep it. The chain does not change while a getheaders is out, so Download::locator gives the same hashes as the getheaders. A copy in Outstanding would be a second source for the same value. Outstanding keeps only our tip, for the assert. The cost is one locator for each headers, at most about 40 hashes.

A skipped branch-tip move is lost. An announcement on branch_tip while our getheaders is out is skipped, and branch_tip stays. The answer then has the branch, so it sets branch_tip again. The model skips in the same way, so the driver can compare the tips after each message.

Genesis in a locator of one hash. If a peer left the branch of that hash, it answers from genesis (validation.cpp:144). Without genesis in the locator, elo skipped that answer, and the connection ended with HeadersTimeout after 2 min. With genesis, the answer is on the locator, and the sync refuses it at once with NotContinuous, as on main. Genesis is not added when the last hash is genesis. The other way was to always read a headers on genesis as the answer. That keeps the old getheaders on the wire, but then the check accepts a hash that elo did not send.

For Download::Chain, the chain does the check. A first version checked the locator in is_answered_by, and extend_on_locator then found the same hash again and crashed when it did not find it. The two searches agreed only because both read locator_heights. Now extend_on_locator gives None off the locator, so there is one search and no crash.

Keep our tip in Outstanding. The check makes our locator again. That is the locator that we sent only while nothing writes the chain while our getheaders is out. Only a convention holds this: receive_inv and receive_announcement act only after the sync. The hash of our tip and the assert make a break of that convention a crash, not a check against the wrong locator. To keep the locator itself is not sufficient: extend_on_locator reads the chain, not the locator that we sent.

No event for a skip, and no limit on the count. The events are for each connection, and a skip is one message. The timeout limits the skips to 2 min after each getheaders, and a skip does no proof-of-work check. A limit on the count is a new rule for peers, so I did not add one.

A skip does not restart the timeout. A peer that sends only headers off our locator still gets HeadersTimeout 2 min after our getheaders.

Not confident:

  • A peer can send headers off our locator for 2 min, and elo decodes each batch. It does no proof-of-work check on them, so a batch costs less than an answer.
  • The getheaders of a branch or a low-work sync has a second hash now. The L3 low-work tests pass with it, but no L3 test makes bitcoind answer from genesis.

…anding `getheaders` (#92)

An announcement can come before the answer to our `getheaders`. elo
read it as the answer, and ended the connection with `NotOnLocator`.
An answer starts on a hash of our locator, so elo now skips a `headers`
that does not. The answer, or the next `getheaders`, gets those headers
again. A skip does not restart the headers timeout.
…h that lost, with no ban (#92)

The ban was only there to avoid #92. The test now waits until bitcoind
and elo relay no more headers, and checks that elo kept its tip and its
connection.
… `getheaders` (#92)

bitcoind drops its tip and mines 2 blocks, 10 times. Each time it
announces the first block off our tip, and elo asks. Its announcement
of the second block can come before the answer. Without the fix, elo
ended the connection with `NotOnLocator` in 5 of 5 runs.
…o that an answer from genesis ends the connection at once (#92)

A peer that does not have the one hash of such a locator on its chain
answers from genesis. elo skipped that answer as an announcement, and
the connection ended with `HeadersTimeout` after 2 min. With genesis in
the locator, the answer starts on a hash of the locator. The sync then
refuses it at once with `NotContinuous`, as before the skip. The model
puts genesis below the last hash too, so the script can write an
answer on it.
…state moves, so that a skip leaves the state as it is (#92)

The skip took the outstanding `getheaders` out of the state and put it
back. The check now reads the state in place, and only an answer moves
it.
…tOnLocator` (#92)

elo now skips a `headers` that does not start on a hash of our locator,
so no peer can send such an answer. `Chain::extend_on_locator` and the
model found the same headers again and returned `NotOnLocator`. That
error can no longer happen, so both now crash on it, as a bug in elo.
A Peer test takes the place of the Chain test: a branch on a header in a
gap of our locator is skipped.
…nd drop the crash (#92)

The `headers` check found the first hash in our locator, and gave only
`true` or `false`. `Chain::extend_on_locator` then found the same hash
again, and crashed when it did not find it. The two searches agreed
only because both read `locator_heights`.

Now `Chain::extend_on_locator` gives `None` for headers that do not
start on our locator, and changes nothing. For `Download::Chain`, this
search is the check. `receive_answer` gives `Answer::Skipped` with the
download, and the `Peer` puts it back with the same `sent`. The check of
the other downloads moves into `receive_answer`, before the state
moves.

`receive_on_genesis` uses `Chain::extend`: the check already found that
the answer starts on genesis, which is our tip. Its match on `Extended`
and its crash go.

The model finds the height of the first hash in the check, and gives
it to its `extend_on_locator`, so the crash of the model goes too.

Removed: 3 crashes, the match on `Extended` in `receive_on_genesis`, the
second search of our locator, and the `if let` before the match on the
state. Added: `Answer`, with 2 variants.
…n it changed before the answer (#92)

The check of a `headers` makes our locator again. It is the locator that
we sent only while nothing writes the chain while our `getheaders` is
out. Only a convention holds this: `receive_inv` and
`receive_announcement` act only after the sync. A change that takes an
announcement while our `getheaders` is out breaks it, and the check then
reads the wrong locator with no error.

`SyncState::Outstanding` now keeps the hash of our tip when we send the
`getheaders`. `receive_headers` crashes when our tip is a different one.
To keep the locator itself is not sufficient: `extend_on_locator` reads
the chain, not the locator that we sent.

The hash makes `Peer` 32 bytes larger, and `Step::Done` is then 216
bytes larger than `Step::Reply`, above the limit of
`clippy::large_enum_variant`. `Step::Done` now holds a `Box<Peer>`: one
allocation for each connection.
… the sync (#92)

After the change to `Chain::extend_on_locator`, only the downloads that
are not `Download::Chain` use `Download::is_answered_by`. No test that
runs without `cfg(fuzzing)` sent such a download headers off its
locator, so cargo-mutants did not kill `is_answered_by -> true`.
@lorenzolfm
lorenzolfm merged commit 67cf01c into main Nov 14, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An announcement that arrives before the answer to our getheaders ends the connection

1 participant