fix(#116): split parent-walk backfill cap from RPC inline cap#117
Merged
Conversation
Bridge-off cold boot ( --no-fallback-rpc ) reached the seeded anchor via ChainExchange (#91/#109) but never advanced from there. Every gossip block was skipped with backfillFail incrementing 1:1 with sub-rcv, and the terminal log line was 'bridge-off backfill gap N > cap 3' with N in the 40-60 range on any realistic cold boot. Root cause: net/blockingest.parentWalkBackfill gated the anchor->tip gap on g.backfillCap (default 3 epochs) BEFORE calling the chainxchg strategy. That cap was designed for the RPC inline path (inlineBackfill, one glif round-trip per epoch) where 3 is the right ceiling. The parent-walk / chainxchg path has completely different economics: chainExchangeBackfill iterates 900-tipset chunks capped internally at maxBackfillTipsets=20000 and is meant to bridge whole anchor->tip stretches in one shot, as the docstring literally says ('a snapshot-free boot from a stale quorum anchor can be thousands of epochs behind live head - so we LOOP'). The outer 3-epoch gate short-circuited it before it could run. Fix: split into two caps. - backfillCap (existing, RPC path) — stays 3. - parentWalkCap (new) — defaults to 2880 epochs (~24h on 30s epochs). Gates both the upfront gap check in parentWalkBackfill and the inner collected-map check in the bitswap fallback branch. Also: silent failure UX. The ingestor swallowed the inlineBackfill error and just incremented backfillFailed, so an operator saw a stalled sync and no clue why. Adds a Warnw with err+epoch+cur_head+ parent_walk+chain_fetcher at that error-swallow site, and upgrades the chainxchg->per-CID-fallback line from Debugw to Infow for the same reason. Live verification (darwin/arm64 vs calibration, 2026-07-11): - Fresh anchor: 4-min soak, head 3881463 -> 3881477, backfillFail=0. - Stale anchor (30 min old, 63-epoch gap): boot to head 3881481 in seconds via a single chainxchg backfill, backfillFail=0 throughout. - Both: grep -c 'glif.io' <log> = 0. Adds TestIngestor_ParentWalkAcceptsGapAboveRPCCap: a 49-epoch gap (above DefaultBackfillCap=3, below DefaultParentWalkCap=2880) with a fake ChainFetcher succeeds in one FetchTipsetChain call and advances head to the gossip target. Blocked TestIngestor_InlineBackfillRespects CapsToSkip stays green (RPC path cap unchanged). Fixes #116. Blocks Phase 1 of the #104 calibration dress rehearsal soak until landed.
This was referenced Jul 11, 2026
Reiers
added a commit
that referenced
this pull request
Jul 11, 2026
…120) Bridge-off mode (--no-fallback-rpc) has no polling Sync stale-reset because the polling head-source is a no-op — so a daemon that was stopped past parentWalkCap (~24h with #117 defaults) boots wedged: the persisted anchor + header store are old and every parent-walk from the fresh gossip head exceeds the cap. #51 solved this in the RPC path; this change is the parallel code for bridge-off. On daemon boot, if the persisted bootstrap-anchor.json is older than --anchor-max-age (default 12h), re-run the same multi-source quorum probe 'lantern repair' uses and overwrite the anchor with a fresh finality. The subsequent boot then treats this as a normal fresh-anchor start; the ChainExchange seed re-seats the header store from the fresh anchor tipset, and #117's parentWalkCap covers the remaining gap. Additionally: if the persisted store head is more than parentWalkCap epochs behind the fresh anchor, drop the header store (chain state only, same allow-list as reset.go) and let the seed re-seat it — covers the 'store so far behind that no parent-walk stitches it back' case. New flags: --anchor-max-age <duration> (default 12h; 0 disables) --auto-stale-reset <bool> (default true; no-op outside bridge-off) Guarantees: - No-op outside bridge-off mode (RPC path handled by #51's StaleResetThreshold). - Never touches keystore, JWT, tokens, secrets/, or backups/. - Quorum probe failure is fail-warn (not fail-fatal): boot continues with the stale anchor; a probe blip must never brick the daemon. - Write failure is fatal (partial anchor is worse than stale). Tests: 8 unit tests covering triggered/skipped/probe-failure/ write-failure/disabled/RPC-mode/max-age-zero/no-anchor-file. Helper is factored so the probe + writer are injectable — no live network needed. Live smoke on darwin/calibration: 24h-stale anchor triggered the probe (3/3 quorum in 328ms), fresh anchor written at epoch 3881573 (was 3881418, +155 epochs stale), ChainExchange seed re-seated the store, gossip caught up with backfillFail=0 and zero Glif hits. Fixes #118. Co-authored-by: Reiers <reiers@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #116.
What
Bridge-off cold boot (
--no-fallback-rpc) reached the seeded anchor via ChainExchange (#91/#109) and then never advanced. Every gossip block was skipped withbackfillFailincrementing 1:1 withsub-rcv, and (once visibility was added) the terminal warn wasbridge-off backfill gap 45 > cap 3.Root cause:
net/blockingest.parentWalkBackfillgated the anchor→tip gap ong.backfillCap(default 3 epochs) BEFORE callingchainExchangeBackfill. That 3-epoch cap was designed for the RPC inline path (inlineBackfill, one glif round-trip per epoch) where 3 is the right ceiling. The parent-walk path has completely different economics:chainExchangeBackfilliterates 900-tipset chunks capped internally atmaxBackfillTipsets = 20000and is designed to bridge whole anchor→tip stretches in one shot. The 3-epoch outer gate short-circuited it before it could run.The docstring on
chainExchangeBackfilleven says so: "a snapshot-free boot from a stale quorum anchor can be thousands of epochs behind live head - so we LOOP."Fix
Split the two paths' caps.
backfillCapinlineBackfill)3(unchanged)parentWalkCap(new)parentWalkBackfill)2880(~24h on 30s epochs)parentWalkCapgates:gap > capcheck at the top ofparentWalkBackfill.len(collected) > capcheck inside the bitswap fallback walk.chainExchangeBackfill's internalmaxBackfillTipsets = 20000remains the ultimate ceiling — nothing changes on that inner path.Also fixed: silent backfill failure
process()swallowed theinlineBackfillerror and just incrementedbackfillFailed, so an operator staring at a stalled sync sawbackfillFail=Nin the periodic stats and had no way to know why. This PR:log.Warnw("gossip inline-backfill failed", err, block_epoch, cur_head, parent_walk, chain_fetcher)at the error-swallow site.chainxchg backfill failed; falling back to per-CID walkfromDebugwtoInfow(same reason).Verification
Live darwin/arm64 vs calibration, 2026-07-11
Test 1 (fresh anchor, small gap): 4-min soak. Head advanced 3881463 → 3881477.
sub-rcv=22 installed=4 backfilled=14 backfillFail=0. Zero Glif hits.Test 2 (stale anchor, 63-epoch gap): Used a 30-minute-old anchor at epoch 3881418 (live head was 3881481 at boot). First gossip block triggered a single chainxchg backfill for the whole 63-epoch gap; head advanced to
bh.Height - 1in seconds and continued from there.backfillFail=0throughout the run. Zero Glif hits.Regression test
TestIngestor_ParentWalkAcceptsGapAboveRPCCap(new): 49-epoch gap (aboveDefaultBackfillCap=3, belowDefaultParentWalkCap=2880) withparentWalk=true+ fakeChainFetcher. Verifies:backfillFailed = 0(no cap rejection)backfilled = 1(one pass fired)installed = 1(head adopted the gossip block)HeadEpoch = 60(advanced to gossip target)fetcher.calls = 1(single chainxchg request for the whole gap)TestIngestor_InlineBackfillRespectsCapsToSkip(existing) still passes — RPC-path 3-epoch ceiling is unchanged.Suite
CGO_ENABLED=0 go build ./...cleanCGO_ENABLED=0 go vet ./...cleango test -count=1 -timeout 300s ./...— all 55 packages ok, including the new test innet/blockingest#104 impact
This blocks Phase 1 (snapshot-free boot) of the calibration dress rehearsal until landed. Without it, any bridge-off cold boot where init and daemon-start aren't back-to-back is wedged; with it, a stale-anchor cold boot stitches to live tip in one chainxchg round-trip and stays in sync via gossip thereafter.
Non-scope
parentWalkBackfillcode path itself is unchanged: chainxchg first, bitswap fallback on error, both bounded now by the sameparentWalkCap.