Skip to content

fix(#116): split parent-walk backfill cap from RPC inline cap#117

Merged
Reiers merged 2 commits into
mainfrom
fix/116-parentwalk-cap
Jul 11, 2026
Merged

fix(#116): split parent-walk backfill cap from RPC inline cap#117
Reiers merged 2 commits into
mainfrom
fix/116-parentwalk-cap

Conversation

@Reiers

@Reiers Reiers commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Closes #116.

What

Bridge-off cold boot (--no-fallback-rpc) reached the seeded anchor via ChainExchange (#91/#109) and then never advanced. Every gossip block was skipped with backfillFail incrementing 1:1 with sub-rcv, and (once visibility was added) the terminal warn was bridge-off backfill gap 45 > cap 3.

Root cause: net/blockingest.parentWalkBackfill gated the anchor→tip gap on g.backfillCap (default 3 epochs) BEFORE calling chainExchangeBackfill. That 3-epoch cap was designed for the RPC inline path (inlineBackfill, one glif round-trip per epoch) where 3 is the right ceiling. The parent-walk path has completely different economics: chainExchangeBackfill iterates 900-tipset chunks capped internally at maxBackfillTipsets = 20000 and is designed to bridge whole anchor→tip stretches in one shot. The 3-epoch outer gate short-circuited it before it could run.

The docstring on chainExchangeBackfill even says so: "a snapshot-free boot from a stale quorum anchor can be thousands of epochs behind live head - so we LOOP."

Fix

Split the two paths' caps.

cap scope default rationale
backfillCap RPC inline path (inlineBackfill) 3 (unchanged) one glif round-trip per epoch — small ceiling protects the upstream
parentWalkCap (new) parent-walk path (parentWalkBackfill) 2880 (~24h on 30s epochs) chainxchg does one iterative request per 900-tipset chunk, bitswap fallback linear but bounded

parentWalkCap gates:

  1. The upfront gap > cap check at the top of parentWalkBackfill.
  2. The len(collected) > cap check inside the bitswap fallback walk.

chainExchangeBackfill's internal maxBackfillTipsets = 20000 remains the ultimate ceiling — nothing changes on that inner path.

Also fixed: silent backfill failure

process() swallowed the inlineBackfill error and just incremented backfillFailed, so an operator staring at a stalled sync saw backfillFail=N in the periodic stats and had no way to know why. This PR:

  • Adds log.Warnw("gossip inline-backfill failed", err, block_epoch, cur_head, parent_walk, chain_fetcher) at the error-swallow site.
  • Upgrades the chainxchg backfill failed; falling back to per-CID walk from Debugw to Infow (same reason).

Verification

Live darwin/arm64 vs calibration, 2026-07-11

Test 1 (fresh anchor, small gap): 4-min soak. Head advanced 3881463 → 3881477. sub-rcv=22 installed=4 backfilled=14 backfillFail=0. Zero Glif hits.

Test 2 (stale anchor, 63-epoch gap): Used a 30-minute-old anchor at epoch 3881418 (live head was 3881481 at boot). First gossip block triggered a single chainxchg backfill for the whole 63-epoch gap; head advanced to bh.Height - 1 in seconds and continued from there. backfillFail=0 throughout the run. Zero Glif hits.

Regression test

TestIngestor_ParentWalkAcceptsGapAboveRPCCap (new): 49-epoch gap (above DefaultBackfillCap=3, below DefaultParentWalkCap=2880) with parentWalk=true + fake ChainFetcher. Verifies:

  • backfillFailed = 0 (no cap rejection)
  • backfilled = 1 (one pass fired)
  • installed = 1 (head adopted the gossip block)
  • HeadEpoch = 60 (advanced to gossip target)
  • fetcher.calls = 1 (single chainxchg request for the whole gap)

TestIngestor_InlineBackfillRespectsCapsToSkip (existing) still passes — RPC-path 3-epoch ceiling is unchanged.

Suite

  • CGO_ENABLED=0 go build ./... clean
  • CGO_ENABLED=0 go vet ./... clean
  • go test -count=1 -timeout 300s ./... — all 55 packages ok, including the new test in net/blockingest

#104 impact

This blocks Phase 1 (snapshot-free boot) of the calibration dress rehearsal until landed. Without it, any bridge-off cold boot where init and daemon-start aren't back-to-back is wedged; with it, a stale-anchor cold boot stitches to live tip in one chainxchg round-trip and stays in sync via gossip thereafter.

Non-scope

  • The unified parentWalkBackfill code path itself is unchanged: chainxchg first, bitswap fallback on error, both bounded now by the same parentWalkCap.
  • No change to the RPC/glif path defaults or semantics.
  • Log-visibility change is opinionated but justified by the diagnosis experience (it took real live-log instrumentation to find this; the operator UX shouldn't require that).

Reiers added 2 commits July 11, 2026 13:56
Bridge-off cold boot ( --no-fallback-rpc ) reached the seeded anchor
via ChainExchange (#91/#109) but never advanced from there. Every
gossip block was skipped with backfillFail incrementing 1:1 with
sub-rcv, and the terminal log line was 'bridge-off backfill gap N >
cap 3' with N in the 40-60 range on any realistic cold boot.

Root cause: net/blockingest.parentWalkBackfill gated the anchor->tip
gap on g.backfillCap (default 3 epochs) BEFORE calling the chainxchg
strategy. That cap was designed for the RPC inline path
(inlineBackfill, one glif round-trip per epoch) where 3 is the right
ceiling. The parent-walk / chainxchg path has completely different
economics: chainExchangeBackfill iterates 900-tipset chunks capped
internally at maxBackfillTipsets=20000 and is meant to bridge whole
anchor->tip stretches in one shot, as the docstring literally says
('a snapshot-free boot from a stale quorum anchor can be thousands
of epochs behind live head - so we LOOP'). The outer 3-epoch gate
short-circuited it before it could run.

Fix: split into two caps.
  - backfillCap (existing, RPC path) — stays 3.
  - parentWalkCap (new) — defaults to 2880 epochs (~24h on 30s epochs).
    Gates both the upfront gap check in parentWalkBackfill and the
    inner collected-map check in the bitswap fallback branch.

Also: silent failure UX. The ingestor swallowed the inlineBackfill
error and just incremented backfillFailed, so an operator saw a
stalled sync and no clue why. Adds a Warnw with err+epoch+cur_head+
parent_walk+chain_fetcher at that error-swallow site, and upgrades
the chainxchg->per-CID-fallback line from Debugw to Infow for the
same reason.

Live verification (darwin/arm64 vs calibration, 2026-07-11):
  - Fresh anchor: 4-min soak, head 3881463 -> 3881477, backfillFail=0.
  - Stale anchor (30 min old, 63-epoch gap): boot to head 3881481 in
    seconds via a single chainxchg backfill, backfillFail=0 throughout.
  - Both: grep -c 'glif.io' <log> = 0.

Adds TestIngestor_ParentWalkAcceptsGapAboveRPCCap: a 49-epoch gap
(above DefaultBackfillCap=3, below DefaultParentWalkCap=2880) with a
fake ChainFetcher succeeds in one FetchTipsetChain call and advances
head to the gossip target. Blocked TestIngestor_InlineBackfillRespects
CapsToSkip stays green (RPC path cap unchanged).

Fixes #116. Blocks Phase 1 of the #104 calibration dress rehearsal
soak until landed.
@Reiers Reiers merged commit 44df3a5 into main Jul 11, 2026
1 check passed
@Reiers Reiers deleted the fix/116-parentwalk-cap branch July 11, 2026 12:08
Reiers added a commit that referenced this pull request Jul 11, 2026
…120)

Bridge-off mode (--no-fallback-rpc) has no polling Sync stale-reset
because the polling head-source is a no-op — so a daemon that was
stopped past parentWalkCap (~24h with #117 defaults) boots wedged: the
persisted anchor + header store are old and every parent-walk from the
fresh gossip head exceeds the cap. #51 solved this in the RPC path; this
change is the parallel code for bridge-off.

On daemon boot, if the persisted bootstrap-anchor.json is older than
--anchor-max-age (default 12h), re-run the same multi-source quorum
probe 'lantern repair' uses and overwrite the anchor with a fresh
finality. The subsequent boot then treats this as a normal fresh-anchor
start; the ChainExchange seed re-seats the header store from the fresh
anchor tipset, and #117's parentWalkCap covers the remaining gap.

Additionally: if the persisted store head is more than parentWalkCap
epochs behind the fresh anchor, drop the header store (chain state only,
same allow-list as reset.go) and let the seed re-seat it — covers the
'store so far behind that no parent-walk stitches it back' case.

New flags:
  --anchor-max-age <duration>  (default 12h; 0 disables)
  --auto-stale-reset <bool>    (default true; no-op outside bridge-off)

Guarantees:
- No-op outside bridge-off mode (RPC path handled by #51's
  StaleResetThreshold).
- Never touches keystore, JWT, tokens, secrets/, or backups/.
- Quorum probe failure is fail-warn (not fail-fatal): boot continues
  with the stale anchor; a probe blip must never brick the daemon.
- Write failure is fatal (partial anchor is worse than stale).

Tests: 8 unit tests covering triggered/skipped/probe-failure/
write-failure/disabled/RPC-mode/max-age-zero/no-anchor-file. Helper is
factored so the probe + writer are injectable — no live network needed.

Live smoke on darwin/calibration: 24h-stale anchor triggered the probe
(3/3 quorum in 328ms), fresh anchor written at epoch 3881573 (was
3881418, +155 epochs stale), ChainExchange seed re-seated the store,
gossip caught up with backfillFail=0 and zero Glif hits.

Fixes #118.

Co-authored-by: Reiers <reiers@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bridge-off cold boot: parentWalkBackfill outer cap (3 epochs) short-circuits chainxchg's stale-anchor stitching

1 participant