Header backfill falls back to Glif → node loses sync when Glif is slow

[Reiers]

Symptom

Tester reports (Rabinovitch, Beck — 2026-06-22): node periodically falls behind live head; logs show:

[sync] polls=49 advances=18 ... lastErr="fetch epoch 6127847: Post \"https://api.node.glif.io/rpc/v1\": context deadline exceeded"
[sync] ... lastErr="glif: client canceled request (code=-32232)"
[gossip-block] ... backfilled=20 backfillFail=75 ...

Curio dashboard then shows "No, slow (Ns behind)". (0–2 epochs behind is normal/UI-strict; 5–6+ is the real problem.)

Root cause (confirmed in code)

In cmd/lantern/main.go both the polling header Sync agent and the gossip ingestor's inline backfill source are hardcoded to a Glif client (8s timeout):

• Sync agent: src := glif.New(glifURLForNetwork(network), 8*time.Second) → hstore.NewSync(store, src, ...)
• Gossip backfill: gossipSrc := glif.New(glifURLForNetwork(network), 8*time.Second) → startGossipBlocks(..., gossipSrc, ...)

Live blocks arrive via gossipsub fine, but when a block lands at head+N (N>1), the parent backfill = Glif FetchBlock calls. When Glif is slow/rate-limited those time out → backfillFail climbs → head can't advance contiguously → desync. The embedded bitswap fetcher (already running for state reads) is not in the header-sync path.

Fix plan

RPCSource = {HeadEpoch, TipsetCIDsByHeight(h), FetchBlock(cid)}. FetchBlock(cid) is content-addressed and can be served by the bitswap/combined fetcher (net/bitswap Client.Get → decode BlockHeader). HeadEpoch / TipsetCIDsByHeight are inherently RPC-shaped, but gossipsub already supplies live heads, so during normal run Glif is only needed for parent backfill — which is the bitswap-able part.

• New RPCSource adapter: FetchBlock via combined/bitswap fetcher first, Glif fallback. Keep HeadEpoch/TipsetCIDsByHeight on Glif (last resort) or gossip-head.
• Wire the adapter into both the Sync agent and the gossip ingestor backfill source.
• Net: parent backfill stops hitting Glif → kills the timeout-driven desync. Glif drops to last-resort for the two genuinely-RPC methods only.
• Bonus: revisit gossip-backfill timeout + MaxBacktrack.

Related: #50 (message-block availability over bitswap), PR #52 (prefetch-on-send).

Refs

• cmd/lantern/main.go Sync wiring (~line 726) + gossip wiring (~line 855)
• net/glif/client.go FetchBlock (223), net/bitswap/client.go Get (168), net/combined/fetcher.go Get (115)