fix(#53): serve header backfill from bitswap/gateway, Glif as last resort

CHANGELOG.md

@@ -2,6 +2,32 @@
 
 All notable changes to Lantern.
 
+## v1.8.1 (2026-06-29)
+
+Sync resilience + chain-watcher fixes. Two field-reported bugs that together
+made a node fall behind live head and then stall its consumer's chain watcher.
+No protocol/wire changes, no new external dependencies.
+
+### Fixed
+
+- **#53 — header backfill off the Glif critical path.** Parent backfill (for
+  blocks landing at `head+N`) was served by Glif `FetchBlock` with an 8s timeout
+  in both the sync agent and the gossip ingestor; a slow/rate-limited Glif drove
+  `backfillFail` up and desynced the node (surfacing downstream as e.g.
+  `cannot draw randomness from future epoch ... (head ...)`). Backfill is now
+  served from the combined bitswap+gateway content-addressed fetcher, Glif as
+  last resort; `HeadEpoch`/`TipsetCIDsByHeight` stay RPC-shaped (gossipsub
+  supplies live heads). Fetcher resolved lazily to see the bitswap-enabled
+  fetcher rebuilt after libp2p start.
+- **#68 — `ChainGetTipSet(key)` served from the header store.** It previously
+  resolved only the synthetic current head and returned `ErrTipSetNotFound` for
+  any other key even with a populated header store, so Curio's `message/watch.go`
+  / `deps/apiinfo.go` looking up recent non-head tipset keys looped on
+  `tipset not in local store (only current head is cached in V1)`. New
+  `Store.GetTipSet(key)` reassembles any persisted tipset by key; `ChainGetTipSet`
+  falls through to it after the head fast-path. Distinct from #53 (this surfaces
+  even on a healthy synced node).
+
 ## v1.8.0 (2026-06-26)
 
 Security hardening: trust model, bootstrap, and auth. Self-audit (#60) of the

RELEASE-NOTES-v1.8.1.md

@@ -0,0 +1,78 @@
+# Lantern v1.8.1 — sync resilience + chain-watcher fixes
+
+**Update if:** you run a Lantern node behind Curio (or any Lotus-API consumer
+that watches the chain), especially in a zero/low-Glif configuration. This
+release fixes two distinct bugs that, together, made a node fall behind live
+head and then stall its consumer's chain watcher. Both were reported from the
+field. No protocol or wire changes.
+
+## Highlights
+
+### Header backfill off the Glif critical path (#53)
+Live blocks arrive over gossipsub fine, but when a block lands at `head+N`
+(N>1) the **parent backfill** was served by Glif `FetchBlock` calls with an 8s
+timeout, in *both* the polling sync agent and the gossip ingestor. When Glif was
+slow or rate-limited, those calls timed out, `backfillFail` climbed, the head
+could not advance contiguously, and the node desynced — surfacing downstream as,
+for example:
+
+```
+cannot draw randomness from future epoch <E> (head <E-thousands>)
+```
+
+(The PDP prover asks for randomness at an epoch the stalled node hasn't reached
+yet, so the request is correctly refused — a *symptom* of the desync, not a
+proving bug.)
+
+Backfill is now served from the combined **bitswap + gateway** fetcher (the same
+content-addressed, CID-verified fetcher already running for state reads), with
+Glif demoted to last resort. `HeadEpoch` / `TipsetCIDsByHeight` stay RPC-shaped
+(gossipsub already supplies live heads), so during normal operation Glif is off
+the head-advancement path entirely. The fetcher is resolved lazily so the
+sync/gossip paths see the bitswap-enabled fetcher (rebuilt after libp2p comes
+up), not a stale snapshot.
+
+### ChainGetTipSet served from the header store (#68)
+`ChainGetTipSet(key)` only ever resolved the synthetic current head and returned
+`ErrTipSetNotFound` for any other key — **even with a populated header store
+wired**. Its siblings (`ChainGetTipSetByHeight`, `ChainGetBlock`, the randomness
+path) all already fell through to the header store; only the by-key lookup was
+left stubbed.
+
+Curio's `message/watch.go` and `deps/apiinfo.go` request specific recent
+(non-head) tipset keys. Before this fix they were refused, producing a tight
+error loop:
+
+```
+lantern: tipset not in local store (only current head is cached in V1)
+failed to get tipset: chain: lantern: tipset not in local store ...
+no new tipset in CurioChainSched.update
+```
+
+`ChainGetTipSet` now falls through to a new `Store.GetTipSet(key)` that
+reassembles any persisted tipset directly from its block headers (returning
+`ErrNotFound` only when a constituent block is genuinely missing). This is
+distinct from #53: #53 is the node falling behind; #68 surfaces even on a
+healthy, synced node.
+
+## On `--vm-bridge-rpc` as a default
+
+A tester asked whether `--vm-bridge-rpc https://api.node.glif.io/rpc/v1` should
+be added by default to dodge these failures. It is a legitimate **fallback** and
+fine to set today if you need resilience right now — but it is **not** the
+default, by design. Defaulting it on would route every node silently back
+through Glif and mask exactly the bugs above. The fixes here address the root
+causes so the bridge stays optional.
+
+## Upgrade notes
+
+- No configuration changes required. Drop in the new binary and restart.
+- No new external dependencies.
+
+## Verification
+
+CGO-free build, `go vet`, `gofmt`, and the hermetic test suite
+(`-short`, `LANTERN_OFFLINE=1`) are green across the module. New unit tests
+cover the #68 by-key lookup (head, non-head historical, empty-key, and
+missing-block cases); the #53 adapter is covered by its existing suite. The live
+mainnet node was not touched during development.

chain/header/store/store.go

@@ -262,6 +262,19 @@ func (s *Store) canonicalAt(epoch abi.ChainEpoch) (*ltypes.TipSet, error) {
 	return s.loadTipSet(tsk)
 }
 
+// GetTipSet resolves a tipset directly by its key (the set of block
+// CIDs). It reads each block header from the store and reassembles the
+// tipset. Returns ErrNotFound if any constituent block is missing. Unlike
+// GetTipSetByHeight this does NOT require the tipset to be on the
+// canonical chain — it serves any tipset whose headers were persisted,
+// which is what callers like ChainGetTipSet(key) need.
+func (s *Store) GetTipSet(tsk ltypes.TipSetKey) (*ltypes.TipSet, error) {
+	if tsk.IsEmpty() {
+		return nil, ErrNotFound
+	}
+	return s.loadTipSet(tsk)
+}
+
 func (s *Store) loadTipSet(tsk ltypes.TipSetKey) (*ltypes.TipSet, error) {
 	cids := tsk.Cids()
 	blocks := make([]*ltypes.BlockHeader, 0, len(cids))

chain/header/store/store_test.go

@@ -2,6 +2,7 @@ package store_test
 
 import (
 	"context"
+	"errors"
 	"path/filepath"
 	"testing"
 
@@ -215,3 +216,52 @@ func TestOnHeadChange(t *testing.T) {
 		t.Fatal("OnHeadChange callback did not fire")
 	}
 }
+
+// TestGetTipSetByKey verifies GetTipSet(key) resolves a tipset directly
+// from persisted headers, including a NON-head historical tipset (the
+// Curio chain-watcher scenario from #68), and returns ErrNotFound when a
+// constituent block is missing or the key is empty.
+func TestGetTipSetByKey(t *testing.T) {
+	s, _ := newStore(t, false)
+
+	// Build a short linear chain: genesis -> h1 -> h2 (h2 is head).
+	g := mkBlock(t, 0, nil, 1000, "g")
+	require.NoError(t, s.Put(g))
+	gts, err := ltypes.NewTipSet([]*ltypes.BlockHeader{g})
+	require.NoError(t, err)
+	require.NoError(t, s.SetHead(context.Background(), gts))
+
+	b1 := mkBlock(t, 1, []cid.Cid{g.Cid()}, 1000, "1")
+	require.NoError(t, s.Put(b1))
+	ts1, err := ltypes.NewTipSet([]*ltypes.BlockHeader{b1})
+	require.NoError(t, err)
+	require.NoError(t, s.SetHead(context.Background(), ts1))
+
+	b2 := mkBlock(t, 2, []cid.Cid{b1.Cid()}, 1000, "2")
+	require.NoError(t, s.Put(b2))
+	ts2, err := ltypes.NewTipSet([]*ltypes.BlockHeader{b2})
+	require.NoError(t, err)
+	require.NoError(t, s.SetHead(context.Background(), ts2))
+
+	// Resolve a NON-head historical tipset (epoch 1) directly by key.
+	got, err := s.GetTipSet(ts1.Key())
+	require.NoError(t, err)
+	require.Equal(t, abi.ChainEpoch(1), got.Height())
+	require.Equal(t, ts1.Key().String(), got.Key().String())
+
+	// Resolve the head tipset by key too.
+	gotHead, err := s.GetTipSet(ts2.Key())
+	require.NoError(t, err)
+	require.Equal(t, abi.ChainEpoch(2), gotHead.Height())
+
+	// Empty key -> ErrNotFound.
+	_, err = s.GetTipSet(ltypes.EmptyTSK)
+	require.ErrorIs(t, err, store.ErrNotFound)
+
+	// Key referencing a block we never persisted -> ErrNotFound.
+	missing := mkBlock(t, 3, []cid.Cid{b2.Cid()}, 1000, "missing")
+	missingTS, err := ltypes.NewTipSet([]*ltypes.BlockHeader{missing})
+	require.NoError(t, err)
+	_, err = s.GetTipSet(missingTS.Key())
+	require.True(t, errors.Is(err, store.ErrNotFound), "missing block should be ErrNotFound, got %v", err)
+}

cmd/lantern/bitswap_backed_source.go

@@ -0,0 +1,105 @@
+// bitswapBackedSource: an RPCSource / BackfillSource that serves the
+// content-addressed FetchBlock(cid) call from the combined (gateway +
+// bitswap) fetcher first, falling back to Glif only on a miss. The two
+// genuinely RPC-shaped methods (HeadEpoch, TipsetCIDsByHeight) stay on
+// Glif as a last resort.
+//
+// Issue #53: header backfill was hardcoded to a Glif client. When Glif was
+// slow/rate-limited the parent-backfill FetchBlock calls timed out, the
+// header store couldn't advance contiguously, and the node fell behind live
+// head. FetchBlock is content-addressed, so it can be served by the same
+// bitswap/gateway fetcher already running for state reads — taking parent
+// backfill off the Glif critical path.
+
+package main
+
+import (
+	"context"
+
+	"github.com/filecoin-project/go-state-types/abi"
+	"github.com/ipfs/go-cid"
+
+	hstore "github.com/Reiers/lantern/chain/header/store"
+	"github.com/Reiers/lantern/chain/types"
+	"github.com/Reiers/lantern/net/blockingest"
+	"github.com/Reiers/lantern/net/combined"
+)
+
+// blockGetter is the minimal surface the adapter needs from the combined
+// fetcher: a content-addressed, CID-verified Get. *combined.Fetcher
+// satisfies it.
+type blockGetter interface {
+	Get(ctx context.Context, c cid.Cid) ([]byte, error)
+}
+
+// rpcBlockSource is the RPC-shaped fallback the adapter delegates to.
+// *glif.Client satisfies it. Kept as an interface so the fallback path is
+// unit-testable without a live endpoint.
+type rpcBlockSource interface {
+	HeadEpoch(ctx context.Context) (abi.ChainEpoch, error)
+	TipsetCIDsByHeight(ctx context.Context, h abi.ChainEpoch) ([]cid.Cid, error)
+	FetchBlock(ctx context.Context, k cid.Cid) (*types.BlockHeader, error)
+}
+
+// bitswapBackedSource wraps a Glif client and a content-addressed block
+// getter. HeadEpoch/TipsetCIDsByHeight delegate to Glif (inherently
+// RPC-shaped). FetchBlock tries the content-addressed getter first and
+// falls back to Glif on any error, so parent backfill no longer sits on
+// the Glif critical path.
+//
+// The getter is resolved lazily via getFetcher() on each call. This matters
+// because the combined fetcher is REBUILT after libp2p comes up (to add the
+// bitswap source), and the sync/gossip wiring is constructed before that
+// rebuild — a lazily-resolved getter always sees the current, bitswap-
+// enabled fetcher rather than a stale gateway+glif-only snapshot.
+type bitswapBackedSource struct {
+	glif       rpcBlockSource
+	getFetcher func() blockGetter
+}
+
+func newBitswapBackedSource(g rpcBlockSource, getFetcher func() blockGetter) *bitswapBackedSource {
+	return &bitswapBackedSource{glif: g, getFetcher: getFetcher}
+}
+
+// HeadEpoch delegates to Glif (live head also arrives via gossipsub, so in
+// steady state this is rarely the limiting call).
+func (s *bitswapBackedSource) HeadEpoch(ctx context.Context) (abi.ChainEpoch, error) {
+	return s.glif.HeadEpoch(ctx)
+}
+
+// TipsetCIDsByHeight delegates to Glif (RPC-shaped: maps height -> the
+// canonical tipset's block CIDs).
+func (s *bitswapBackedSource) TipsetCIDsByHeight(ctx context.Context, h abi.ChainEpoch) ([]cid.Cid, error) {
+	return s.glif.TipsetCIDsByHeight(ctx, h)
+}
+
+// FetchBlock returns a single CID-verified BlockHeader. It is served from
+// the content-addressed fetcher (gateway race + bitswap) first; on any miss
+// or decode error it falls back to Glif. The fetcher's Get is already
+// CID-verified, and DecodeBlock re-derives the header from the raw bytes,
+// so a bad block can't slip through this path.
+func (s *bitswapBackedSource) FetchBlock(ctx context.Context, k cid.Cid) (*types.BlockHeader, error) {
+	var f blockGetter
+	if s.getFetcher != nil {
+		f = s.getFetcher()
+	}
+	if f != nil {
+		if raw, err := f.Get(ctx, k); err == nil {
+			if bh, derr := types.DecodeBlock(raw); derr == nil {
+				return bh, nil
+			}
+			// Decode failed on otherwise-fetched bytes: fall through to
+			// Glif rather than fail the backfill outright.
+		}
+	}
+	return s.glif.FetchBlock(ctx, k)
+}
+
+// Compile-time guards against the REAL consumer interfaces, so this file
+// breaks the build if either consumer's surface drifts from what the
+// adapter provides.
+var (
+	_ hstore.RPCSource           = (*bitswapBackedSource)(nil)
+	_ blockingest.BackfillSource = (*bitswapBackedSource)(nil)
+	_ blockGetter                = (*combined.Fetcher)(nil)
+)