diff --git a/CHANGELOG.md b/CHANGELOG.md index 9246b2e..a572465 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,32 @@ All notable changes to Lantern. +## v1.8.1 (2026-06-29) + +Sync resilience + chain-watcher fixes. Two field-reported bugs that together +made a node fall behind live head and then stall its consumer's chain watcher. +No protocol/wire changes, no new external dependencies. + +### Fixed + +- **#53 — header backfill off the Glif critical path.** Parent backfill (for + blocks landing at `head+N`) was served by Glif `FetchBlock` with an 8s timeout + in both the sync agent and the gossip ingestor; a slow/rate-limited Glif drove + `backfillFail` up and desynced the node (surfacing downstream as e.g. + `cannot draw randomness from future epoch ... (head ...)`). Backfill is now + served from the combined bitswap+gateway content-addressed fetcher, Glif as + last resort; `HeadEpoch`/`TipsetCIDsByHeight` stay RPC-shaped (gossipsub + supplies live heads). Fetcher resolved lazily to see the bitswap-enabled + fetcher rebuilt after libp2p start. +- **#68 — `ChainGetTipSet(key)` served from the header store.** It previously + resolved only the synthetic current head and returned `ErrTipSetNotFound` for + any other key even with a populated header store, so Curio's `message/watch.go` + / `deps/apiinfo.go` looking up recent non-head tipset keys looped on + `tipset not in local store (only current head is cached in V1)`. New + `Store.GetTipSet(key)` reassembles any persisted tipset by key; `ChainGetTipSet` + falls through to it after the head fast-path. Distinct from #53 (this surfaces + even on a healthy synced node). + ## v1.8.0 (2026-06-26) Security hardening: trust model, bootstrap, and auth. Self-audit (#60) of the diff --git a/RELEASE-NOTES-v1.8.1.md b/RELEASE-NOTES-v1.8.1.md new file mode 100644 index 0000000..f7bd50e --- /dev/null +++ b/RELEASE-NOTES-v1.8.1.md @@ -0,0 +1,78 @@ +# Lantern v1.8.1 — sync resilience + chain-watcher fixes + +**Update if:** you run a Lantern node behind Curio (or any Lotus-API consumer +that watches the chain), especially in a zero/low-Glif configuration. This +release fixes two distinct bugs that, together, made a node fall behind live +head and then stall its consumer's chain watcher. Both were reported from the +field. No protocol or wire changes. + +## Highlights + +### Header backfill off the Glif critical path (#53) +Live blocks arrive over gossipsub fine, but when a block lands at `head+N` +(N>1) the **parent backfill** was served by Glif `FetchBlock` calls with an 8s +timeout, in *both* the polling sync agent and the gossip ingestor. When Glif was +slow or rate-limited, those calls timed out, `backfillFail` climbed, the head +could not advance contiguously, and the node desynced — surfacing downstream as, +for example: + +``` +cannot draw randomness from future epoch (head ) +``` + +(The PDP prover asks for randomness at an epoch the stalled node hasn't reached +yet, so the request is correctly refused — a *symptom* of the desync, not a +proving bug.) + +Backfill is now served from the combined **bitswap + gateway** fetcher (the same +content-addressed, CID-verified fetcher already running for state reads), with +Glif demoted to last resort. `HeadEpoch` / `TipsetCIDsByHeight` stay RPC-shaped +(gossipsub already supplies live heads), so during normal operation Glif is off +the head-advancement path entirely. The fetcher is resolved lazily so the +sync/gossip paths see the bitswap-enabled fetcher (rebuilt after libp2p comes +up), not a stale snapshot. + +### ChainGetTipSet served from the header store (#68) +`ChainGetTipSet(key)` only ever resolved the synthetic current head and returned +`ErrTipSetNotFound` for any other key — **even with a populated header store +wired**. Its siblings (`ChainGetTipSetByHeight`, `ChainGetBlock`, the randomness +path) all already fell through to the header store; only the by-key lookup was +left stubbed. + +Curio's `message/watch.go` and `deps/apiinfo.go` request specific recent +(non-head) tipset keys. Before this fix they were refused, producing a tight +error loop: + +``` +lantern: tipset not in local store (only current head is cached in V1) +failed to get tipset: chain: lantern: tipset not in local store ... +no new tipset in CurioChainSched.update +``` + +`ChainGetTipSet` now falls through to a new `Store.GetTipSet(key)` that +reassembles any persisted tipset directly from its block headers (returning +`ErrNotFound` only when a constituent block is genuinely missing). This is +distinct from #53: #53 is the node falling behind; #68 surfaces even on a +healthy, synced node. + +## On `--vm-bridge-rpc` as a default + +A tester asked whether `--vm-bridge-rpc https://api.node.glif.io/rpc/v1` should +be added by default to dodge these failures. It is a legitimate **fallback** and +fine to set today if you need resilience right now — but it is **not** the +default, by design. Defaulting it on would route every node silently back +through Glif and mask exactly the bugs above. The fixes here address the root +causes so the bridge stays optional. + +## Upgrade notes + +- No configuration changes required. Drop in the new binary and restart. +- No new external dependencies. + +## Verification + +CGO-free build, `go vet`, `gofmt`, and the hermetic test suite +(`-short`, `LANTERN_OFFLINE=1`) are green across the module. New unit tests +cover the #68 by-key lookup (head, non-head historical, empty-key, and +missing-block cases); the #53 adapter is covered by its existing suite. The live +mainnet node was not touched during development. diff --git a/chain/header/store/store.go b/chain/header/store/store.go index 2e35d89..d2a01c1 100644 --- a/chain/header/store/store.go +++ b/chain/header/store/store.go @@ -262,6 +262,19 @@ func (s *Store) canonicalAt(epoch abi.ChainEpoch) (*ltypes.TipSet, error) { return s.loadTipSet(tsk) } +// GetTipSet resolves a tipset directly by its key (the set of block +// CIDs). It reads each block header from the store and reassembles the +// tipset. Returns ErrNotFound if any constituent block is missing. Unlike +// GetTipSetByHeight this does NOT require the tipset to be on the +// canonical chain — it serves any tipset whose headers were persisted, +// which is what callers like ChainGetTipSet(key) need. +func (s *Store) GetTipSet(tsk ltypes.TipSetKey) (*ltypes.TipSet, error) { + if tsk.IsEmpty() { + return nil, ErrNotFound + } + return s.loadTipSet(tsk) +} + func (s *Store) loadTipSet(tsk ltypes.TipSetKey) (*ltypes.TipSet, error) { cids := tsk.Cids() blocks := make([]*ltypes.BlockHeader, 0, len(cids)) diff --git a/chain/header/store/store_test.go b/chain/header/store/store_test.go index 88a4e8d..3f80ccd 100644 --- a/chain/header/store/store_test.go +++ b/chain/header/store/store_test.go @@ -2,6 +2,7 @@ package store_test import ( "context" + "errors" "path/filepath" "testing" @@ -215,3 +216,52 @@ func TestOnHeadChange(t *testing.T) { t.Fatal("OnHeadChange callback did not fire") } } + +// TestGetTipSetByKey verifies GetTipSet(key) resolves a tipset directly +// from persisted headers, including a NON-head historical tipset (the +// Curio chain-watcher scenario from #68), and returns ErrNotFound when a +// constituent block is missing or the key is empty. +func TestGetTipSetByKey(t *testing.T) { + s, _ := newStore(t, false) + + // Build a short linear chain: genesis -> h1 -> h2 (h2 is head). + g := mkBlock(t, 0, nil, 1000, "g") + require.NoError(t, s.Put(g)) + gts, err := ltypes.NewTipSet([]*ltypes.BlockHeader{g}) + require.NoError(t, err) + require.NoError(t, s.SetHead(context.Background(), gts)) + + b1 := mkBlock(t, 1, []cid.Cid{g.Cid()}, 1000, "1") + require.NoError(t, s.Put(b1)) + ts1, err := ltypes.NewTipSet([]*ltypes.BlockHeader{b1}) + require.NoError(t, err) + require.NoError(t, s.SetHead(context.Background(), ts1)) + + b2 := mkBlock(t, 2, []cid.Cid{b1.Cid()}, 1000, "2") + require.NoError(t, s.Put(b2)) + ts2, err := ltypes.NewTipSet([]*ltypes.BlockHeader{b2}) + require.NoError(t, err) + require.NoError(t, s.SetHead(context.Background(), ts2)) + + // Resolve a NON-head historical tipset (epoch 1) directly by key. + got, err := s.GetTipSet(ts1.Key()) + require.NoError(t, err) + require.Equal(t, abi.ChainEpoch(1), got.Height()) + require.Equal(t, ts1.Key().String(), got.Key().String()) + + // Resolve the head tipset by key too. + gotHead, err := s.GetTipSet(ts2.Key()) + require.NoError(t, err) + require.Equal(t, abi.ChainEpoch(2), gotHead.Height()) + + // Empty key -> ErrNotFound. + _, err = s.GetTipSet(ltypes.EmptyTSK) + require.ErrorIs(t, err, store.ErrNotFound) + + // Key referencing a block we never persisted -> ErrNotFound. + missing := mkBlock(t, 3, []cid.Cid{b2.Cid()}, 1000, "missing") + missingTS, err := ltypes.NewTipSet([]*ltypes.BlockHeader{missing}) + require.NoError(t, err) + _, err = s.GetTipSet(missingTS.Key()) + require.True(t, errors.Is(err, store.ErrNotFound), "missing block should be ErrNotFound, got %v", err) +} diff --git a/cmd/lantern/bitswap_backed_source.go b/cmd/lantern/bitswap_backed_source.go new file mode 100644 index 0000000..b33bd60 --- /dev/null +++ b/cmd/lantern/bitswap_backed_source.go @@ -0,0 +1,105 @@ +// bitswapBackedSource: an RPCSource / BackfillSource that serves the +// content-addressed FetchBlock(cid) call from the combined (gateway + +// bitswap) fetcher first, falling back to Glif only on a miss. The two +// genuinely RPC-shaped methods (HeadEpoch, TipsetCIDsByHeight) stay on +// Glif as a last resort. +// +// Issue #53: header backfill was hardcoded to a Glif client. When Glif was +// slow/rate-limited the parent-backfill FetchBlock calls timed out, the +// header store couldn't advance contiguously, and the node fell behind live +// head. FetchBlock is content-addressed, so it can be served by the same +// bitswap/gateway fetcher already running for state reads — taking parent +// backfill off the Glif critical path. + +package main + +import ( + "context" + + "github.com/filecoin-project/go-state-types/abi" + "github.com/ipfs/go-cid" + + hstore "github.com/Reiers/lantern/chain/header/store" + "github.com/Reiers/lantern/chain/types" + "github.com/Reiers/lantern/net/blockingest" + "github.com/Reiers/lantern/net/combined" +) + +// blockGetter is the minimal surface the adapter needs from the combined +// fetcher: a content-addressed, CID-verified Get. *combined.Fetcher +// satisfies it. +type blockGetter interface { + Get(ctx context.Context, c cid.Cid) ([]byte, error) +} + +// rpcBlockSource is the RPC-shaped fallback the adapter delegates to. +// *glif.Client satisfies it. Kept as an interface so the fallback path is +// unit-testable without a live endpoint. +type rpcBlockSource interface { + HeadEpoch(ctx context.Context) (abi.ChainEpoch, error) + TipsetCIDsByHeight(ctx context.Context, h abi.ChainEpoch) ([]cid.Cid, error) + FetchBlock(ctx context.Context, k cid.Cid) (*types.BlockHeader, error) +} + +// bitswapBackedSource wraps a Glif client and a content-addressed block +// getter. HeadEpoch/TipsetCIDsByHeight delegate to Glif (inherently +// RPC-shaped). FetchBlock tries the content-addressed getter first and +// falls back to Glif on any error, so parent backfill no longer sits on +// the Glif critical path. +// +// The getter is resolved lazily via getFetcher() on each call. This matters +// because the combined fetcher is REBUILT after libp2p comes up (to add the +// bitswap source), and the sync/gossip wiring is constructed before that +// rebuild — a lazily-resolved getter always sees the current, bitswap- +// enabled fetcher rather than a stale gateway+glif-only snapshot. +type bitswapBackedSource struct { + glif rpcBlockSource + getFetcher func() blockGetter +} + +func newBitswapBackedSource(g rpcBlockSource, getFetcher func() blockGetter) *bitswapBackedSource { + return &bitswapBackedSource{glif: g, getFetcher: getFetcher} +} + +// HeadEpoch delegates to Glif (live head also arrives via gossipsub, so in +// steady state this is rarely the limiting call). +func (s *bitswapBackedSource) HeadEpoch(ctx context.Context) (abi.ChainEpoch, error) { + return s.glif.HeadEpoch(ctx) +} + +// TipsetCIDsByHeight delegates to Glif (RPC-shaped: maps height -> the +// canonical tipset's block CIDs). +func (s *bitswapBackedSource) TipsetCIDsByHeight(ctx context.Context, h abi.ChainEpoch) ([]cid.Cid, error) { + return s.glif.TipsetCIDsByHeight(ctx, h) +} + +// FetchBlock returns a single CID-verified BlockHeader. It is served from +// the content-addressed fetcher (gateway race + bitswap) first; on any miss +// or decode error it falls back to Glif. The fetcher's Get is already +// CID-verified, and DecodeBlock re-derives the header from the raw bytes, +// so a bad block can't slip through this path. +func (s *bitswapBackedSource) FetchBlock(ctx context.Context, k cid.Cid) (*types.BlockHeader, error) { + var f blockGetter + if s.getFetcher != nil { + f = s.getFetcher() + } + if f != nil { + if raw, err := f.Get(ctx, k); err == nil { + if bh, derr := types.DecodeBlock(raw); derr == nil { + return bh, nil + } + // Decode failed on otherwise-fetched bytes: fall through to + // Glif rather than fail the backfill outright. + } + } + return s.glif.FetchBlock(ctx, k) +} + +// Compile-time guards against the REAL consumer interfaces, so this file +// breaks the build if either consumer's surface drifts from what the +// adapter provides. +var ( + _ hstore.RPCSource = (*bitswapBackedSource)(nil) + _ blockingest.BackfillSource = (*bitswapBackedSource)(nil) + _ blockGetter = (*combined.Fetcher)(nil) +) diff --git a/cmd/lantern/bitswap_backed_source_test.go b/cmd/lantern/bitswap_backed_source_test.go new file mode 100644 index 0000000..1c7acdc --- /dev/null +++ b/cmd/lantern/bitswap_backed_source_test.go @@ -0,0 +1,181 @@ +package main + +import ( + "context" + "errors" + "testing" + + "github.com/filecoin-project/go-address" + "github.com/filecoin-project/go-state-types/abi" + "github.com/ipfs/go-cid" + + "github.com/Reiers/lantern/chain/types" +) + +// --- fakes --- + +type fakeGetter struct { + raw []byte + err error + hits int +} + +func (f *fakeGetter) Get(_ context.Context, _ cid.Cid) ([]byte, error) { + f.hits++ + return f.raw, f.err +} + +type fakeGlif struct { + bh *types.BlockHeader + err error + fetchHits int + headCalls int + tipsetCall int +} + +func (g *fakeGlif) HeadEpoch(context.Context) (abi.ChainEpoch, error) { + g.headCalls++ + return 100, nil +} +func (g *fakeGlif) TipsetCIDsByHeight(context.Context, abi.ChainEpoch) ([]cid.Cid, error) { + g.tipsetCall++ + return nil, nil +} +func (g *fakeGlif) FetchBlock(context.Context, cid.Cid) (*types.BlockHeader, error) { + g.fetchHits++ + return g.bh, g.err +} + +func mustAddr(t *testing.T) address.Address { + t.Helper() + a, err := address.NewIDAddress(1000) + if err != nil { + t.Fatalf("addr: %v", err) + } + return a +} + +func mustCID(t *testing.T) cid.Cid { + t.Helper() + c, err := abi.CidBuilder.Sum([]byte("sample-state")) + if err != nil { + t.Fatalf("cid: %v", err) + } + return c +} + +// minimal valid block we can serialize/decode round-trip. +func sampleBlock(t *testing.T) (*types.BlockHeader, []byte, cid.Cid) { + t.Helper() + bh := &types.BlockHeader{ + Miner: mustAddr(t), + ParentWeight: types.NewInt(100), + Height: 42, + ParentStateRoot: mustCID(t), + ParentMessageReceipts: mustCID(t), + Messages: mustCID(t), + Timestamp: 1234, + ParentBaseFee: types.NewInt(100), + ForkSignaling: 0, + } + raw, err := bh.Serialize() + if err != nil { + t.Fatalf("serialize: %v", err) + } + // round-trip sanity + if _, err := types.DecodeBlock(raw); err != nil { + t.Fatalf("decode round-trip: %v", err) + } + return bh, raw, bh.Cid() +} + +// --- tests --- + +// fetcher hit: served from bitswap/gateway, Glif never touched. +func TestBitswapBacked_FetcherHit(t *testing.T) { + _, raw, c := sampleBlock(t) + fg := &fakeGetter{raw: raw} + gl := &fakeGlif{} + s := newBitswapBackedSource(gl, func() blockGetter { return fg }) + + bh, err := s.FetchBlock(context.Background(), c) + if err != nil { + t.Fatalf("FetchBlock: %v", err) + } + if bh == nil || bh.Height != 42 { + t.Fatalf("wrong block: %+v", bh) + } + if fg.hits != 1 { + t.Errorf("fetcher hits = %d, want 1", fg.hits) + } + if gl.fetchHits != 0 { + t.Errorf("glif should NOT be called on a fetcher hit, got %d", gl.fetchHits) + } +} + +// fetcher miss (error) -> falls back to Glif. +func TestBitswapBacked_FetcherMissFallsBackToGlif(t *testing.T) { + bh, _, c := sampleBlock(t) + fg := &fakeGetter{err: errors.New("bitswap miss")} + gl := &fakeGlif{bh: bh} + s := newBitswapBackedSource(gl, func() blockGetter { return fg }) + + got, err := s.FetchBlock(context.Background(), c) + if err != nil { + t.Fatalf("FetchBlock: %v", err) + } + if got == nil || got.Height != 42 { + t.Fatalf("wrong block from glif fallback: %+v", got) + } + if gl.fetchHits != 1 { + t.Errorf("glif fetch hits = %d, want 1 (fallback)", gl.fetchHits) + } +} + +// fetcher returns undecodable bytes -> falls back to Glif rather than failing. +func TestBitswapBacked_DecodeFailFallsBackToGlif(t *testing.T) { + bh, _, c := sampleBlock(t) + fg := &fakeGetter{raw: []byte("not-cbor")} + gl := &fakeGlif{bh: bh} + s := newBitswapBackedSource(gl, func() blockGetter { return fg }) + + got, err := s.FetchBlock(context.Background(), c) + if err != nil { + t.Fatalf("FetchBlock: %v", err) + } + if got == nil || got.Height != 42 { + t.Fatalf("expected glif fallback block, got %+v", got) + } + if gl.fetchHits != 1 { + t.Errorf("glif fetch hits = %d, want 1", gl.fetchHits) + } +} + +// nil fetcher (getter returns nil) -> straight to Glif, no panic. +func TestBitswapBacked_NilFetcher(t *testing.T) { + bh, _, c := sampleBlock(t) + gl := &fakeGlif{bh: bh} + s := newBitswapBackedSource(gl, func() blockGetter { return nil }) + got, err := s.FetchBlock(context.Background(), c) + if err != nil || got == nil { + t.Fatalf("nil-fetcher path: got=%v err=%v", got, err) + } + if gl.fetchHits != 1 { + t.Errorf("glif fetch hits = %d, want 1", gl.fetchHits) + } +} + +// HeadEpoch / TipsetCIDsByHeight delegate to Glif. +func TestBitswapBacked_RPCMethodsDelegate(t *testing.T) { + gl := &fakeGlif{} + s := newBitswapBackedSource(gl, func() blockGetter { return nil }) + if _, err := s.HeadEpoch(context.Background()); err != nil { + t.Fatal(err) + } + if _, err := s.TipsetCIDsByHeight(context.Background(), 1); err != nil { + t.Fatal(err) + } + if gl.headCalls != 1 || gl.tipsetCall != 1 { + t.Errorf("delegation miss: head=%d tipset=%d", gl.headCalls, gl.tipsetCall) + } +} diff --git a/cmd/lantern/main.go b/cmd/lantern/main.go index de028a4..3ed4662 100644 --- a/cmd/lantern/main.go +++ b/cmd/lantern/main.go @@ -848,14 +848,17 @@ func cmdDaemon(args []string) error { dist.Start() chainAPI.HeadNotify = dist - // Sync source: a Glif client. The combined fetcher in gatewayClient - // is hamt-shaped (only Get), but Sync needs RPC-shaped - // HeadEpoch/TipsetCIDsByHeight/FetchBlock — that's exactly what - // glif.Client exposes. Network-aware: calibration daemon pulls - // from calibration Glif, mainnet daemon pulls from mainnet Glif. - // Without this, the header store fills with the wrong chain's - // headers (silent corruption). - src := glif.New(glifURLForNetwork(network), 8*time.Second) + // Sync source: #53 — a bitswap-backed adapter. HeadEpoch / + // TipsetCIDsByHeight stay on Glif (RPC-shaped), but the + // content-addressed FetchBlock (parent backfill) is served from the + // combined gateway+bitswap fetcher first, with Glif fallback. This + // takes parent backfill off the Glif critical path, so a slow / + // rate-limited Glif no longer stalls contiguous head advancement. + // Network-aware: calibration daemon pulls from calibration Glif, + // mainnet daemon pulls from mainnet Glif. Without the network split + // the header store fills with the wrong chain's headers (silent + // corruption). + src := newBitswapBackedSource(glif.New(glifURLForNetwork(network), 8*time.Second), func() blockGetter { return fetcher }) sync = hstore.NewSync(store, src, hstore.SyncOptions{ Interval: *syncInterval, MaxBacktrack: 60, @@ -973,12 +976,14 @@ func cmdDaemon(args []string) error { // blocks that gossipsub missed (connectivity blips, late join, // etc.) and for the first install on a cold start. // - // We pass the same glif client the polling Sync uses so the - // ingestor can do bounded inline backfill when a gossipsub - // arrival lands at head+N>1 (rather than skipping and waiting - // the full poll cycle). + // We pass the same bitswap-backed adapter the polling Sync uses so + // the ingestor's inline backfill (when a gossipsub arrival lands at + // head+N>1) serves parent blocks from the content-addressed fetcher + // first and only falls back to Glif (#53). Previously this was a raw + // Glif client, so a slow Glif drove backfillFail up and desynced the + // node despite live blocks arriving fine over gossipsub. if store != nil && p2pHost.PubSub != nil { - gossipSrc := glif.New(glifURLForNetwork(network), 8*time.Second) + gossipSrc := newBitswapBackedSource(glif.New(glifURLForNetwork(network), 8*time.Second), func() blockGetter { return fetcher }) if ing, _, gerr := startGossipBlocks(ctx, p2pHost.PubSub, store, gossipSrc, network.GossipTopicBlocks()); gerr != nil { fmt.Printf(" gossipsub-blocks: failed to start: %v (continuing without)\n", gerr) } else { diff --git a/rpc/handlers/chain_api.go b/rpc/handlers/chain_api.go index 17e236b..349c91e 100644 --- a/rpc/handlers/chain_api.go +++ b/rpc/handlers/chain_api.go @@ -365,18 +365,29 @@ func mustZeroIDAddr() address.Address { // ChainGetTipSet returns a tipset by key. Tier 1 (#18). // -// V1 implementation only knows the current trusted tipset. Requests for -// other keys return ErrTipSetNotFound. Phase 5 wires the header-store -// lookup. +// Resolution order: +// 1. The synthetic current trusted head (matches empty key or head key). +// 2. The persistent header store, by key (#68) — serves any historical +// tipset whose headers we've persisted. Curio's message/watch.go and +// apiinfo.go ask for specific recent (non-head) tipset keys; before +// this fell straight through to ErrTipSetNotFound, surfacing as +// "tipset not in local store (only current head is cached in V1)" and +// stalling Curio's chain watcher. func (c *ChainAPI) ChainGetTipSet(_ context.Context, key types.TipSetKey) (*types.TipSet, error) { if c.Trusted == nil { return nil, errors.New("trusted root not initialised") } - // Heuristic match: requested key matches the synthetic head's key. + // Fast path: requested key matches (or omits) the synthetic head's key. cur := synthesizeTipSet(c.Trusted) if key.IsEmpty() || tipsetKeyMatches(cur.Key(), key) { return cur, nil } + // Fall through to the header store for historical / non-head tipsets. + if c.HeaderStore != nil { + if ts, err := c.HeaderStore.GetTipSet(key); err == nil { + return ts, nil + } + } return nil, ErrTipSetNotFound }