Message-block availability: StateSearchMsg fetches message blocks via bridge, stalls bridge-off

[Reiers]

Found via bridge-off write-confirm testing on cc-smoke (after #49)

With #49 fixed, StateSearchMsg now parses block message AMTs and ParentMessageReceipts correctly (no more 'wrong number of fields'). But resolving a receipt / tx-by-hash bridge-off now fails one layer deeper:

eth_getTransactionByHash: StateSearchMsg error, bridging
  err: block <blk> msgmeta: secp amt: https://api.calibration.node.glif.io/.../block/<msgblk>: context deadline exceeded

msgsearch.fetchMsgMeta → enumerateAMTCIDs walks the message AMT and, when a message block (MsgMeta / AMT node / message body) isn't already in the embedded blockstore, asks the BlockGetter. In embedded curio-core that getter's source set includes the Glif HTTP fetcher; with --vm-bridge-rpc-disable those fetches time out, so StateSearchMsg errors and the receipt/by-hash path declines.

This is #44-class state availability, not an AMT or engine bug

• Embedded state-block availability for local eth_call (last blocker for --vm-bridge-rpc-disable) #44 prefetches FEVM contract storage-trie nodes on head advance (PDPVerifier / FWSS / registry / USDFC). It does not prefetch message/receipt blocks for the tipsets a writing SP needs to confirm its own txs.
• Reads (eth_call) are ~100% local; the gap is the message/receipt block availability for the write-confirm loop.

Also observed: head-sync HTTP fallback points at Glif

On this bridge-off boot, chain/header/store/sync.go logged sync poll: Post https://…glif.io…: context deadline exceeded and gossipsub head-tracking sometimes took >30s to activate (flaky calibration libp2p peering). When gossipsub is healthy the head advances fine; when it isn't, the HTTP sync fallback is a hidden Glif dependency. Worth making the bridge-off mode refuse/!configure the HTTP sync fallback so the Glif dependency is explicit.

Fix direction

1. Prefetch message + receipt blocks for recently-sent txs (or for the last N tipsets) into the embedded Bitswap cache, same mechanism as Embedded state-block availability for local eth_call (last blocker for --vm-bridge-rpc-disable) #44's storage-trie prefetch. A writing SP only needs its own in-flight txs' tipsets.
2. Make msgsearch's BlockGetter use a bitswap-with-retry source bridge-off (mirror evmexec_fetch.go's retryingBlockGetter) so a momentarily-uncached block resolves on a 3-5s retry instead of erroring out.
3. Ensure header-store HTTP sync fallback is disabled (not silently Glif) under --vm-bridge-rpc-disable.

Status

• Send path: local + on-chain proven (Local FEVM write path — the zero-Glif keystone (estimateGas/nonce/sendRawTx/receipt) #45).
• Receipt + tx-by-hash decode: local + Glif-parity proven (StateSearchMsg fails decoding block message AMT ('cbor input had wrong number of fields') — blocks local eth_getTransactionReceipt #49 + this session's eth_getTransactionByHash).
• This issue is the last availability gap before a sustained bridge-off soak. Parent: #45 Stage 7: real SP tx declines local send path bridge-off (parser mismatch + uncovered maxPriorityFeePerGas dep) #46, Head catch-up wedges behind tip: gossip-fresh skip (#71) is not lag-aware #83 (curio-core).

[Reiers]

Major progress (2026-06-15): Glif fully removed from the block path; bitswap is now the source

Landed across v1.7.14 → v1.7.16 and live-tested bridge-off on cc-smoke:

What's fixed

1. Bitswap mounted as embedded block source (v1.7.14). combined.Fetcher.AddSource (thread-safe, race-tested) lets startGossipHead inject the libp2p Bitswap client as a high-priority race source once the gossip host is up. Replaces the Glif-only block path.
2. Filecoin bitswap protocol prefix (v1.7.15) — the keystone. The v1.7.14 bitswap client connected but couldn't exchange blocks: protocols not supported: [/ipfs/bitswap/...]. Filecoin (Lotus/Forest) serves bitswap under /chain/ipfs/bitswap/... (the bitswap analogue of the /fil/kad/<net> DHT prefix). Added bitswap.Config.ProtocolPrefix + build.Network.BitswapProtocolPrefix() => "/chain". After this: protocols not supported → 0.
3. msgsearch retry-on-miss + budget tuning (v1.7.16). Wrapped the StateSearchMsg getter in the Embedded state-block availability for local eth_call (last blocker for --vm-bridge-rpc-disable) #44 retrying getter, widened to 2 retries × ~6s = 18s so a cold block gets a full bitswap round (the 6s/4 budget was slicing attempts to ~1.2s and canceling bitswap mid-broadcast).

Live evidence bridge-off on cc-smoke (calibration)

• gossipsub head-tracking active + bitswap block source mounted + peers=14 + bitswap newly available: [12D3KooW...].
• State reads fully local: eth_getBalance and eth_getTransactionCount (nonce=0x3) both served LOCAL bridge-off, fed by bitswap state-tree blocks.
• Send local: eth_sendRawTransaction published to the local mempool.
• Glif block fetches: 0. The whole block path now goes through bitswap; api.calibration.node.glif.io/.../block/ fetch count is zero during the bridge-off window (was every block before).

What remains (the honest residual)

• One message block per write-confirm cycle still occasionally fails with net/bitswap: context canceled (not deadline exceeded) — i.e. a parent context (RPC handler or polling client) cancels while bitswap is still broadcasting a WANT for a freshly-produced message block that connected peers haven't served yet.
• Root cause is calibration sparsity: the just-sent tx's message block isn't reliably served by the random gossip peer set within the poll window. Likely fixes:
  1. Targeted peer preference: connect bitswap's PreferredPeers to the SP's own block-producing/storage peers (the nodes that definitely hold the freshly-produced block), instead of relying on random swarm broadcast.
  2. Message-block prefetch on send: when eth_sendRawTransaction publishes, proactively warm the message block + (on inclusion) the receipt block into the bitswap blockstore, so the watcher's poll hits a warm cache.
  3. Confirm the cancel isn't client-side (the go-ethereum poll loop's per-call timeout) — re-test with a server-side-only harness.

Status

The Glif dependency in the block path is eliminated and the architecture is proven correct. The remaining work is bitswap reliability for freshly-produced message blocks on a sparse test network, tracked here. cc-smoke restored to safe bridge-ON between tests.