Skip to content

fix(#119): persist pending mpool messages across daemon restart#121

Merged
Reiers merged 1 commit into
mainfrom
fix/119-mpool-persist
Jul 11, 2026
Merged

fix(#119): persist pending mpool messages across daemon restart#121
Reiers merged 1 commit into
mainfrom
fix/119-mpool-persist

Conversation

@Reiers

@Reiers Reiers commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Closes #119. Companion to #120 (#118 auto-refresh stale anchor). Together they close the peace-of-mind contract: a Lantern user can stop and restart the daemon at any time, after any duration, and both chain state and their pending sends survive without operator intervention.

Why

#47's rebroadcast loop tracked pending signed messages in memory only. A daemon restart threw away the pending set, so any message the user pushed pre-restart:

  • had a nonce consumed on the sender's on-chain account view,
  • was gossiped once but may not have been mined,
  • was no longer tracked (no retry, no confirm poll, no OnFailed fire).

If the message wasn't mined before the restart, it was silently dead. The next MpoolPush from the same account then hit an "expected nonce X, got X+1" gap and stalled indefinitely.

#51 covers "chain state can be rebuilt from the network." Signed pending messages cannot be rebuilt (the key signed them once, only the daemon that pushed them has the bytes), so they need durable persistence with the same discipline as user state.

What

`net/mpool/persist.go`: append-only JSONL journal + in-memory index.

  • `add` lines carry the raw signed bytes (base64) + metadata.
  • `retry` lines bump the rebroadcast counter + anchored epoch.
  • `tombstone` lines mark confirmed / max-retries / `Forget` removals.
  • On `openPersistStore`, replay all lines; later entries win. Automatic compaction when tombstones dominate (`compactMinLines=8`, `compactRatio=2`), and on corrupt-tail recovery.
  • `fsync` on every write so a crash after `Publish` keeps the message.

Pool integration (`net/mpool/mpool.go`):

  • `Config.PersistPath` opts in. When set, `New()` opens the journal + re-registers every live entry into the in-memory pending map.
  • `Publish()` fsyncs an `add` line after successful gossipsub push. If the fsync fails, we surface the error to the caller (safe to retry: identical bytes = same CID = same nonce).
  • `Forget()` writes a tombstone.
  • `Close()` flushes and releases the journal handle.
  • New `Stats.Restored` + `Stats.PersistPath` surface observability for the dashboard + tests.

Reconcile integration (`net/mpool/reconcile.go`):

  • Confirmed branch writes a tombstone (so restart doesn't re-register a message that already landed).
  • Max-retries branch writes a tombstone. `OnFailed` fires exactly once even across a crash-and-restart: on load the entry is gone, no restore.
  • Rebroadcast writes a `retry` line with the bumped counter + anchor, so a restart doesn't silently reset progress toward `MaxRetries`.

`pkg/daemon` wiring:

  • `Config.MpoolPersistDisabled` (zero value = enabled) toggles persistence.
  • `Config.MpoolPersistPath` overrides the default path.
  • `start.go` derives `//mpool/pending.jsonl` by default and passes it to `mpool.New`.
  • New `Daemon.MpoolStats()` accessor exposes counters (including `Restored` and `PersistPath`) for the dashboard + soak tests.

Guarantees

  • Rebroadcast is byte-identical (same nonce, same CID). The store keeps the ORIGINAL raw bytes and never re-signs or bumps fee. RBF stays explicitly out of scope, matching `mpool: pending-tx confirm + rebroadcast loop (published-but-unmined messages silently stall) #47`.
  • Never touches private keys, JWT, or tokens. The journal lives in `/mpool/`, chain-side of the secrets boundary.
  • `lantern reset --chain-state` does not wipe `pending.jsonl`: user's pending sends are user state, not rebuildable chain state. The reset allow-list only knows about `headerstore` + `bootstrap-anchor.json`, so this file is already safe from that path.
  • Backwards-compatible: `PersistPath=""` preserves pre-mpool: persist pending messages across daemon restart (#47 gap) #119 memory-only behaviour. All existing `mpool: pending-tx confirm + rebroadcast loop (published-but-unmined messages silently stall) #47` reconcile tests + `mpool` tests pass unchanged.
  • Nonce coordination is transparent: `MpoolGetNonce` already reads `Pending()` (`rpc/handlers/chain_api.go`). The restored set is loaded before `mpool.New()` returns, so the very first `MpoolGetNonce` after restart sees the pre-outage pending nonces.
  • Corrupt-tail recovery: a torn write on the last journal line is dropped on load + the file is compacted, so damage never propagates into subsequent runs.

Tests

10 unit tests in `net/mpool/persist_test.go`:

  • `TestPersist_AddAndReload`
  • `TestPersist_RemoveThenReload`
  • `TestPersist_UpdateOnRebroadcastReload`
  • `TestPersist_RemoveOnMissingCID` (no-op)
  • `TestPersist_UpdateOnRebroadcastMissingCID` (no-op)
  • `TestPersist_CompactAfterTombstoneChurn`
  • `TestPersist_CorruptTailIsRecovered`
  • `TestPersist_JournalLineShape`
  • `TestPersist_EmptyPathRejected`
  • `TestPersist_MkdirParent`

5 pool-level restart integration tests in `net/mpool/persist_pool_test.go`:

  • `TestPersist_RestartPreservesPending` — publish 3 → close → reopen → assert all 3 restored.
  • `TestPersist_ConfirmedDropsAcrossRestart` — publish 2 + confirm 1 → restart → only 1 survives.
  • `TestPersist_ForgetDropsAcrossRestart` — imperative `Forget` survives restart.
  • `TestPersist_RetryCountSurvivesRestart` — retries counter persists; `OnFailed` fires exactly once when the counter crosses `MaxRetries` across a restart.
  • `TestPersist_EmptyPathIsMemoryOnly` — backwards-compat: no path → old behaviour.

Full `go test -count=1 -short ./...` green across the tree.

Live smoke

Not run as part of this PR: an end-to-end smoke would need a funded wallet + a real `MpoolPushMessage` on calibration, which isn't safe to run mechanically. The unit + pool-level restart tests cover the load-bearing paths (Publish → journal → Close → reopen → restore → Reconcile) end-to-end at the mpool boundary. Integration verification on `pkg/daemon` is the config wiring pass-through covered by the existing `pkg/daemon` test suite.

Peace-of-mind contract now closed

The user contract Lantern implies is now: push a message and I'll keep gossiping it until it lands or a bounded retry budget runs out, restart-safe.

Depends on / relates to

@Reiers
fix(#119): persist pending mpool messages across daemon restart
11b68bd 
#47's rebroadcast loop tracked pending signed messages in memory only. A
daemon restart threw away the pending set, so any message the user
pushed pre-restart:
- had a nonce consumed on the sender's on-chain account view,
- was gossiped once but may not have been mined,
- was no longer tracked (no retry, no confirm poll, no OnFailed fire).

If the message wasn't mined before the restart, it was silently dead.
The next MpoolPush from the same account then hit an 'expected nonce X,
got X+1' gap and stalled indefinitely.

#51 covers 'chain state can be rebuilt from the network.' Signed pending
messages cannot be rebuilt (the key signed them once, only the daemon
that pushed them has the bytes), so they need durable persistence.

## What's new

net/mpool/persist.go: append-only JSONL journal + in-memory index.
- 'add' lines carry the raw signed bytes (base64) + metadata.
- 'retry' lines bump the rebroadcast counter + anchored epoch.
- 'tombstone' lines mark confirmed / max-retries / Forget removals.
- On Open, replay all lines; later entries win. Automatic compaction when
  tombstones dominate (compactMinLines=8, compactRatio=2), and on
  corrupt-tail recovery.
- Fsync on every write so a crash after Publish keeps the message.

Pool integration (net/mpool/mpool.go):
- Config.PersistPath opts in. When set, New() opens the journal + re-
  registers every live entry into the in-memory pending map.
- Publish() fsyncs an 'add' line after gossipsub push. If the fsync
  fails, we surface the error to the caller (safe to retry: identical
  bytes = same CID = same nonce).
- Forget() writes a tombstone.
- Close() flushes and releases the journal handle.
- New Stats.Restored + Stats.PersistPath surface observability.

Reconcile integration (net/mpool/reconcile.go):
- Confirmed branch writes a tombstone (so restart doesn't re-register a
  message that already landed).
- Max-retries branch writes a tombstone (OnFailed fires exactly once
  even across a crash-and-restart: on load the entry is gone, no restore).
- Rebroadcast writes a retry line with the bumped counter + anchor.

pkg/daemon wiring:
- Config.MpoolPersistDisabled (zero value = enabled) toggles persistence.
- Config.MpoolPersistPath overrides the default path.
- start.go derives <DataDir>/<Network>/mpool/pending.jsonl by default and
  passes it to mpool.New.
- New Daemon.MpoolStats() accessor exposes counters for dashboard + tests.

## Guarantees preserved

- Rebroadcast IS byte-identical (same nonce, same CID). We store the
  ORIGINAL raw bytes and never re-sign or bump fee. RBF stays out of
  scope.
- Never touches private keys, JWT, or tokens. The journal lives in
  <net>/mpool/, chain-side of the secrets boundary.
- 'lantern reset --chain-state' does NOT wipe pending.jsonl (user's
  pending sends are user state; the reset allow-list only knows about
  'headerstore' + 'bootstrap-anchor.json').
- Backwards-compatible: PersistPath='' preserves pre-#119 memory-only
  behaviour. All #47 tests still pass unchanged.
- Nonce coordination is transparent: MpoolGetNonce already reads
  Pending() (rpc/handlers/chain_api.go), and the restored set is loaded
  before New() returns, so the very first MpoolGetNonce after restart
  sees the pre-outage pending nonces.

## Tests

- 10 unit tests in net/mpool/persist_test.go: add/reload,
  remove-then-reload, update-on-rebroadcast reload, no-op on missing
  cid, tombstone-churn compaction, corrupt-tail recovery, journal-line
  shape, empty-path rejection, parent-dir mkdir.
- 5 pool-level restart integration tests in net/mpool/persist_pool_test.go:
  restart-preserves-pending, confirmed-drops-across-restart,
  forget-drops-across-restart, retry-count-survives-restart,
  empty-path-is-memory-only.
- All existing #47 reconcile tests + mpool tests remain green.
- Full 'go test -count=1 -short ./...' green across the tree.

## Second half of the peace-of-mind contract

Companion to #118 (auto-refresh stale anchor across restart). Together:
a Lantern user can stop and restart the daemon at any time, after any
duration, and the node catches up on its own AND their pending sends
still land, without them needing to know a single reset command exists.

Fixes #119.
@Reiers Reiers merged commit 735a06a into main Jul 11, 2026
1 check passed
@Reiers Reiers deleted the fix/119-mpool-persist branch July 11, 2026 13:06
This was referenced Jul 11, 2026
Closed
Merged
Reiers pushed a commit that referenced this pull request Jul 12, 2026
@Reiers
fix(#123): close all ten devnet RPC parity gaps against local Curio d…
2d94fb1 
…evnet

Every change is gated on network == "devnet"; mainnet + calibration
paths are byte-identical to pre-change behaviour.

- eth_chainId + net_version return the devnet's actual EIP-155 chainId
  instead of the mainnet fallback 0x13a / "314". devnet-init reads
  eth_chainId from the running devnet lotus and persists it into
  devnet-config.json as ethChainID. This is the P0 finding: an EVM
  client signing with the wrong chainId gets "invalid signature" back
  from lotus and takes hours to diagnose.

- Filecoin.Version.BlockDelay returns the devnet's actual cadence
  instead of hardcoded 30. devnet-init reads BlockDelay from
  Filecoin.Version and persists as blockDelaySecs.

- Filecoin.StateNetworkName returns the devnet's localnet-uuid instead
  of the literal string "devnet". Handler now reads from the persisted
  devnet-config instead of a literal. Fixes gossipsub topic identity
  (/fil/blocks/<name>) and DHT protocol prefix alignment.

- Filecoin.ChainGetGenesis on devnet returns the persisted genesis CID
  instead of "not implemented". Compile-time genesis constants exist
  only for mainnet + calibration; devnet's genesis is per-boot.

- Filecoin.MpoolGetConfig returns a stable snapshot instead of HTTP 500.
  Values match lotus defaults so consumers (curio at boot) don't have
  to special-case Lantern. PriorityAddrs nil (marshals to JSON null,
  matching lotus's wire shape byte-for-byte).

- Filecoin.MpoolPushMessage now works on single-node devnet. The
  gossipsub mesh can't form on a single-node docker devnet, so the
  standard mpool.Pool had nothing to publish onto. Daemon now wires a
  Pool whose Config.Sink POSTs directly to the devnet lotus via
  Filecoin.MpoolPush. All other Pool semantics (persist journal,
  pending set, nonce derivation, #47 reconcile/rebroadcast, #121
  restart-persist) work identically; only the wire transport changes.
  Trust posture unchanged from #122: devnet is single-source by design
  (operator owns both the lotus and the Lantern client).

- Devnet mode auto-wires the devnet lotus as VMBridge when the operator
  did not pass --vm-bridge-rpc. Fixes eth_getCode / eth_call /
  eth_getStorageAt timeouts on cold state (single-node devnet has no
  libp2p, so bitswap can't fetch state blocks; daemon fell through to
  errBridgeUnconfigured).

- mpool.Config.Sink hook lets Pool run with a nil pubsub instance.
  When Sink is non-nil and ps is nil, New() skips the gossipsub
  join+subscribe entirely and Publish/Rebroadcast route through the
  sink. Only used by devnet mode; mainnet/calibration paths continue
  to require a pubsub instance.

- net/glif.Client.EthChainID, .BlockDelaySecs, .MpoolPush — three new
  methods on the shared JSON-RPC client. Used by devnet-init (boot-time
  discovery) and by the daemon (devnet mpool sink).

- build.DevnetConfig gained EthChainID + BlockDelaySecs fields. Older
  configs written before this change have zero values; handlers hint
  `lantern devnet-init --force` to re-capture from the running lotus.

- 5 new unit tests in net/mpool/sink_test.go cover the nil-pubsub +
  Sink path: nil-pubsub requires Sink; Publish invokes Sink instead of
  gossipsub byte-identically; persist journal entry matches published
  raw; Close is a no-op on nil topic/subscription; sink error surfaces
  from Publish without recording pending.

- Existing cmd/lantern devnet-init tests updated: fake lotus stub now
  answers eth_chainId + Filecoin.Version so the boot-time discovery
  succeeds; new asserts check EthChainID = 31415926 and BlockDelaySecs
  = 4 land in the persisted config.

- Full suite green: go test ./... reports 51 packages ok, no failures.

- ChainHead head-poll lag on devnet (#123 findings 8+9) and eth_getCode
  proxy-address slowness (finding 7). Both are latency issues, not
  correctness. Folding into a separate --devnet-head-poll-interval knob
  PR after this batch lands.
Reiers added a commit that referenced this pull request Jul 12, 2026
@Reiers
fix(#123): close all ten devnet RPC parity gaps against local Curio d…
035a291 
…evnet (#125)

* fix(#123): close all ten devnet RPC parity gaps against local Curio devnet

Every change is gated on network == "devnet"; mainnet + calibration
paths are byte-identical to pre-change behaviour.

- eth_chainId + net_version return the devnet's actual EIP-155 chainId
  instead of the mainnet fallback 0x13a / "314". devnet-init reads
  eth_chainId from the running devnet lotus and persists it into
  devnet-config.json as ethChainID. This is the P0 finding: an EVM
  client signing with the wrong chainId gets "invalid signature" back
  from lotus and takes hours to diagnose.

- Filecoin.Version.BlockDelay returns the devnet's actual cadence
  instead of hardcoded 30. devnet-init reads BlockDelay from
  Filecoin.Version and persists as blockDelaySecs.

- Filecoin.StateNetworkName returns the devnet's localnet-uuid instead
  of the literal string "devnet". Handler now reads from the persisted
  devnet-config instead of a literal. Fixes gossipsub topic identity
  (/fil/blocks/<name>) and DHT protocol prefix alignment.

- Filecoin.ChainGetGenesis on devnet returns the persisted genesis CID
  instead of "not implemented". Compile-time genesis constants exist
  only for mainnet + calibration; devnet's genesis is per-boot.

- Filecoin.MpoolGetConfig returns a stable snapshot instead of HTTP 500.
  Values match lotus defaults so consumers (curio at boot) don't have
  to special-case Lantern. PriorityAddrs nil (marshals to JSON null,
  matching lotus's wire shape byte-for-byte).

- Filecoin.MpoolPushMessage now works on single-node devnet. The
  gossipsub mesh can't form on a single-node docker devnet, so the
  standard mpool.Pool had nothing to publish onto. Daemon now wires a
  Pool whose Config.Sink POSTs directly to the devnet lotus via
  Filecoin.MpoolPush. All other Pool semantics (persist journal,
  pending set, nonce derivation, #47 reconcile/rebroadcast, #121
  restart-persist) work identically; only the wire transport changes.
  Trust posture unchanged from #122: devnet is single-source by design
  (operator owns both the lotus and the Lantern client).

- Devnet mode auto-wires the devnet lotus as VMBridge when the operator
  did not pass --vm-bridge-rpc. Fixes eth_getCode / eth_call /
  eth_getStorageAt timeouts on cold state (single-node devnet has no
  libp2p, so bitswap can't fetch state blocks; daemon fell through to
  errBridgeUnconfigured).

- mpool.Config.Sink hook lets Pool run with a nil pubsub instance.
  When Sink is non-nil and ps is nil, New() skips the gossipsub
  join+subscribe entirely and Publish/Rebroadcast route through the
  sink. Only used by devnet mode; mainnet/calibration paths continue
  to require a pubsub instance.

- net/glif.Client.EthChainID, .BlockDelaySecs, .MpoolPush — three new
  methods on the shared JSON-RPC client. Used by devnet-init (boot-time
  discovery) and by the daemon (devnet mpool sink).

- build.DevnetConfig gained EthChainID + BlockDelaySecs fields. Older
  configs written before this change have zero values; handlers hint
  `lantern devnet-init --force` to re-capture from the running lotus.

- 5 new unit tests in net/mpool/sink_test.go cover the nil-pubsub +
  Sink path: nil-pubsub requires Sink; Publish invokes Sink instead of
  gossipsub byte-identically; persist journal entry matches published
  raw; Close is a no-op on nil topic/subscription; sink error surfaces
  from Publish without recording pending.

- Existing cmd/lantern devnet-init tests updated: fake lotus stub now
  answers eth_chainId + Filecoin.Version so the boot-time discovery
  succeeds; new asserts check EthChainID = 31415926 and BlockDelaySecs
  = 4 land in the persisted config.

- Full suite green: go test ./... reports 51 packages ok, no failures.

- ChainHead head-poll lag on devnet (#123 findings 8+9) and eth_getCode
  proxy-address slowness (finding 7). Both are latency issues, not
  correctness. Folding into a separate --devnet-head-poll-interval knob
  PR after this batch lands.

* fix(#123): auto-wire devnet VMBridge in standalone CLI + skip gossipsub mpool wiring on devnet

Two follow-up fixes surfaced by a live smoke against a running docker
devnet (curio-fork make docker/devnet, lotus v1.35.1, BlockDelay=4):

1. cmd/lantern/main.go standalone daemon didn't share the pkg/daemon
   VMBridge auto-wire I added in the previous commit. When `lantern
   daemon --network devnet` runs without an explicit --vm-bridge-rpc,
   any eth_call / eth_getCode / eth_getStorageAt returned
   "FEVM method requires --vm-bridge-rpc" instead of transparently
   using the devnet's own lotus. Now the CLI applies the same
   `--network devnet && !--vm-bridge-rpc → devnetCfg.LotusRPC` fallback.
   Verified live: after the fix, eth_getCode against the deployed
   PDPVerifier reaches the bridge (result is served instead of erroring).

2. pkg/daemon/start.go's startGossipHead was joining the gossipsub
   mempool topic on devnet EVEN THOUGH there's no gossipsub mesh on a
   single-node devnet. Once that mpool was in place, my devnet
   lotus-RPC sink block (which triggers only when chainAPI.Mpool == nil)
   never fired, so MpoolPushMessage silently dropped signed messages
   into the void. Fix: startGossipHead now short-circuits before the
   gossipsub mpool wiring on devnet, logs the reason, and leaves
   chainAPI.Mpool nil so the devnet sink block in startInternal
   activates. Bitswap wiring after the mpool block is preserved via
   a goto over a scoped variable-declaration block (Go-legal, verified
   with go build + go vet).

Live-verified end-to-end on the docker devnet:
  Filecoin.StateNetworkName  → "localnet-1069c685-..." (was "devnet")
  eth_chainId                → 0x1df5e76 (31415926) (was 0x13a)
  net_version                → "31415926" (was "314")
  Filecoin.Version.BlockDelay→ 4 (was 30)
  Filecoin.ChainGetGenesis   → real CID (was not-implemented)
  Filecoin.MpoolGetConfig    → lotus-shape defaults (was HTTP 500)
  vm-bridge auto-wired to devnet lotus (bridge log line + eth_getCode reaches bridge)

eth_getCode is still slow on cold state (local hamt walk tries bitswap
first, and single-node devnet has no bitswap peers, so the walk waits
the full 15s timeout before falling through to the bridge). That's
#123 finding #7 in a narrower form. Fix candidate: short-circuit the
local-first order to bridge-first on devnet. Out of scope for this
batch; filing a follow-up.

Tests: full suite (`go test ./...`) 51/51 packages ok.

---------

Co-authored-by: Nicklas Reiersen <nicklas@reiers.io>
This was referenced Jul 12, 2026
Merged
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

1 participant

@Reiers