[Soak] Calibration dress rehearsal: snapshot-free boot, sustained sync, block production, PDP — zero Lotus (v1.9.0 final gate)

[Reiers]

Goal

One supervised live session that proves the whole three-tier story end-to-end on calibration, with no Lotus anywhere in the stack: Lantern boots without a snapshot, stays in sync, serves a real Curio binary, produces a block, and runs the PDP flow. This is the release gate for v1.9.0 final, and the acceptance evidence for #90, #94 and #76 in one run.

Every checkbox below carries an explicit Test / Pass / Evidence so a tick is never a vibe — it's a measurement posted as a comment.

Ground rules

• Our node is part of what keeps calibration stable. The soak node runs alongside existing infra, never replacing a production component until its phase has passed. No experiment may disrupt the network: block production uses our own miner identity, PDP uses our own datasets.
• Rollback is always one step: stop the soak binary, existing infra unaffected.
• Every phase gets its numbers written into the issue as a comment (counters, epochs, tx hashes) - this document is the evidence trail.
• Node under test is always the soak Lantern (127.0.0.1:1235), never the Lotus-backed production stack running alongside.

Tier matrix — Light / PDP / Full

The three tiers are one binary + two dials (retention, verification). They differ, so the soak exercises each where it diverges rather than running three identical soaks.

Tier	Retention	Validation	Head corroboration	Block submit	Soak coverage
Light	in-memory (~1 GB)	none	off (divergence monitor + fork choice)	no	Phase 1 boot + read-serve + zero-Glif check
PDP	persistent 2-5 GB cache	none	off	opt-in	Phase 1 boot + Phases 3-4 (Curio + PDP e2e)
Full	persistent cache + full chain surface	observe-mode pure-Go consensus (#90)	2 distinct peers (auto)	opt-in	Phases 1-2 + Phase 5 (superset)

Full is the end-to-end superset. Light and PDP get targeted differential checks (boot, footprint, cache present/absent, validation on/off, corroboration default). Each tier's boot is its own checkbox in Phase 1.

Setup

• Build soak binary from main with feat(#80 part 2, #97): head-source corroboration + per-topic gossipsub scoring #101 (head corroboration + topic scoring), chore(#102): build with Go 1.26.x #103 (Go 1.26.4), feat(#96): FRC-0089 EC finality calculator (observed-data finality) #105 (EC finality), feat(#93): wallet export/import (Lotus hex-KeyInfo) + import-lotus #106 (wallet import), feat(#98): VM-bridge cross-check auditor (observe-only divergence alarm) #107 (VM cross-check) merged; record commit hash + lantern version
  • Test: git describe on the running binary; confirm all five PRs are in the commit.
  • Pass: version string embeds the main-HEAD commit and all five PRs are merged into it.
  • Evidence: ✅ commit 3021887, lantern v1.9.0-rc1-11-g3021887 (comment #4864587138).
• Optional: soak build with GOEXPERIMENT=goroutineleakprofile for the leak check in Phase 5 (Move toolchain to Go 1.26.x (Green Tea GC default, heap ASLR, goroutine-leak profiler) #102 follow-up) — deferred to Phase 5 (optional; avoid resetting the sustained-sync clock)
  • Test: build with the experiment flag; curl /debug/pprof/goroutineleak.
  • Pass: endpoint responds; profile parses.
  • Evidence: deferred; only built if we want the Phase-5 leak data point.
• Full tier selected (nodeprofile Full), FullValidation on, observe mode (not fatal)
  • Test: lantern node-type full; on next boot the log prints full block validation ON (observe mode, #90).
  • Pass: node-profile.json has tier=full; FullValidation() true; validator wired non-fatal.
  • Evidence: ✅ { "tier": "full", "allow_block_submit": true } staged for Phase-1 boot (comment #4864626157). Boot-log line confirmed at Phase 1.
• Bridge-off config: NoFallbackRPC / no Glif; gateway configured only if a phase explicitly tests fallback
  • Test: launch with the explicit no-RPC-fallback head source; grep launch flags + startup log for the head-source mode; confirm no Glif endpoint configured.
  • Pass: head source = live gossip/quorum only, no silent RPC fallback; gateway present only as last-resort cold-block fetch (not head).
  • Evidence: exact launch command + startup head-source log line.
• VM bridge target for block production is Forest (never Lotus - the point of this soak)
  • Test: stand up a Forest node on calibration; point --vm-bridge-rpc at Forest; confirm it is Forest (version string), not the prod Lotus.
  • Pass: bridge RPC answers and self-identifies as Forest; Phase 5 uses only this endpoint for the post-execution state root.
  • Evidence: Forest version + RPC URL; note that reads/validation do NOT use it (proven by the Phase-5 outage drill).
• --vm-crosscheck enabled (FVM cross-check mode: use the VM bridge as an auditor (state-root divergence alarm) #98, landed): the auditor rides the same Forest bridge
  • Test: launch with --vm-crosscheck --vm-bridge-rpc <forest>; watch the cross-check card/counter.
  • Pass: auditor runs each interval, compares local vs Forest state root, zero DIVERGE under normal operation.
  • Evidence: cross-check counter + zero-DIVERGE over the window.
• Dashboard reachable; note the dev-page URL for stat screenshots
  • Test: curl the --metrics dashboard/dev page over the tunnel.
  • Pass: dashboard serves; dev page shows gossip/corroboration/EC-finality/cross-check cards.
  • Evidence: dev-page URL + a screenshot reference.

Phase 1 — Snapshot-free boot (per tier)

• Light cold start on calibration with no snapshot import: boot from the multi-source F3-finalized anchor quorum
  • Test: wipe chain state (keys untouched), boot tier=light, time from process start to live head.
  • Pass: reaches tip with no snapshot; in-memory footprint stays ~1 GB.
  • Evidence: time-to-tip + RSS.
• PDP cold start, snapshot-free, persistent cache provisioned
  • Test: boot tier=pdp, confirm persistent cache dir created and populated.
  • Pass: reaches tip; disk footprint in the 2-5 GB cache budget.
  • Evidence: time-to-tip + cache-dir size.
• Full cold start, snapshot-free (this is the soak node for Phases 2 & 5)
  • Test: boot tier=full; confirm full block validation ON (observe mode) log line.
  • Pass: reaches tip; validator active; footprint single-digit GB.
  • Evidence: time-to-tip + footprint after boot and after 1h.
• If chain/exchange BlockSync port: snapshot-free full-node boot #91 (BlockSync tail-fill) has landed: verify headers-only deep-tail fetch + record header-store depth; if not, note boot anchor depth as the chain/exchange BlockSync port: snapshot-free full-node boot #91 baseline
  • Test: inspect header-store depth vs anchor epoch.
  • Pass: N/A until chain/exchange BlockSync port: snapshot-free full-node boot #91; record baseline depth now.
  • Evidence: anchor epoch + header-store depth.

Phase 2 — Sustained sync (24h minimum, Full tier)

• Head lag ≤ 1 epoch sustained; no wedge across the full window
  • Test: sample gossip.last_install_epoch vs wall-clock epoch over 24h.
  • Pass: lag ≤ 1 epoch at every sample; no stall.
  • Evidence: min/median/max lag; sample count.
• fullvalidate observe counters: 100% agreement, zero validation failures on canonical blocks (Full-node tier: pure-Go block-validation pipeline + tunable resident state #90 evidence)
  • Test: read the observe-mode validator counters over the window.
  • Pass: 100% agreement on canonical blocks; zero unexplained failures (non-canonical/orphan disagreements noted, not counted against).
  • Evidence: blocks validated / agreed / failed counts.
• Corroboration stats sane (Peer-table eclipse hardening for the head path: pin anchor peers + head-source diversity #80 part 2 / PR feat(#80 part 2, #97): head-source corroboration + per-topic gossipsub scoring #101): held_uncorrob low and non-wedging, heads adopt within seconds
  • Test: read corroboration held/adopted counters.
  • Pass: heads adopt within seconds; held-uncorroborated never wedges the head.
  • Evidence: held vs adopted counts; max hold time.
• Divergence gate: zero false-positive holds during normal operation (Running-head fork choice: heaviest-ParentWeight + periodic re-quorum (not boot-only) #79/feat(#85): header-propagation integrity gate + running-head divergence monitor #86)
  • Test: watch divergence-monitor holds.
  • Pass: zero false-positive holds on healthy chain.
  • Evidence: hold count + any reasons.
• Zero Glif / zero gateway hits in bridge-off mode ([Tracking] Lantern bridge-off parity for stock Curio (maxboom / PR #1311) #76 evidence): fetch counters show cache/bitswap only
  • Test: read fetch-source counters (cache / bitswap / gateway / glif) over the window.
  • Pass: gateway = 0, glif = 0; all fetches served by cache/bitswap.
  • Evidence: per-source fetch counts.
• Peer health: mesh ≥ 6 peers, trusted floor peers still connected at end of window (Peer-table eclipse hardening for the head path: pin anchor peers + head-source diversity #80 part 1)
  • Test: sample mesh size + trusted-floor peer connectivity.
  • Pass: mesh ≥ 6 throughout; trusted floor peers connected at end.
  • Evidence: mesh min/max; floor-peer status.
• EC finality card (Lift FRC-0089 EC finality calculator (lotus chain/ecfinality): observed-data finality without F3 #96): computed 2^-30 depth settles ~30 epochs on healthy calibration; window_epochs grows with uptime; computed ec-finalized epoch tracks F3 plausibly
  • Test: read the EC-finality card over the window.
  • Pass: depth ~30 epochs healthy; window grows; ec-finalized tracks F3.
  • Evidence: computed depth + window_epochs + ec-finalized vs F3-finalized.

Phase 3 — Curio binds (real binary, no Lotus; PDP/Full tier)

• Stock Curio binary binds the soak Lantern as its chain node; startup clean, no missing-RPC errors
  • Test: point a separate soak Curio (not prod) at FULLNODE_API_INFO=<soak Lantern :1235>; start it.
  • Pass: Curio starts; zero missing-RPC / method-not-found errors in the chain path.
  • Evidence: Curio startup log excerpt; RPC-error count = 0.
• Curio chain-facing pages/ops work against Lantern (head, messages, wallet balances)
  • Test: drive Curio's chain views/ops.
  • Pass: head, messages, balances all render correct values matching Lantern RPC.
  • Evidence: side-by-side value spot-checks.
• Message path: send → confirm → receipt through Lantern's mpool (pending → confirmed, rebroadcast if needed)
  • Test: send a small self-message via Curio→Lantern; track through mpool.
  • Pass: pending → confirmed → receipt, on our own identity.
  • Evidence: message CID + confirm epoch + receipt.
• 24h co-run: no RPC-parity regressions logged ([Tracking] Lantern bridge-off parity for stock Curio (maxboom / PR #1311) #76 evidence)
  • Test: run the Curio↔Lantern co-run 24h; scan for parity errors.
  • Pass: zero RPC-parity regressions.
  • Evidence: error scan summary.

Phase 4 — PDP end-to-end (PDP tier, our own datasets)

• add-piece through Lantern-served chain state
  • Test: add a piece to our own dataset via Curio→Lantern.
  • Pass: piece added, dataset state advances.
  • Evidence: dataset id + add-piece tx.
• Prove window: challenge randomness drawn bridge-off (Bridge-off PDP prove fails 'cannot draw randomness from future epoch' — no bridge-off fallback + head lags tip #82 path), proof submitted, on-chain success
  • Test: at the prove window, confirm randomness is drawn from live head bridge-off; submit proof.
  • Pass: proof tx status 0x1 on calibration.
  • Evidence: prove tx hash + epoch; randomness-source log line.
• ≥ 3 consecutive prove windows land without manual intervention (PDP/mid tier (F1): persistent cache + pinning + block-submit + install-time tier [needs live soak] #94 evidence)
  • Test: watch three consecutive windows.
  • Pass: 3/3 land, zero manual repair.
  • Evidence: three tx hashes + epochs.
• Settlement/payment rails exercised where applicable; record txs
  • Test: run any applicable settlement path.
  • Pass: settlement tx lands.
  • Evidence: tx hashes.

Phase 5a — Block production (zero Lotus; Forest as VM bridge; Full tier) — v1.9.0 gate

Honest scope: this binary has no pure-Go FVM (#89). Creating a block requires the post-execution ParentStateRoot, which is an FVM execution. So block production uses an external VM bridge = Forest for that state root only. Everything else (base info, ticket/election, block assembly, gossip submit, all reads/validation) is Lantern. This proves we are not Lotus-locked, not that we are zero-external. Uses our own miner identity.

• MinerGetBaseInfo / MinerCreateBlock / SyncSubmitBlock served by Lantern, VM bridge = Forest for the post-execution state root
  • Test: exercise the three miner RPCs via Curio→Lantern with --vm-bridge-rpc <forest>.
  • Pass: all three return correct data; only the state-root call touches Forest.
  • Evidence: RPC traces; confirm Forest is the sole external touch.
• Winning epoch produces a block: created, submitted, lands on calibration and propagates (seen by an independent node)
  • Test: at a winning epoch, create + submit a block on our identity.
  • Pass: block lands and is seen by an independent node.
  • Evidence: height + block CID + independent-node confirmation.
• Produced block passes our own fullvalidate + is adopted by our own head path
  • Test: run our observe-validator against our own produced block.
  • Pass: passes; adopted by our head.
  • Evidence: validator result + head adoption.
• Restart resilience: restart Lantern mid-soak → Curio reconnects, proving resumes, no manual repair
  • Test: restart the soak daemon during the run.
  • Pass: Curio reconnects; proving resumes automatically.
  • Evidence: restart timestamp + resume evidence.
• Bridge outage drill: kill Forest bridge → reads keep working (bridge is production-only), alarms fire, block production degrades gracefully and resumes when bridge returns
  • Test: kill Forest mid-run; verify reads/validation continue; restore.
  • Pass: reads/validation unaffected (proves bridge-off reads); production pauses + alarms; resumes on restore.
  • Evidence: read continuity during outage + alarm + resume.
• If leak-profile build (Setup Keystore passphrase silently falls back to empty string when LANTERN_PASS is unset #2): /debug/pprof/goroutineleak clean after the full soak
  • Test: query the leak profiler post-soak (only if the leak build was used).
  • Pass: no leaked goroutines.
  • Evidence: profile summary.

Phase 5b — Zero-external block production (#89, pure-Go FVM) — NOT v1.9.0; future gate

The real full-node claim: compute the post-execution state root in-process, pure-Go FVM (wazero), so block production needs zero external VM (no Forest, no Lotus). This is gated on #89 Stage C and is explicitly out of scope for v1.9.0. Tracked here so we never overclaim "full node" while Phase 5a still leans on Forest.

• (Blocked on Stage C: pure-Go FVM via wazero (remove filecoin-ffi dep #2) #89) MinerCreateBlock computes ParentStateRoot via pure-Go FVM; no --vm-bridge-rpc; block lands on calibration
  • Test: produce a block with the VM bridge fully disconnected.
  • Pass: block lands with a state root computed in-process.
  • Evidence: height + CID + confirmation the bridge was down.

Exit criteria

Phases 1-4 + 5a checked, numbers posted as comments. Then:

• Close Full-node tier: pure-Go block-validation pipeline + tunable resident state #90 (observe-mode validated), PDP/mid tier (F1): persistent cache + pinning + block-submit + install-time tier [needs live soak] #94 (PDP tier live), [Tracking] Lantern bridge-off parity for stock Curio (maxboom / PR #1311) #76 (bridge-off parity) with links to the evidence comments
• Ship v1.9.0 final from the soaked commit
• Phase 5b / Stage C: pure-Go FVM via wazero (remove filecoin-ffi dep #2) #89 remains open — v1.9.0 is honestly "bridge-off reads + observe-validation + PDP; block production still needs an external VM (Forest, not Lotus)." Zero-external block production is the next milestone, not this release.
• Any failure: file the specific issue, fix, re-run only the failed phase

Related: #87 (epic), #100 (head-trust epic), #89 (pure-Go FVM, Phase 5b gate), PR #101, PR #103.

[Reiers]

Setup — checkbox 1: soak binary built from main ✅

Commit: 3021887 (main HEAD = merge of #107). Confirms the five soak prerequisites are baked in:

• feat(#80 part 2, #97): head-source corroboration + per-topic gossipsub scoring #101 head-source corroboration + per-topic gossipsub scoring
• chore(#102): build with Go 1.26.x #103 Go 1.26.x toolchain
• feat(#96): FRC-0089 EC finality calculator (observed-data finality) #105 FRC-0089 EC finality calculator
• feat(#93): wallet export/import (Lotus hex-KeyInfo) + import-lotus #106 wallet export/import (Lotus KeyInfo)
• feat(#98): VM-bridge cross-check auditor (observe-only divergence alarm) #107 VM-bridge cross-check auditor

Version string (running binary): lantern v1.9.0-rc1-11-g3021887 (git describe → 11 commits past v1.9.0-rc1, at 3021887).

Topology of the soak (evidence trail, per ground rules)

• Node under test = soak Lantern, ./lantern daemon --network calibration --listen 127.0.0.1:1235 --metrics 127.0.0.1:9092, pid 1023478, cwd ~/lantern-soak, on host doctor-calibration-techx.
• Runs alongside the production stack: prod chain node is Lotus (lotus daemon, separate tmux), prod Curio --layers=gui,post,storage rides Lotus. The soak Lantern replaces nothing.
• Rollback = one step: stop pid 1023478; prod Lotus + Curio unaffected.
• Soak Lantern is live at tip: ChainHead returns a real calibration tipset (head block miner f0143103, our own identity).

All subsequent phase numbers in this thread are measured against the soak Lantern on :1235, never the Lotus-backed prod node.

[Reiers]

Setup checkbox 2 — deferred (optional). Not building the GOEXPERIMENT=goroutineleakprofile binary now: it only feeds the Phase 5 /debug/pprof/goroutineleak end-check, and swapping the daemon would reset the Phase 2 sustained-sync clock. Will do a dedicated leak-profile build at Phase 5 if we want that data point. Box left unchecked.

[Reiers]

Setup — checkbox 3: Full tier + FullValidation observe ✅ (staged for Phase 1 boot)

Before: no profile file → daemon defaulted to tier=light (FullValidation OFF). So this was NOT satisfied by the running warm-up daemon.

Action: lantern node-type full --network calibration --allow-block-submit
Result on disk ~/.lantern/calibration/node-profile.json:

{ "tier": "full", "allow_block_submit": true }

Read-back: node-type: full, persistent cache 3072 MiB, block-submit opted in. Takes effect at the next daemon start (the Phase 1 cold boot) — current Light warm-up daemon left running until then. Rollback: delete node-profile.json → Light default.

What 'full' actually turns on in this binary (3021887) — honest scope

• FullValidation observe mode (Full-node tier: pure-Go block-validation pipeline + tunable resident state #90): profile.FullValidation() is true only for tier=full. At boot the daemon wires sync.SetBlockValidator(... fullvalidate.ValidateBlockConsensus ...) in observe mode (not fatal) — logs node tier: full block validation ON (observe mode, #90). Per-block it re-verifies signature / election VRF / ticket VRF / win-count / miner power against resident F3-anchored state (chainAPI.FullValidateView()), NOT the VM bridge → runs bridge-off.
• Head corroboration = 2 distinct peers (auto for Full; Light/PDP off).
• Persistent 3 GB cache + block-submit surface recorded.

Honest caveat (documented per ground rules)

'full' here does not do native bridge-off FVM state re-execution (needs CGo/filecoin-ffi; #89 Stage C, not in this binary). The source is explicit: full tier 'behaves like PDP with block-submit until the full-node track lands.' Native FVM state is only required for Phase 5 block production, where the VM bridge = Forest supplies the post-execution state root. Phase 2's fullvalidate observe counters: 100% agreement line is precisely the evidence that resident-state consensus validation actually agrees bridge-off — that's #90's acceptance.

[Reiers]

Issue restructured (per soak-owner direction, 2026-07-02)

Three changes, no scope loss:

1. Every checkbox now carries Test / Pass / Evidence — a tick is a measurement, not a vibe.
2. Tier matrix added (Light / PDP / Full) — the soak now exercises each tier where it diverges (boot + footprint + cache + validation + corroboration), not just Full. Phase 1 boots all three; Full is the end-to-end superset for Phases 2 & 5; PDP drives Phases 3-4.
3. Phase 5 split honestly:
   • 5a (v1.9.0 gate): block production is zero-Lotus, Forest supplies the post-execution state root only — proves we're not Lotus-locked. This binary has no pure-Go FVM (Stage C: pure-Go FVM via wazero (remove filecoin-ffi dep #2) #89), so an external VM is still required to stamp ParentStateRoot.
   • 5b (future, blocked on Stage C: pure-Go FVM via wazero (remove filecoin-ffi dep #2) #89): zero-external block production via in-process pure-Go FVM (no Forest, no Lotus). Explicitly not part of v1.9.0.

v1.9.0 is therefore claimed honestly: bridge-off reads + observe-validation + PDP; block production still leans on an external VM (Forest, not Lotus). The true zero-external full node is the next milestone (#89), tracked as Phase 5b.

Prior body archived internally; ticked boxes (Setup 1, Setup 3) preserved.

[Reiers]

Setup — checkbox 7: dashboard reachable ✅

Test: curl the --metrics endpoints on the soak Lantern over the tunnel.
Result:

• http://127.0.0.1:9092/dashboard/ → HTTP 200
• http://127.0.0.1:9092/metrics → HTTP 200

Dev-page URL (via tunnel): http://127.0.0.1:9092/dashboard/ (soak Lantern pid 1023478). This is where the gossip / corroboration / EC-finality / VM-crosscheck cards render for the phase screenshots.

Note: dashboard bound loopback-only (127.0.0.1), consistent with #57 (non-loopback bind would require a Bearer token).

[Reiers]

Setup — checkbox 4: bridge-off / NoFallbackRPC ✅ (capability implemented)

Finding: the running warm-up binary (3021887) had no way to disable Glif — the CLI wired it unconditionally as the last-resort runtime source at 4 sites, and default fallback = api.calibration.node.glif.io. So a provably Glif-free node wasn't achievable. (Per soak-owner call: build the real thing, not a reworded weaker claim.)

Action: implemented --no-fallback-rpc on the standalone CLI → PR #108.

• Gates all 4 runtime Glif sites (initial fetcher, polling Sync head, gossip inline-backfill, rebuilt bitswap fetcher).
• Sync + backfill get truly-nil interfaces so hstore.Sync / net/blockingest nil-guards fire (no Glif client constructed; both become no-ops). Head = gossipsub only.
• Startup guard: refuses to boot without libp2p (gossip is the sole head source).

Verified (Test/Pass):

• go build + vet + gofmt clean (incl. linux/amd64 cross); hstore + blockingest nil-guard tests pass.
• On the soak node: lantern.new version = v1.9.0-rc1-12-g896f41b; daemon --help shows the flag; guard fires on --no-libp2p and empty --p2p-listen.
• Binary staged at ~/lantern-soak/lantern.new (running daemon untouched — still serving HTTP 200).

Evidence (runtime log line) captured at Phase 1 when we cold-boot with --no-fallback-rpc and see bridge-off: no upstream RPC fallback wired (#76); head=gossipsub-only.

Honest boundary: the one-time boot anchor (#54) still uses gateway+Glif multi-source agreement — separate from runtime fetch counters. Follow-up #109 filed for anchor Glif-independence. Not a v1.9.0 blocker; runtime bridge-off is proven by Phase 2 counters.