feat(node): keep intra-block ledger states live instead of persisting them

[ozgb]

Overview

Alternative implementation of #1442, replacing the approach in #1443 (that
branch's LedgerStateKey enum and per-transaction persist/unpersist are
reverted here — see "Relationship to #1443" below).

Every intra-block ledger state was made a GC root so that it stayed addressable
across the host_api boundary: apply_transaction / apply_system_transaction
persisted the state they produced and unpersisted their predecessor. The persist
existed for one mechanical reason — Sp::drop runs when the Bridge method
returns, and uncache on a non-persisted CacheValue::Create removes it from
the arena entirely — so rooting it was the only thing keeping it addressable for
the next call.

This holds the Sp in a process-global keep-alive cache instead, released by the
successor call. One persist() per block (the post-block tip) replaces one
persist plus one unpersist per transaction.

The win is fewer deserializations, not fewer DB transactions

Worth stating plainly because it changes what to benchmark. Sp::persist /
StorageBackend::unpersist were already single-key in-memory refcount updates on
the write cache, and the only parity-db commit is flush_all_changes_to_db, once
per block from on_finalize. observe_storage_flush_time is not expected to
move.

What is removed: the arena's caches hold binary OnDiskObjects and sp_cache
holds only Weak refs, so dropping the Sp tree at every host-call boundary
meant every host call re-deserialized the ledger working set from scratch
(Arena::get_lazy → force_as_arc → T::from_binary_repr). One transaction
crosses that boundary ~3x — get_tx_weight → get_transaction_cost,
pre_dispatch → validate_guaranteed_execution, then apply_transaction —
plus once at on_finalize. Holding the live Sp, for the intra-block
intermediates and for the post-block tip, collapses that to one materialisation
per block.

The two caches

	transient	anchored
holds	intra-block intermediates	post-block tips
persisted?	no — the cache is the only thing keeping them addressable	yes
capacity / TTI	1024 / 60s	4 / 300s
release	explicit, by the successor call	none needed
refcounted	yes	no
a miss is	fatal (NoLedgerState)	one re-materialisation

The transient half is refcounted because two block executions can be in flight at
once (authoring alongside import, unguarded outside warp recovery) and two forks
off the same parent applying the same transaction produce the same content
hash; without a count the first release kills the second execution's keep-alive.
This is not new bookkeeping so much as relocated bookkeeping — today's safety in
that scenario comes from the arena's root counter being a counter rather than a
flag. Its capacity is deliberately ~1000x a resident set of one: moka's TinyLFU
admission filter can reject an insert when the cache is at capacity, and here a
rejected insert is a fatal mid-block miss. The TTI is the leak bound, for an
execution that abandons its tail (an unsealed proposal, a failed import).

ledger_state_cache_size{cache_type="transient"|"anchored"} reports both,
sampled once per block at the post-block flush — after the block's last
intermediate has been released, which is what makes transient == 0 the clean
signal.

Consensus safety

The emitted state key cannot change. The Ledger root is always an
ArenaKey::Ref, so Sp::persist's Direct → Ref promotion never fired and
as_typed_key() is byte-identical with or without the persist.

Relationship to #1443

With release keyed on cache membership, the Anchored/Transient tag gates
nothing: the output variant was already a per-function constant, so the decision
came from which Bridge fn you are in and never from the input. Reverting it
deletes a permanent storage migration (pallet-midnight v2→v3) and a permanent
host-fn ABI version at zero behavioural cost. pallet_midnight::StateKey is
Vec<u8> again, STORAGE_VERSION is 2 again, and the four #[version(2)]
ledger-9 host functions are gone.

git diff origin/main -- runtime/ pallets/ primitives/ ledger/src/{common,host_api}/
is empty apart from a comment, the test-only mock / test-utils additions and
a host-side metric, so the runtime's SCALE surface is unchanged from main and
the checked-in .scale metadata files are back to main's bytes. Will still run
/bot rebuild-metadata to confirm.

Also fixes has_ledger_state, which is documented to return false for an
unresolvable state key (and used that way by warp ledger-sync's recovery monitor)
but reached that branch by panic.

Drive-by: deletes pre_fetch_storage, which had no caller anywhere in the tree.

Dependency note: moka 0.11.3 → 0.12 (needed for the atomic entry().and_compute_with
refcount, and 0.12 dropped the background housekeeping thread, so an Sp can no
longer be dropped — i.e. mutate the arena — from outside block execution). It is
a net dependency reduction: 0.12.15 was already in the lock via
hickory-resolver, so this collapses two vendored copies into one and drops 9
transitive crates.

🗹 TODO before merging

• Performance evaluation (see "Not measured here" below) — this is why the PR is a draft
• /bot rebuild-metadata to confirm the metadata revert
  • Not needed - no metadata changes, no need to rebuild
• Confirm no qanet/preview/dev chain is running fix: stop persisting intermediate ledger states #1443's runtime (reverting STORAGE_VERSION 3→2 is only safe on chains that never applied v3; no tag or release contains those commits and spec_version never changed, but an ad-hoc devnet from that branch would need a v3→v4 down-migration instead)
• fix: stop persisting intermediate ledger states #1443 closed or retargeted, and christos-ab-control-no-unpersist (which builds on the enum) rebased
  • Pending performance eval of this branch
• Ready

📌 Submission Checklist

• All commits are signed off (git commit -s) for the DCO
• Changes are backward-compatible (or flagged if breaking)
• Pull request description explains why the change is needed
• Self-reviewed the diff
• I have included a change file, or skipped for this reason:
• If the changes introduce a new feature, I have bumped the node minor version
• Update documentation (if relevant)
• Updated AGENTS.md if build commands, architecture, or workflows changed
• No new todos introduced

🧪 Testing Evidence

pallets/midnight/tests/persist_refcount.rs is rewritten around the keep-alive
contract and asserts, at every transition of a three-block lifecycle:

• intermediates are never GC roots (get_state_root_count == None) and are
  addressable only because the cache holds them (transient_state_is_retained),
• the successor call releases its predecessor, so the transient cache is empty
  after on_finalize — the unit-test mirror of the gauge,
• reads through an intermediate tip mid-block succeed:
  get_ledger_state_root(), get_transaction_cost() (the #[pallet::weight]
  path), and validate_unsigned / pre_dispatch resolving the state rather than
  failing NoLedgerState,
• a released intermediate is gone from the arena entirely after the block's
  flush (has_ledger_state == false) while the persisted tip survives — i.e.
  intermediates never reach disk,
• the refcount: apply tx to parent → H; rewind StateKey and apply the
  same tx again → the same H; advance one branch → H still retained; advance
  the other → released.

Both mechanisms were mutation-tested — making release_transient unconditionally
Remove fails the fork assertion, stubbing out retain_transient fails the
keep-alive assertion.

cargo test -p midnight-node-ledger                     78 passed
cargo test -p pallet-midnight --features test-utils     24 passed, 2 ignored
cargo test -p midnight-node --test storage_migration     1 passed
cargo test -p midnight-node warp_ledger_sync            14 passed
cargo test -p midnight-node --lib                      148 passed, 2 ignored
cargo clippy --workspace --all-targets                   0 warnings (1 pre-existing profile-spec warning)
cargo fmt --all --check                                  clean

storage_migration is the regression guard that matters for teardown: each
retained Sp holds Arcs into Storage<D>, so a surviving cache entry keeps the
parity-db handle and its exclusive file lock alive past
unsafe_drop_default_storage. It caught two real bugs during development —
clearing the caches only inside the per-DB branches (ledger 8 and 9 share one
storage backend but have their own copies of the statics), and relying on moka's
invalidate_all, which only stamps a valid_after marker instead of dropping
values.

Local dev node (CFG_PRESET=dev, release build, --alice --validator), authoring
and finalizing normally, no ledger errors:

ledger_state_cache_size{cache_type="transient"} 0
ledger_state_cache_size{cache_type="anchored"}  4

transient at 0 at every block boundary is the leak check. anchored sits at its
capacity of 4 rather than 1 — with no explicit release, consecutive tips
accumulate until the LRU evicts, which is by design and always safe there since
they are persisted (four consecutive tips share nearly all structure).

Not measured here — over to the performance team

1. The actual win. With RUST_LOG=midnight::ledger_v2=trace, submit a burst
   so several txs land in one block and diff the ⏱️ Ledger loaded elapsed for
   the 1st vs 2nd+ tx of a block, and for the 1st tx of consecutive blocks.
   Before, all three are comparable; after, the 2nd+ within a block should be
   ~0 (transient cache) and the 1st of each block ~0 too (anchored cache). Those
   two numbers are the whole justification for the change. They also confirm the
   design assumption that arena.get_lazy resolves through sp_cache to the
   retained Arc — if they don't move, an explicit cache lookup in get_ledger
   is needed after all.
2. observe_txs_processing_time should drop;
   observe_storage_flush_time("ledger_state") should be flat (if it moves,
   something else changed).
3. RSS across a few hundred blocks of catch-up sync. Retention now grows
   monotonically within a block rather than resetting at each host boundary, so
   peak could step up by roughly one block working set plus the retained tips.
   Measure, don't argue.

hardfork_e2e has not been run locally (needs CI images); the enum revert
restores main's host ABI and StateKey layout exactly, so the fork path is
byte-identical to main.

• Additional tests are provided (if possible)

🔱 Fork Strategy

• Node Runtime Update
• Node Client Update
• Other:
• N/A

Client-side only. The runtime's SCALE surface, StateKey layout, host-function
ABI and emitted state keys are all identical to main, so this needs no runtime
upgrade and no migration, and a node running it can sync a chain built by one
that isn't.

Links

Closes #1442
Supersedes #1443

[gilescope]

I can attest that this is a problem worth solving. @ozgb I'd suggest syncbox testing this PR to make sure it doesn't upset anything. If that's fine then gets a thumbs up from me.