Skip to content

refactor(ledger): give each ledger version its own wrapper - #2059

Open
ozgb wants to merge 6 commits into
mainfrom
ozgb-split-ledger-common
Open

refactor(ledger): give each ledger version its own wrapper#2059
ozgb wants to merge 6 commits into
mainfrom
ozgb-split-ledger-common

Conversation

@ozgb

@ozgb ozgb commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Overview

Closes #1768.

ledger/src/versions/common/** (4451 lines) was compiled twice through a
#[path = "versions"] module-parameterization trick — once bound to the ledger-8
crates, once to the ledger-9 crates — with super:: resolving differently in each
instantiation:

#[path = "versions"]
pub mod ledger_8 {
    pub(crate) use { mn_ledger_8 as mn_ledger_local, ... };   // the parameters
    #[allow(clippy::duplicate_mod)]
    mod common;                 // -> ledger/src/versions/common/**
    pub use common::*;
}

A reader of versions/common/api/ledger.rs could not tell what mn_ledger_local
referred to, and an edit meant for one version silently applied to both. Per the
issue, that footgun cost several days of investigation and crashed nodes.

Each version now owns its files, with absolute crate::ledger_N::… imports so
every file states the version it binds:

ledger/src/
  lib.rs        pub mod ledger_8;  pub mod ledger_9;   (no #[path], no duplicate_mod)
  boundary/     was common/ — the SCALE types crossing the runtime/client
                interface, version-independent by design, still ONE copy
  host_api/     already used absolute paths; 2 lines changed
  ledger_8/     mod, types, storage, conversions, utxo_ordering_override,
                block_context, error_ext, system_tx, guaranteed_validation,
                post_block_update, api/{mod,ledger,transaction}
  ledger_9/     same 12 files

Nothing is called common any more, which is the other half of the issue: the
crate-level common/ was never shared behaviour — it is the runtime/client wire
format, deliberately version-agnostic, and the pallet has one decoder for it. It is
renamed to boundary/ rather than duplicated; versioning that wire format would be
a design change, not a refactor.

No behaviour change. rustc already emitted both instantiations, so the +4.6k
lines only make the existing duplication visible in the source tree — no new machine
code, no binary-size or compile-time change. There was no non-test version branching
in the shared half to reinterpret (the only CRATE_NAME comparisons are
if CRATE_NAME != crate::latest::CRATE_NAME { return } test guards, which still
work unchanged). Every import rewrite is compile-checked, so there is no
silent-failure mode.

Public paths are untouched — ledger_8::…, ledger_9::…, latest::…, types::…
(incl. types::active_version and types::active_ledger_bridge), host_api::….
The 27 files outside ledger/ that import from this crate needed zero changes;
the only edits outside lib.rs/ledger_8/ledger_9 are two common::
boundary:: import lines in host_api/.

🗹 TODO before merging

  • Ready

📌 Submission Checklist

  • All commits are signed off (git commit -s) for the DCO
  • Changes are backward-compatible (or flagged if breaking)
  • Pull request description explains why the change is needed
  • Self-reviewed the diff
  • I have included a change file, or skipped for this reason:
  • If the changes introduce a new feature, I have bumped the node minor version
  • Update documentation (if relevant)
  • Updated AGENTS.md if build commands, architecture, or workflows changed
  • No new todos introduced

🧪 Testing Evidence

The review artifact

The point of the layout is that the two directories can be diffed. Normalise the
version path away, then diff:

cp -r ledger/src/ledger_9 /tmp/n9
grep -rl ledger_9 /tmp/n9 | xargs sed -i 's/ledger_9::/ledger_8::/g'
diff -ru ledger/src/ledger_8 /tmp/n9

Exactly five files differ, and they are precisely the ones that were already
per-version before this PR:

file divergence
error_ext.rs GenerationInfoAlreadyPresent (v8) vs InitialNonceAlreadyPresent (v9)
guaranteed_validation.rs full apply() dry-run (v8) vs apply_guaranteed_only (v9)
post_block_update.rs no-op (v8) vs prevalidate_post_block_update (v9)
system_tx.rs tuple vs struct SystemTransaction variants; PayBlockRewardsToTreasury only on v8
mod.rs the crate aliases, CRATE_NAME, TransactionSignature

Everything else — types.rs, storage.rs, conversions.rs,
utxo_ordering_override.rs, block_context.rs, api/{mod,ledger,transaction}.rs
is byte-identical modulo the version path. That is the evidence the copy is faithful,
and going forward it is the standing check for what the two versions actually do
differently.

Builds and tests

check result
cargo check -p midnight-node-ledger --all-targets 0 errors, no new warnings
cargo clippy --workspace --all-targets --features runtime-benchmarks,try-runtime 0 errors, no new warnings
SKIP_WASM_BUILD=1 cargo check --workspace --all-targets 0 errors
cargo build -p midnight-node-runtime (real no_std / WASM path) builds
cargo test -p midnight-node-ledger 82 passed — same count as before the change
cargo test -p pallet-midnight -p pallet-midnight-system -p pallet-c2m-bridge (+ ledger) 124 passed, 2 ignored
external consumers: rg 'midnight_node_ledger::' -g '!ledger/**' 27 files, unchanged

Note: bare cargo check -p midnight-node-ledger --no-default-features fails, but it
fails identically on main (405 errors) — it is not a supported feature
configuration. The no_std path is exercised by the runtime WASM build above.

Metadata rebuild required

scale-info's #[derive(TypeInfo)] records module_path!() in each type's Path,
so the runtime metadata embeds Rust module paths. Moving these modules therefore
changes the metadata even though nothing about the encoding changed — no variant,
discriminant, field, or field order moved, and boundary/types.rs is a 100% rename
while ledger_9/types.rs differs from its old copy by a single use line.

Decoding every midnight_node_ledger-rooted path out of the old and new blobs
gives 18 renames, 18 unchanged type names, nothing added or removed:

  • common::types::{Op, Tx, UtxoInfo} -> boundary::types::{…}
  • ledger_9::common::types::{DeserializationError, DisjointCheckErrorCode, EffectsCheckErrorCode, FeeCalculationErrorCode, InvalidError, LedgerApiError, MalformedContractDeployErrorCode, MalformedError, MalformedZswapErrorCode, SequencingCheckErrorCode, SerializationError, SystemTransactionError, TransactionApplicationErrorCode, TransactionError, ZswapInvalidErrorCode}
    -> ledger_9::types::{…}

The blob shrinks by exactly 99 bytes (148858 -> 148759), which is the arithmetic of
those renames and nothing else: 15 paths drop the common segment (-7 bytes each, a
compact length prefix plus six characters) and 3 grow from common to boundary
(+2 bytes each). Anything already encoded on chain still decodes identically.

Renaming ledger_9::common::types is unavoidable — that module is what the issue
asks to remove — so metadata had to be rebuilt either way; the crate-root
common -> boundary rename adds three more paths to the same churn. Downstream
subxt codegen will see the new runtime_types::… module paths. No in-repo consumer
references them (the indexer's runtimes/v*.rs are pinned to older metadata
snapshots), but it is worth a heads-up to anyone generating fresh bindings.

No genesis rebuild: no storage item, extrinsic, or runtime API changed.

  • Additional tests are provided (if possible)

🔱 Fork Strategy

  • N/A

Source-layout only. The runtime WASM blob may differ byte-wise since panic locations
embed source paths, but spec_version and behaviour are unchanged, so no coordinated
update is needed.

Links

Closes #1768

Follow-ups (deliberately not in this PR)

  • Hoist what is genuinely version-agnostic back to one copy. Now that the copies
    are separate you can diff them and decide per item — but that judgement call is the
    multi-day analysis the issue describes. Pure duplication first.
  • ledger/helpers/src (6166 lines) and util/toolkit/src
    (commands/fork/common, tx_generator/builder/builders/common, 4003 lines) use the
    same trick. Same recipe, separate PRs.
  • Drop the non-latest test copies. The
    if CRATE_NAME != crate::latest::CRATE_NAME { return } guards already make them
    no-ops.

🤖 Generated with Claude Code

@ozgb
ozgb requested a review from a team as a code owner August 21, 2026 10:26
@ozgb ozgb added the bot:ai-assisted Authored or substantially edited by an AI agent label Aug 21, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 294f681d57

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

}

#[derive(Debug, Encode, Decode, DecodeWithMemTracking, Clone, TypeInfo, PalletError, PartialEq)]
pub enum LedgerApiError {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the error-encoding invariant across copies

When either independently editable copy gains or reorders a version-specific error variant, the SCALE discriminants can diverge, but host_api/ledger_9.rs::as_ledger_9_error still converts v8 errors by encoding and decoding them as this v9 enum. The existing test samples only a few variants and does not cover nested transaction errors, so historical v8 reads can silently return the wrong v9 error or fall back to HostApiError. Keep the encoding-bearing error definitions shared, use an explicit conversion, or exhaustively enforce their compatibility.

Useful? React with 👍 / 👎.

@datadog-official

This comment has been minimized.

@ozgb

ozgb commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

/bot rebuild-metadata

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b9425e310a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ledger/src/ledger_9/types.rs
@ozgb ozgb changed the title refactor(ledger): give each ledger generation its own copy of the wrapper refactor(ledger): give each ledger version its own wrapper Aug 21, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

LGLO
LGLO previously approved these changes Aug 28, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 60fb008a52

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@@ -0,0 +1,67 @@
#[cfg(feature = "std")]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add the required license header

This newly added Rust source file starts directly with a cfg attribute and omits the repository’s mandatory license header. Prepend the contents of LICENSE_HEADER.txt so the new source file complies with the project’s licensing requirements.

AGENTS.md reference: AGENTS.md:L239-L241

Useful? React with 👍 / 👎.

gilescope
gilescope previously approved these changes Sep 1, 2026
ozgb added 5 commits September 1, 2026 13:11
…pper

`ledger/src/versions/common/**` was compiled twice through a
`#[path = "versions"]` module-parameterization trick — once bound to the
ledger-8 crates, once to the ledger-9 crates — with `super::` resolving
differently in each instantiation. A reader of a file under `versions/common/`
could not tell which ledger crate `mn_ledger_local` referred to, and an edit
meant for one generation silently applied to both.

Each generation now owns its files under `ledger/src/ledger_8/` and
`ledger/src/ledger_9/`, with absolute `crate::ledger_N::…` imports so every
file states the generation it binds. `ledger/src/common/` — the SCALE types
crossing the runtime/client interface, version-independent by design and
compiled once — becomes `ledger/src/boundary/`, so no folder is called
`common` any more. All `#[path]` and `#[allow(clippy::duplicate_mod)]`
scaffolding is gone from `lib.rs`.

No behaviour change: rustc already emitted both instantiations, so this only
makes the existing duplication visible in the source tree. There was no
non-test version branching in the shared half to reinterpret, public paths
(`ledger_8::…`, `ledger_9::…`, `latest::…`, `types::…`, `host_api::…`) are
unchanged, and the only edits outside `ledger/src/{lib.rs,ledger_8,ledger_9}`
are two `common::` -> `boundary::` import lines in `host_api/`.

Normalising the version path away, `diff -r src/ledger_8 src/ledger_9` leaves
exactly five differing files — `error_ext.rs`, `guaranteed_validation.rs`,
`post_block_update.rs`, `system_tx.rs`, and `mod.rs`'s crate aliases,
`CRATE_NAME` and `TransactionSignature`. Everything else is identical, which
is the check this layout buys.

Issue: #1768

Assisted-by: Claude:claude-opus-5
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Assisted-by: Claude:claude-opus-5
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
`scale-info`'s `#[derive(TypeInfo)]` records `module_path!()` in each type's
`Path`, so the runtime metadata embeds Rust module paths verbatim. Moving the
ledger wrapper into per-generation directories therefore changes the metadata's
type registry even though nothing about the SCALE encoding moved, and
`+check-metadata` byte-diffs the blob.

Decoding every `midnight_node_ledger`-rooted path out of the old and new blobs
gives 18 renames, 18 unchanged type names, and nothing added or removed:

  common::types::{Op, Tx, UtxoInfo}
    -> boundary::types::{Op, Tx, UtxoInfo}

  ledger_9::common::types::{DeserializationError, DisjointCheckErrorCode,
    EffectsCheckErrorCode, FeeCalculationErrorCode, InvalidError, LedgerApiError,
    MalformedContractDeployErrorCode, MalformedError, MalformedZswapErrorCode,
    SequencingCheckErrorCode, SerializationError, SystemTransactionError,
    TransactionApplicationErrorCode, TransactionError, ZswapInvalidErrorCode}
    -> ledger_9::types::{...}

The whole blob shrinks by exactly 99 bytes (148858 -> 148759), which is the
arithmetic of those renames and nothing else: 15 paths drop the `common`
segment (-7 bytes each, one compact length prefix plus six characters) and 3
grow from `common` to `boundary` (+2 bytes each).

No variant, discriminant, field, or field order changed -- `boundary/types.rs`
is a 100% rename of `common/types.rs`, and `ledger_9/types.rs` differs from the
`versions/common/types.rs` it came from by a single `use` line. Anything already
encoded on chain still decodes identically; only the names a fresh subxt codegen
emits are different.

`midnight_metadata_2.1.0.scale` is regenerated alongside `midnight_metadata.scale`
because the two are the same blob (that file tracks main rather than the release
it names).

Assisted-by: Claude:claude-opus-5
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
ozgb dismissed stale reviews from gilescope and LGLO via d692478 September 1, 2026 12:12
@ozgb
ozgb force-pushed the ozgb-split-ledger-common branch from 60fb008 to d692478 Compare September 1, 2026 12:12
ozgb added a commit that referenced this pull request Sep 1, 2026
…e helpers

`ledger/helpers/src/versions/common/**` was compiled twice via a
`#[path = "versions"]` module-parameterization trick, once bound to the ledger-8
crates and once to the ledger-9 crates, with `super::` resolving differently in
each instantiation. A reader of a file under `versions/common/` could not tell
which ledger crate `mn_ledger` referred to, and an edit meant for one version
silently applied to both — the same footgun #2059 removed one layer down.

Each version now has its own directory, `ledger_8/` and `ledger_9/`, with
absolute `crate::ledger_N::…` imports so every file states the version it binds.
The two inline modules in `lib.rs` become the respective `mod.rs` files, and the
version-specific single files are renamed to line up across the two trees:

  versions/block_context/post_ledger_8.rs -> ledger_N/block_context.rs
  versions/ecdsa_unimpl.rs                -> ledger_8/ecdsa.rs
  versions/test_utilities_compat.rs       -> ledger_8/test_utilities_local.rs
  versions/ecdsa_wallet_tests.rs          -> ledger_9/ecdsa_wallet_tests.rs

`ledger_9/ecdsa.rs` and `ledger_9/test_utilities_local.rs` are new one-line
re-exports of what `lib.rs` used to alias inline, so the two directories hold
the same filenames and `diff -r` is meaningful.

No behaviour change: rustc already emitted both instantiations. Public paths are
unchanged (`pub use common::*;` already flattened `common` away), so no consumer
needed an edit and the runtime metadata is untouched. The same 150 unit tests
and 2 doctests run, under the same names minus the `common::` segment.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
ozgb added a commit that referenced this pull request Sep 1, 2026
`tx_generator/builder/builders/common/**` and `commands/fork/common/**` were
each compiled twice via a `#[path = "common"] pub mod inner { … }` trick, once
bound to the ledger-8 helpers and once to the ledger-9 helpers, with
`ledger_helpers_local` resolving differently in each instantiation. A reader of
a file under `common/` could not tell which ledger version it was looking at,
and an edit meant for one version silently applied to both. This finishes the
job #2059 and the ledger-helpers split started.

Each version now has its own directory — `builders/{ledger_8,ledger_9}/` and
`commands/fork/{ledger_8,ledger_9}/` — and each file names its own version with
a `use midnight_node_ledger_helpers::ledger_N as ledger_helpers_local;` line, so
the two copies stay byte-identical apart from that one word and `diff -r` can
police them. The only genuine divergence left is `builders/*/mod.rs`, which
carries the per-version `serialize_tx`.

The `inner` wrapper module is gone (nothing referenced it), so every external
path — `builders::ledger_8::SingleTxBuilder`, `fork::ledger_9::show_wallet`, … —
is unchanged, as is the CLI surface.
`impl_encoded_zswap_conversions!` stays in `builders/mod.rs`: `ledger_storage`
still aliases to `ledger_storage_ledger_8` in both versions, so duplicating the
impls per version would still hit E0119.

With this, `grep -r '#\[path' --include='*.rs'` over the workspace returns
nothing, and no directory is compiled more than once.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
ozgb added a commit that referenced this pull request Sep 1, 2026
…e helpers

`ledger/helpers/src/versions/common/**` was compiled twice via a
`#[path = "versions"]` module-parameterization trick, once bound to the ledger-8
crates and once to the ledger-9 crates, with `super::` resolving differently in
each instantiation. A reader of a file under `versions/common/` could not tell
which ledger crate `mn_ledger` referred to, and an edit meant for one version
silently applied to both — the same footgun #2059 removed one layer down.

Each version now has its own directory, `ledger_8/` and `ledger_9/`, with
absolute `crate::ledger_N::…` imports so every file states the version it binds.
The two inline modules in `lib.rs` become the respective `mod.rs` files, and the
version-specific single files are renamed to line up across the two trees:

  versions/block_context/post_ledger_8.rs -> ledger_N/block_context.rs
  versions/ecdsa_unimpl.rs                -> ledger_8/ecdsa.rs
  versions/test_utilities_compat.rs       -> ledger_8/test_utilities_local.rs
  versions/ecdsa_wallet_tests.rs          -> ledger_9/ecdsa_wallet_tests.rs

`ledger_9/ecdsa.rs` and `ledger_9/test_utilities_local.rs` are new one-line
re-exports of what `lib.rs` used to alias inline, so the two directories hold
the same filenames and `diff -r` is meaningful.

No behaviour change: rustc already emitted both instantiations. Public paths are
unchanged (`pub use common::*;` already flattened `common` away), so no consumer
needed an edit and the runtime metadata is untouched. The same 150 unit tests
and 2 doctests run, under the same names minus the `common::` segment.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
ozgb added a commit that referenced this pull request Sep 1, 2026
`tx_generator/builder/builders/common/**` and `commands/fork/common/**` were
each compiled twice via a `#[path = "common"] pub mod inner { … }` trick, once
bound to the ledger-8 helpers and once to the ledger-9 helpers, with
`ledger_helpers_local` resolving differently in each instantiation. A reader of
a file under `common/` could not tell which ledger version it was looking at,
and an edit meant for one version silently applied to both. This finishes the
job #2059 and the ledger-helpers split started.

Each version now has its own directory — `builders/{ledger_8,ledger_9}/` and
`commands/fork/{ledger_8,ledger_9}/` — and each file names its own version with
a `use midnight_node_ledger_helpers::ledger_N as ledger_helpers_local;` line, so
the two copies stay byte-identical apart from that one word and `diff -r` can
police them. The only genuine divergence left is `builders/*/mod.rs`, which
carries the per-version `serialize_tx`.

The `inner` wrapper module is gone (nothing referenced it), so every external
path — `builders::ledger_8::SingleTxBuilder`, `fork::ledger_9::show_wallet`, … —
is unchanged, as is the CLI surface.
`impl_encoded_zswap_conversions!` stays in `builders/mod.rs`: `ledger_storage`
still aliases to `ledger_storage_ledger_8` in both versions, so duplicating the
impls per version would still hit E0119.

With this, `grep -r '#\[path' --include='*.rs'` over the workspace returns
nothing, and no directory is compiled more than once.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:ai-assisted Authored or substantially edited by an AI agent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove shared common modules to isolate ledger-version behavior

3 participants