feat: Deferred block proving by sergerad · Pull Request #1725 · 0xMiden/node

sergerad · 2026-03-02T00:20:51Z

Context

We are adding deferred (asynchronous) block proving for the node, as described #1592. Currently, block proving happens synchronously during apply_block, which means block commitment is blocked until the proof is generated.

Blocks will now exhibit committed (not yet proven) and proven states. A committed block is already part of the canonical chain and fully usable. Clients that require proof-level finality can opt into it via the new finality parameter on SyncChainMmr.

Changes

Schema: Added proving_inputs BLOB and is_proven BOOLEAN columns to the block_headers table, with partial indexes for querying unproven (is_proven = 0) and proven (is_proven = 1) blocks.
Block proof file storage: Block proofs are stored as files via BlockStore (following the existing block file pattern) rather than as BLOBs in SQLite. The DB tracks only an is_proven flag.
DB queries: Added mark_block_proven, select_unproven_blocks (excludes genesis block 0), select_block_proving_inputs (returns deserialized BlockProofRequest), and select_latest_proven_block_num.
Decoupled proving from apply_block: The BlockProofRequest is now serialized and persisted alongside the block during apply_block.
Proof scheduler: Added a background task (proof_scheduler.rs) that drives deferred proving. It queries unproven blocks on startup (restart recovery), listens for new block commits via Notify, and proves blocks concurrently using FuturesOrdered for FIFO completion ordering. Proofs are saved to files, then the block is marked proven in the DB.
Finality on SyncChainMmr: Added a Finality enum (COMMITTED, PROVEN) to the protobuf and a finality field on SyncChainMmrRequest.
Refactored apply_block query: Introduced ApplyBlockData struct to replace the 7-parameter function signature.

…ferred-block-proving

crates/store/src/db/models/queries/block_headers.rs

drahnr · 2026-03-03T14:29:21Z

crates/store/src/db/models/queries/block_headers.rs

+    let block_nums: Vec<i64> =
+        SelectDsl::select(schema::block_headers::table, schema::block_headers::block_num)
+            .filter(schema::block_headers::block_proof.is_null())
+            .filter(schema::block_headers::block_num.gt(0i64))


Q; are there proven blocks at genesis?

Genesis block is never proven.

I had to think about this; but I think it follows from the fact that the genesis block is probably invalid. It has no transactions; we just magically make state appear in it.

crates/store/src/db/models/queries/block_headers.rs

drahnr · 2026-03-03T14:41:33Z

crates/store/src/server/proof_scheduler.rs

+// ================================================================================================
+
+/// Overall timeout for proving a single block.
+const BLOCK_PROVE_TIMEOUT: Duration = Duration::from_mins(2);


We should probably make this a bit more relaxed, particularly for CI runs 🫠

Bumped to 4. Leaving comment here for others to weigh in

drahnr · 2026-03-03T14:46:17Z

crates/store/src/server/proof_scheduler.rs

+                continue;
+            },
+        };
+


I think this is racy and we need atomicity in general. The current way of calling works. I'd appreciate a comment on why this works (we have the outer Mutex<()> to protect both)

I have updated the logic to construct the notify before the db query. See the comment above that line. LMK if my understanding is incorrect.

drahnr · 2026-03-03T14:51:03Z

crates/store/src/server/proof_scheduler.rs

+        };
+        futures.push_back(fut);
+    }
+    futures


FuturesOrdered doesn't have a concurrency execution limit, it only guarantees the results are in the input order (FIFO). So https://docs.rs/futures/latest/futures/stream/trait.StreamExt.html#method.buffered might a strictly better choice, to limit the optimistic proving.

I went with adding a batch limit for the db query instead

crates/store/src/server/proof_scheduler.rs

proto/proto/rpc.proto

crates/store/src/server/proof_scheduler.rs

drahnr

Generally like the approach/LGTM, a few edges to be smoothed

…ferred-block-proving

Mirko-von-Leipzig · 2026-03-04T10:10:14Z

CHANGELOG.md

 - Introduce `SyncChainMmr` RPC endpoint to sync chain MMR deltas within specified block ranges ([#1591](https://github.com/0xMiden/node/issues/1591)).
 - Fixed `TransactionHeader` serialization for row insertion on database & fixed transaction cursor on retrievals ([#1701](https://github.com/0xMiden/node/issues/1701)).
 - Added KMS signing support in validator ([#1677](https://github.com/0xMiden/node/pull/1677)).
+- Restructured block proving to be asynchronous and added finality field for `SyncChainMmr` requests ([#1725](https://github.com/0xMiden/miden-node/pull/1725)).


Suggested change

- Restructured block proving to be asynchronous and added finality field for `SyncChainMmr` requests ([#1725](https://github.com/0xMiden/miden-node/pull/1725)).

- Added finality field for `SyncChainMmr` requests ([#1725](https://github.com/0xMiden/miden-node/pull/1725)).

Mirko-von-Leipzig · 2026-03-04T10:13:34Z

proto/proto/rpc.proto

+    // Return data up to the latest committed block (default).
+    FINALITY_COMMITTED = 0;
+    // Return data only up to the latest proven block.
+    FINALITY_PROVEN = 1;


Is the prefix a protobuf thing? (I think so).

https://protobuf.dev/best-practices/dos-donts/#unspecified-enum:

Suggested change

// Return data up to the latest committed block (default).

FINALITY_COMMITTED = 0;

// Return data only up to the latest proven block.

FINALITY_PROVEN = 1;

FINALITY_UNSPECIFIED = 0;

// Return data up to the latest committed block (default).

FINALITY_COMMITTED = 1;

// Return data only up to the latest proven block.

FINALITY_PROVEN = 2;

Mirko-von-Leipzig · 2026-03-04T10:16:40Z

proto/proto/rpc.proto

+
+    // The finality level to use when clamping the upper bound of the block range.
+    //
+    // When set to `FINALITY_COMMITTED` (default), the upper bound is clamped to the chain tip.


You shouldn't have a default value here iiuc.

Mirko-von-Leipzig · 2026-03-04T10:16:46Z

crates/store/src/errors.rs

+    NoProvenBlocks,
+    #[error("database error")]
+    #[grpc(internal)]
+    DatabaseError(#[from] DatabaseError),


Suggested change

DatabaseError(#[from] DatabaseError),

DatabaseError(#[source] DatabaseError),

Mirko-von-Leipzig · 2026-03-04T10:18:10Z

crates/store/src/state/mod.rs


+    /// Returns the database.
+    pub(crate) fn db(&self) -> Arc<Db> {
+        self.db.clone()


Suggested change

self.db.clone()

Arc::clone(&self.db)

Mirko-von-Leipzig · 2026-03-04T12:29:37Z

crates/store/src/server/proof_scheduler.rs

+    db: Arc<Db>,
+    block_prover: Arc<BlockProver>,
+    block_store: Arc<BlockStore>,
+) -> (ProofSchedulerHandle, JoinHandle<Result<(), ProofSchedulerError>>) {


I find the name a bit confusing since we have two handles. But maybe I just need to shift my perspective a bit.

Mirko-von-Leipzig · 2026-03-04T12:36:04Z

crates/store/src/server/proof_scheduler.rs

+        // Query all unproven blocks. This handles both startup recovery and new blocks.
+        let unproven_blocks = match db.select_unproven_blocks(MAX_PROVING_BATCH_SIZE).await {
+            Ok(blocks) => blocks,
+            Err(err) => {
+                error!(target: COMPONENT, %err, "Failed to query unproven blocks, retrying");
+                tokio::time::sleep(Duration::from_secs(1)).await;
+                continue;
+            },
+        };


As mentioned elsewhere, I don't think you actually need this.

You only need to track chain_tip and latest_proven_block -- which you can do essentially for free after starting up if you replace Notify with a watch::Receiver<BlockNumber>.

Mirko-von-Leipzig · 2026-03-04T12:42:09Z

crates/store/src/server/proof_scheduler.rs

+        // Construct proving jobs and drain results in order.
+        // On any failure we break immediately — dropping remaining futures cancels them.
+        // The outer loop will re-query unproven blocks and restart the sequence, ensuring
+        // we never persist a proof while an ancestor block is still unproven.
+        let mut proving_futures = order_proving_jobs(&db, &block_prover, &unproven_blocks);
+        while let Some(timeout_result) = proving_futures.next().await {


I don't think this works as expected - this essentially means we submit a set of blocks for proving, and then wait until all blocks are proven.

What we want is to constantly have N blocks being proven concurrently, when one completes, another should be queued.

Mirko-von-Leipzig · 2026-03-04T12:46:22Z

crates/store/src/server/proof_scheduler.rs

+                Err(ProveBlockError::Transient(err)) => {
+                    error!(
+                        target: COMPONENT,
+                        %err,
+                        "Block proving failed, abandoning batch and retrying next iteration"
+                    );
+                    break;
+                },


This means a single error in the batch, restarts every block proof in the batch from scratch?

Mirko-von-Leipzig · 2026-03-04T12:54:08Z

crates/store/src/server/proof_scheduler.rs

+    block_prover: Arc<BlockProver>,
+    block_store: Arc<BlockStore>,
+    notify: Arc<Notify>,
+) -> Result<(), ProofSchedulerError> {


I would propose an alternative implementation here, which I think covers my other comments as well.

imo, what we want is:

N ongoing proofs

blocks in database must only be marked as proven in sequence

What I suggest:

move as much as possible into a single proof future

it auto-retries unless fatal or overall timeout is exceeded

it saves the proof to file and essentially does everything except update the database

central loop

tracks chain_tip by replacing Notify with watch::Receiver<BlockNumber>

tracks latest proven block number by getting it as input from outside, or reading it once from database outside the loop

tracks a set of tokio::JoinSet with proof jobs

JoinSet capacity is limited to N, each is mapped to the BlockNumber

Upon task completion store result locally BlockNumber: Result<(), Error>

Add new proof job (if any)

If block number = last proven + 1, then

if Ok(()) -> update database and repeat until sequence gap is hit

if Err then crash out

sergerad added 5 commits March 2, 2026 11:19

add block_proof column and DB queries for storing block proofs

f50435e

decouple proving from apply_block and persist proving inputs

9101df8

add concurrent proof scheduler with FuturesOrdered for FIFO completion

0ef989a

Simplify prove fns

34f0cdc

Simplify retry fn

7a5022a

sergerad changed the title ~~Sergerad deferred block proving~~ feat: Deferred block proving Mar 2, 2026

sergerad added 19 commits March 2, 2026 13:34

Fix data dir issue

5062cd0

add finality parameter to SyncChainMmr endpoint

ecc067e

RM flake

f116fa3

Fix lint

866b324

Undo fmt

1f2919b

Wrap up signed block todo

14e16f4

Pass blockproofrequest down

1b21962

Lint

671630a

Fix stress tests

b6f3d3c

Changelog

6856585

Fix proving inputs

3b0ba10

Handle docstring

98c0aa6

Update genesis comment

1189d46

RM arc clone

a2b5952

load_proving_inputs comments

8a65851

Comments

d355dc5

refactor errors and retries

769d2bf

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-de…

633f761

…ferred-block-proving

Tidy up future results

549d808

sergerad marked this pull request as ready for review March 2, 2026 23:42

sergerad requested review from Mirko-von-Leipzig, bobbinth and drahnr and removed request for Mirko-von-Leipzig March 2, 2026 23:43

drahnr reviewed Mar 3, 2026

View reviewed changes

crates/store/src/db/models/queries/block_headers.rs Outdated Show resolved Hide resolved

drahnr reviewed Mar 3, 2026

View reviewed changes

crates/store/src/db/models/queries/block_headers.rs Outdated Show resolved Hide resolved

drahnr reviewed Mar 3, 2026

View reviewed changes

crates/store/src/db/models/queries/block_headers.rs Outdated Show resolved Hide resolved

drahnr reviewed Mar 3, 2026

View reviewed changes

crates/store/src/db/models/queries/block_headers.rs Outdated Show resolved Hide resolved

drahnr reviewed Mar 3, 2026

View reviewed changes

crates/store/src/server/proof_scheduler.rs Outdated Show resolved Hide resolved

drahnr reviewed Mar 3, 2026

View reviewed changes

proto/proto/rpc.proto Outdated Show resolved Hide resolved

drahnr reviewed Mar 3, 2026

View reviewed changes

crates/store/src/server/proof_scheduler.rs Outdated Show resolved Hide resolved

drahnr requested changes Mar 3, 2026

View reviewed changes

sergerad added 11 commits March 4, 2026 08:48

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-de…

98d56fa

…ferred-block-proving

Fix compile

c08b657

Rm dead code fields

258bafc

impl conv::SqlTypeConv for BlockProof

bd8dad7

Add index update query

2b44192

Bump timeout

92878d8

Update notify

7d5ed4c

Specify proving block batch size

2c5a7d7

static lifetime

8b803f7

backticks

2ca641f

replace match

6e52650

sergerad requested a review from drahnr March 3, 2026 21:07

sergerad added 4 commits March 4, 2026 11:09

Store proofs to file

295f9d8

update select proving inputs return value

5bfba05

RM pub crate

37cb516

Fix changelog

ec89137

sergerad requested a review from bobbinth March 3, 2026 22:32

Mirko-von-Leipzig requested changes Mar 4, 2026

View reviewed changes

	- Restructured block proving to be asynchronous and added finality field for `SyncChainMmr` requests ([#1725](https://github.com/0xMiden/miden-node/pull/1725)).
	- Added finality field for `SyncChainMmr` requests ([#1725](https://github.com/0xMiden/miden-node/pull/1725)).

	DatabaseError(#[from] DatabaseError),
	DatabaseError(#[source] DatabaseError),

Conversation

sergerad commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Changes

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

drahnr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sergerad commented Mar 2, 2026 •

edited

Loading