feat: faster lookups and batch note requests by igamigo · Pull Request #1387 · 0xMiden/rust-sdk

igamigo · 2025-10-09T18:37:11Z

No description provided.

igamigo · 2025-10-09T18:40:17Z

        loop {
-            if !self
-                .sync_state_step(
-                    &mut state_sync_update,
-                    &mut current_partial_mmr,
-                    &accounts,
-                    note_tags.clone(),
-                )
-                .await?
-            {
+            let step = self
+                .sync_state_step(state_sync_update.block_num, &account_ids, note_tags.clone())
+                .await?;
+            let Some(step) = step else {
                break;
+            };
+            state_sync_update.block_num = step.block_header.block_num();
+            state_sync_steps.push(step);
+        }


Most of the code in this file is the same but rearranged:

Before, we'd do a call to sync_state which would return a set of new committed notes some of which could be public (and not currently tracked by the client), so a call to get_notes_by_id would be done subsequently

Now, all calls to sync_state are done down to the chain tip and then all public notes requests are batched into a single get_notes_by_id call

igamigo · 2025-10-09T18:40:36Z

+                "{QUERY} WHERE tx.status_variant IN ({}, {})",
+                TransactionStatusVariant::Pending as u8,
+                TransactionStatusVariant::Discarded as u8


This makes the query hit the index

igamigo · 2025-10-09T18:41:52Z

+CREATE TABLE IF NOT EXISTS tracked_accounts (
+    id TEXT NOT NULL PRIMARY KEY
+);


This is added for faster lookups. Selecting distinct account IDs still does a full table scan which means scanning through 35K rows, even with the index. We could use the settings table for this but it's not present in the 0.11 versions

Selecting distinct account IDs still does a full table scan

You mean selecting unique account ids from the accounts table, right?

I don't have the full context on the problem, but an alternative idea would be to create an index on account id (and potentially nonce?):

CREATE INDEX idx_accounts_id_nonce ON accounts(id, nonce DESC);

This is what I meant with my comment above. Even with that same index (which already exists BTW), the query scans the same amount of rows (although it does not touch the table because the index covers it). There is also concrete difference in the flamegraph when benchmarking this. I can pull up an example if you are interested later since it would require setting it up.
An equivalent alternative could be to add something like is_latest_state and do a partial index (WHERE is_latest_state=1), but not sure we prefer it.

igamigo · 2025-10-09T18:44:35Z

+    /// Applies the state sync update to the store.
+    ///
+    /// See [`crate::Store::apply_state_sync()`] for what the update implies.
+    pub async fn apply_state_sync(&mut self, update: StateSyncUpdate) -> Result<(), ClientError> {


Made this public function so that users can easily apply syncs that they do manually through the StateSync component. This should help the faucet decide what it requests. For instance, we don't need to send the account ID prefix because we don't expect to receive notes from other accounts, we only care about the ones we create.

If we give the users the option to apply this, we might also want to give them the tools to build what the state sync component needs to run correctly. Users can work around this by having an instance of their store though.

igamigo · 2025-10-09T18:47:56Z


-        let return_notes = self.rpc_api.get_public_note_records(query_notes, None).await?;
+        let mut public_notes = BTreeMap::new();
+        for chunk in query_notes.chunks(1_000) {


We need a better structured way of getting limits as defined by the node (#1139)

mmagician · 2025-10-13T15:10:27Z

+CREATE TABLE IF NOT EXISTS tracked_accounts (
+    id TEXT NOT NULL PRIMARY KEY
+);


Selecting distinct account IDs still does a full table scan

You mean selecting unique account ids from the accounts table, right?

I don't have the full context on the problem, but an alternative idea would be to create an index on account id (and potentially nonce?):

CREATE INDEX idx_accounts_id_nonce ON accounts(id, nonce DESC);

mmagician · 2025-10-13T15:11:59Z

+                "{QUERY} WHERE tx.status_variant IN ({}, {})",
+                TransactionStatusVariant::Pending as u8,
+                TransactionStatusVariant::Discarded as u8


mmagician

LGTM barring the lint (it looks like the usage of BlockUpdates::insert makes BlockUpdates::extend unused)

Co-authored-by: Marti <marti@miden.team>

igamigo · 2025-10-15T16:12:19Z

I'm looking at improving one more thing that I should be pushing in the next few minutes.

* feat: faster lookups * chore: lints * reviews: logical or, add comment * refactor: simpler inesrt function and comment * chore: lints * chore: add trace logs * chore: format * Update CHANGELOG.md Co-authored-by: Marti <marti@miden.team> * refactor: improve block header lookup and migration logic * chore: lints --------- Co-authored-by: Marti <marti@miden.team>

igamigo commented Oct 9, 2025

View reviewed changes

Comment thread crates/rust-client/src/sync/state_sync.rs Outdated

igamigo commented Oct 9, 2025

View reviewed changes

igamigo added the no docs This PR does not require an update to documentation files label Oct 9, 2025

feat: faster lookups

fe7615a

igamigo force-pushed the igamigo-sync-improvements branch from b40f3b6 to fe7615a Compare October 9, 2025 20:05

chore: lints

a954349

igamigo marked this pull request as ready for review October 9, 2025 20:23

igamigo requested a review from bobbinth October 9, 2025 20:23

mmagician requested changes Oct 13, 2025

View reviewed changes

reviews: logical or, add comment

85d9ca0

igamigo requested a review from mmagician October 14, 2025 18:42

refactor: simpler inesrt function and comment

cad1510

mmagician approved these changes Oct 15, 2025

View reviewed changes

igamigo added 3 commits October 15, 2025 10:58

chore: lints

cfb0297

chore: add trace logs

f416162

chore: format

057431c

mmagician reviewed Oct 15, 2025

View reviewed changes

Comment thread CHANGELOG.md Outdated

Update CHANGELOG.md

d7b1ebd

Co-authored-by: Marti <marti@miden.team>

igamigo added 3 commits October 15, 2025 14:17

refactor: improve block header lookup and migration logic

fc44934

chore: merge

19a736f

chore: lints

8d3d3c8

mmagician approved these changes Oct 15, 2025

View reviewed changes

mmagician merged commit 8385cbe into main Oct 15, 2025
27 checks passed

mmagician deleted the igamigo-sync-improvements branch October 15, 2025 18:30

Uh oh!

Conversation

igamigo commented Oct 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

igamigo Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mmagician left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

igamigo commented Oct 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

igamigo Oct 9, 2025 •

edited

Loading