feat(rosetta-api): display progress bar during ICP block syncing #8195
Conversation
gregorydemay
left a comment
gregorydemay
left a comment
There was a problem hiding this comment.
LGTM but I will let Mathias approve. Minor comments about the docs.
| Loaded image: localhost/rosetta:image | ||
| ``` | ||
|
|
||
| Finally run the image |
There was a problem hiding this comment.
Here for a lack of a better place, the above link gives a 404
This is a standalone server which implements the https://www.rosetta-api.org/[Rosetta API]
| Finally run the image | ||
|
|
||
| ``` | ||
| $ docker run localhost/rosetta:image |
There was a problem hiding this comment.
Here for a lack of a better place: The image name doesn't seem to match the docs:
docker load -i $(bazel cquery //rs/rosetta-api/icp:rosetta_image.tar --output=files)
...
Loaded image: rosetta:image
And then
docker run -it rosetta:image
(Note the missing localhost/rosetta:image)
mbjorkqvist
left a comment
mbjorkqvist
left a comment
There was a problem hiding this comment.
Thanks a lot for adding this, @ninegua! I have a couple of minor comments.
| ) | ||
| .unwrap() | ||
| .progress_chars("=> "), | ||
| ); |
There was a problem hiding this comment.
ICP Rosetta syncs blocks from the ledger by downloading blocks in batches of const DATABASE_WRITE_BLOCKS_BATCH_SIZE: u64 = 500000 (split up into requests of 2000 blocks each executed in parallel). Once a batch of 500000 blocks has been retrieved, the blocks are processed (blocks parsed and inserted into the database, parent hashes checked, and account balances updated). This processing (done when calling blockchain.push_batch()) can take minutes per batch, and this seemingly dead period was IMO the primary reason for adding the display progress bar. Currently, while push_batch is running, the elapsed time in the progress bar is not updated, and also the number of blocks synced is often slightly below a multiple of 500000 (e.g.,
[00:01:59] [===> ] 996007/32432586
, instead of showing 1000000/32432586). One way to keep the timer moving and perhaps let the user know that Rosetta is not stuck/hung seems to be to add a call to bar.enable_steady_tick(Duration::from_secs(1)).
We could also consider adding a message of the current step by e.g., calling bar.set_message("Retrieving blocks...") here, and calling bar.set_message("Updating balances") before calling blockchain.push_batch() (and then setting the message back to "Retrieving blocks..." after push_batch completes), WDYT?.
| if let Some(bar) = progress_bar { | ||
| bar.finish(); | ||
| } |
There was a problem hiding this comment.
I would move this bar.finish() to after the last call to blockchain.push_batch() (especially if we add the bar.enable_steady_tick call), so that the progress bar gets updated also after the last batch of blocks are processed and account balances updated.
3 participants
DEFI-1815: Print ongoing syncing status when starting up Rosetta