Skip to content

Fast block/blob archive node via ERA files #9285

Description

@dapplion

If a user wants block archive data (all blocks from genesis to head) or blob archive data (all blobs from deneb to head) the network does not naturally serve that data as nodes are not required to keep that data. Discovering other archive nodes to sync from is not guaranteed and sync times are very slow.

ERA files (spec: https://github.com/eth-clients/e2store-format-specs) are a standardized way to distribute block and blob history that we can leverage to bootstrap archive nodes. The process is:

  • Download ERA files (from URL, IPFS, torrent)
  • Load and verify them
  • Reconstruct intermediate states to get a fully synced archive node

However, turns out that writing 800 GB into LevelDB is not a good idea. The node spends (90% of the load routine compacting) and there's a huge write amplification. So this is a good opportunity to switch Lighthouse cold DB backend to static files (like Reth, Geth, and others do). Then you can get an archive node in a matter of a few hours and low I/O.

Three steps to get there:

  1. Merge static files backend (WIP in Generalize cold DB: ColdStore trait + slot-keyed static archive dapplion/lighthouse#75)
  2. Merge ERA files consumer support -> we get a fast block archive node (WIP in ERA file consumer and producer via LCLI #9273)
  3. Merge ERB files consumer support -> we get a fast blob archive node

Metadata

Metadata

Assignees

No one assigned

    Labels

    databasemajor-taskA significant amount of work or conceptual task.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions