PeerDAS - Tracking Issue

[AgeManning]

Description

This is a tracking issue for the work that is being undertaken on building out Data Availability Sampling.

This will be updated and edited as we go.

Instructions for running a DAS devnet can be found in this gist.

Main PR to unstable:

• PeerDAS implementation #5683

Devnet 5

• Implement getBlobSidecars support for PeerDAS #6251
• Improve range sync with PeerDAS #6258
• Reconstruct data columns without blocking processing and import #5990
• Improving blob propagation post-PeerDAS with Decentralized Blob Building #6268
• Subscribe to column subnets from PeerDAS (Fulu) epoch #5894
• Verify kzg_commitments_inclusion_proof on data column by range responses #6111
• Add additional PeerDAS metrics #6018
• Add option to select KZG library for PeerDAS #6107
• Implement validator custody (spec) Implement basic validator data column custody #6767
• Unsubscribe from blob_sidecars gossip topic from the Fulu fork #6854
• Address all TODO(das) comments
• Batch verify KZG proofs where possible Use data column batch verification consistently #6851
• Don't lookup sync individual columns received from gossip. WIP dapplion:dedup-columns-by-root-requests

TODO(das)

• Collection of small / easy TODOs Fix misc PeerDAS todos #6862
• Generalize block available time metric for PeerDAS #6850

Initial implementation

Data format and distribution (#5050)

• Add ColumnSidecar type
• Extend blobs horizontally with dummy data (0x{random_uniq_bits}...{col_id}) and distribute to column subnets
• Create 32 column subnets NUM_COLUMN_SUBNETS
• Compute subnets to subscribe to based on node_id (see att example here)
• Node to subscribe to only CUSTODY_REQUIREMENT column subnets

Data Column Custody

• Add DataColumnSidecarsByRoot req/resp protocol #5196
• Advertise extra custodied columns through ENR (@jacobkaufmann)
• Implement PEERDAS_ACTIVATION_EPOCH to transition from deneb blobs to data columns (@jimmygchen)
• Send custody data column to DataAvailabilityChecker for determining block importability #5570
• PeerDAS custody lookup sync #5684
• ❌ Proposer to store all data columns to prevent data loss (See this comment) - increase peer count and custody requirement instead

Sampling

• Add DataColumnSidecarsByRoot req/resp protocol #5196
• DAS sampling on sync #5616
• Re-process early sampling requests #5569
• Don't attempt downloading sample for custody columns already in DB
• Respond with first byte to extend timeout for early sampling requests #6929
• Network service update to maintain peers on all custody columns #6930

Syncing

• DataColumnByRange boilerplate #5353
• Fetch custody columns in range sync #5747
• Trigger sampling on sync events #5776

Misc

• feat: add DAS KZG in data col construction #5210
• Add data column kzg verification and update c-kzg. #5701
• ❌ Add EIP-7594 boilerplate  #5734
• PeerDAS spec tests #5772
• Rename PEER_DAS_EPOCH to EIP7594_FORK_EPOCH #5750
• Implement unconditional reconstruction for supernodes #5781
• Add column gossip verification and handle unknown parent block #5783
• PeerDAS parameter changes for devnet-0 #5779
• Multithread compute_cells_and_proofs #5805
• Move PeerDAS type-level config to configurable ChainSpec #5828
• Store data columns individually in store and caches #5890

peerdas-devnet-1

• Update custody subnet count format in ENR #5950

peerdas-devnet-2

• Implement MetaDataV3 with cusotdy_subnet_count #6233
• Update PeerDAS network parameters for peerdas-devnet-2 #6289

peerdas-devnet-3

• Progressively merge das to unstable #6072
• Make Checkpoint sync work for PeerDAS #6026
• Extend simulator tests to support PeerDAS #6109

Post-Pectra (Sampling, 2D PeerDAS etc)

• Update fork choice with sampling results (spec)
• Allow retrying sampling of a data column on a peer after certain amount of time #6110

Current behaviour not specified by consensus-specs

• Only supernode performs reconsturctions once >50% of columns received
• LossyDAS implementation
• Re-process DataColumnsByRoot requests and respond when columns are ready

[jimmygchen]

I'm closing this one as most items from the initial list have been completed, and the remaining tasks have either been converted into issue or are no longer necessary (please correct me if I'm wrong @dapplion).

To see the full list of PeerDAS issues, use the das label:
https://github.com/sigp/lighthouse/issues?q=state%3Aopen%20label%3A%22das%22

To see the list of PeerDAS issues required for the fulu fork, use the fulu label filter:
https://github.com/sigp/lighthouse/issues?q=state%3Aopen%20label%3A%22fulu%22