Sidecar KZG Proofs storage

[zilm13]

PR Description

Part of #10132, implementing #10095
Implementing only database part: storing, reading, removing KZG Proofs for DataColumnSidecars.

Main concerns:

• Serializing/deserializing location (Spec)
• Serialization choice: lists written one by one, adds 64*4=256 bytes per slot, which could be shrinked to a single int value, 4 bytes. But both are tiny numbers on the whole scale, say, with 14 target blobs supernode:
  • current storage 508Gb
  • expected: 254Gb + (481464+256)324096 = 254Gb + 5.67Gb = 259.67Gb
  • with 4 bytes instead of 256: 254Gb + (481464+4)324096 = 254Gb + 5.63Gb = 259.63Gb
    Anyway I'm happy to apply any solution which looks more elegant than this one.
• Test coverage, what could be added?

Fixed Issue(s)

Documentation

• I thought about documentation and added the doc-change-required label to this PR if updates are required.

Changelog

• I thought about adding a changelog entry, and added one if I deemed necessary.

---

Note

Adds DB column, serialization, and APIs to store/retrieve/remove KZG proofs for DataColumnSidecars, with query exposure in storage channels and client.

• Database/Schema:
  • Add new column DATA_COLUMN_SIDECARS_PROOFS_BY_SLOT to schemas (V6SchemaCombined*, Schema* adapters).
  • Implement serializer DataColumnSidecarsProofsSerializer and register in KvStoreSerializer.
  • Extend DAO layers (KvStoreCombinedDao, V4FinalizedKvStoreDao, adapters) with methods to add/remove/get List<List<KZGProof>> and fetch last slot.
  • Wire through KvStoreDatabase and Database interfaces to read/write proofs.
• APIs/Channels:
  • Add getDataColumnSidecarsProofs(UInt64 slot) to StorageQueryChannel and implement in ChainStorage, ThrottlingStorageQueryChannel, CombinedStorageChannelSplitter.
• Client:
  • Expose getDataColumnSidecarProofs(UInt64 slot) in CombinedChainDataClient.
• KZG:
  • Update KZGProof: add SIZE constant, remove SSZ helpers; minor refactor.
• Tests:
  • Add serializer property test and integration test covering add/get/remove proofs lifecycle.
• Stubs/No-op:
  • Update no-op and test stubs to include new methods.

^{Written by Cursor Bugbot for commit 6e805d0. This will update automatically on new commits. Configure here.}

[cursor]

Bug: Orphaned Proofs Bloat Database

The pruneDataColumnSidecars method removes individual data column sidecars but fails to remove the corresponding KZG proofs stored in dataColumnSidecarsProofsBySlot. Since proofs are keyed by slot and stored separately, they should be deleted via updater.removeDataColumnSidecarsProofs(slot) when pruning sidecars for each slot. This causes proofs to accumulate indefinitely in the database even after their sidecars are pruned, creating a memory leak.

storage/src/main/java/tech/pegasys/teku/storage/server/kvstore/KvStoreDatabase.java#L1211-L1245

teku/storage/src/main/java/tech/pegasys/teku/storage/server/kvstore/KvStoreDatabase.java

Lines 1211 to 1245 in 17198ce

	boolean pruneDataColumnSidecars(
	final int pruneSlotLimit,
	final Stream<DataColumnSlotAndIdentifier> dataColumnSlotAndIdentifierStream,
	final boolean nonCanonicalBlobSidecars) {
	int prunedSlots = 0;
	final Map<UInt64, List<DataColumnSlotAndIdentifier>> prunableMap = new HashMap<>();
	dataColumnSlotAndIdentifierStream
	.takeWhile(
	item -> prunableMap.size() < pruneSlotLimit || prunableMap.containsKey(item.slot()))
	.forEach(
	item -> prunableMap.computeIfAbsent(item.slot(), k -> new ArrayList<>()).add(item));
	final List<UInt64> slots = prunableMap.keySet().stream().sorted().toList();
	if (!slots.isEmpty()) {
	LOG.debug(
	"Pruning data column sidecars from slots {} to {}", slots.getFirst(), slots.getLast());
	try (final FinalizedUpdater updater = finalizedUpdater()) {
	for (final UInt64 slot : slots) {
	final List<DataColumnSlotAndIdentifier> keys = prunableMap.get(slot);
	for (final DataColumnSlotAndIdentifier key : keys) {
	if (nonCanonicalBlobSidecars) {
	updater.removeNonCanonicalSidecar(key);
	} else {
	updater.removeSidecar(key);
	}
	}
	++prunedSlots;
	}

 

---

[zilm13]

cursotbot is correct on the comment, but it will be a part of another PR together with other pruning changes. This one contains only database API, no logic

[tbenr]

I think we normally have these DB-specific serializations\deserializations here: tech.pegasys.teku.storage.server.kvstore.serialization.

Is there a reason we can't push these down there?

[tbenr]

we also already have toSSZBytes fromSSZBytes serialization\deserialization functions in KZGProof itself. Maybe we could reuse them? (we will have a concatenation of SSZs instead of one SSZ with concatenated bytes, maybe it is equivalent)

[zilm13]

KzgProof's ssz functions looks junky, removed it.
I cannot remember what was my logic to avoid KVSerializer, tried it, looks ok: 7f1c519
So it's now single number only in serialization and all proofs.

[cursor]

Bug: Fixed Size Assumption Corrupts Data

The serializer assumes all inner lists have the same size by using value.getFirst().size() as the blob size for all columns. If the input contains inner lists of varying sizes, deserialization will incorrectly regroup the proofs, causing data corruption. The deserializer splits all proofs using this single blobSize value, which fails when inner lists have different lengths.

 

[zilm13]

it's equal size by the spec

[cursor]

Bug: Empty columns corrupt data on round-trip.

The serializer cannot correctly handle empty inner lists. When serializing [[]] (one column with zero proofs), it writes blobSize=0 and no proof bytes. During deserialization, this produces [] (zero columns) instead of [[]] (one empty column). The serializer loses information about the number of columns when any column is empty, causing data corruption on round-trip.

 

[zilm13]

not a case for real data

[tbenr]

LGTM