Track blinded payload-envelope storage + EL-backed sync serving (incl. BAL restore)

[lodekeeper]

Status (2026-10-01) — decomposed, closing

The serve path for blinded payload-envelope storage shipped in #10089: blinded finalized-archive storage + EL-backed reconstruction for execution_payload_envelopes_by_range / _by_root, with transactions / withdrawals / blockAccessList restored and root-verified on every read (incl. unit handler tests for short-return, RESOURCE_UNAVAILABLE, and by-root omit-on-mismatch).

Remaining work is split into focused follow-ups:

• Retroactively dedupe execution payload envelopes archived while the block was optimistic #10235 — retroactively dedupe envelopes archived while the block was still optimistic
• Cross-client interop/regression test for blinded payload-envelope serve + BAL restore #10236 — cross-client interop/regression test for the serve path + BAL restore (blocked on an EL with engine_getPayloadBodiesByHashV2 + BAL retention, Update EIP-7928: align BAL retention with the history expiry window ethereum/EIPs#12347)

Hot-DB / import-time blinding is intentionally deferred and kept as the TODO GLOAS in packages/beacon-node/src/chain/blocks/writePayloadEnvelopeInputToDb.ts — not tracked as a separate issue.

Acceptance criteria (original)

• Persist execution payload envelope / block metadata in blinded form without duplicating the full EL body — feat: de-duplicate execution payload from archived Gloas envelopes #10089 (finalized archive; hot DB stays full by design)
• Reconstruct missing execution payload body data from EL when serving full data back out — feat: de-duplicate execution payload from archived Gloas envelopes #10089
• Verify execution_payload_envelopes_by_range serving works — feat: de-duplicate execution payload from archived Gloas envelopes #10089
• Verify execution_payload_envelopes_by_root serving works — feat: de-duplicate execution payload from archived Gloas envelopes #10089
• Verify Gloas-specific body fields (including BAL) are restored correctly before serving — feat: de-duplicate execution payload from archived Gloas envelopes #10089
• Mixed-client interop / regression test so we do not repeat Prysm's late unblinding/BAL serve bug → Cross-client interop/regression test for blinded payload-envelope serve + BAL restore #10236
• Retroactively dedupe payloads archived while their block was still optimistic → Retroactively dedupe execution payload envelopes archived while the block was optimistic #10235

Original issue description

Problem

We already know Lodestar should avoid persisting duplicate full execution payload bodies on the CL side and instead keep a blinded representation + reconstruct the missing EL-side body data on demand.

There is already a broad tracking issue for the storage dedup idea:

• De-duplicate payload from persisted beacon blocks #5671 De-duplicate payload from persisted beacon blocks

But we should keep a more specific pre-testnet reminder for the serve path:

if Lodestar stores the execution payload envelope / block in blinded form, it must still be able to serve peers correctly during sync (especially beacon_blocks_by_range / beacon_blocks_by_root style serving), reconstructing the missing execution payload body data from EL and restoring any Gloas-specific fields such as BAL before serving.

Terence warned that Prysm hit a late bug in this area: they were not unblinding / re-adding BAL correctly when serving to peers.

Why this needs a dedicated reminder

This can easily be missed if we only test:

• local import / processing
• proposer-side flows
• REST full-block reconstruction

The tricky part is the sync serving path:

• range sync serving works
• block/root serving works
• the served payload is reconstructed from EL correctly
• BAL / other body-only fields are restored correctly before serving
• peers can consume the served block/payload without blinded/full mismatch bugs

Prysm reference points

I could not cleanly pull the exact ethpandaops/prysm:glamsterdam-devnet-0 branch into the local checkout, but the current local Prysm repo already shows the relevant shape:

1. Store the envelope in blinded form

beacon-chain/db/kv/execution_payload_envelope.go

• SaveExecutionPayloadEnvelope() replaces the full payload with its blockHash
• keeps metadata like executionRequests, builderIndex, beaconBlockRoot, slot, stateRoot
• comment explicitly says the full payload can later be retrieved from EL via engine_getPayloadBodiesByHash

2. Reconstruct full payload bodies from EL on demand

beacon-chain/execution/payload_body.go

• reconstructBlindedBlockBatch()
• requestBodiesByHash() using engine_getPayloadBodiesByHashV1
• fallback requestBodiesByRange() using engine_getPayloadBodiesByRangeV1
• fullPayloadFromPayloadBody() rebuilds the full execution payload from the blinded header + EL body data

3. Use reconstructed full blocks for serving/API paths

beacon-chain/rpc/eth/beacon/handlers.go

• GetBlockV2() reconstructs the full block when the stored block is blinded:
  • ExecutionReconstructor.ReconstructFullBlock(ctx, blk)

Lodestar context

Lodestar already has the relevant TODO in:

• packages/beacon-node/src/chain/blocks/writePayloadEnvelopeInputToDb.ts

// TODO GLOAS: Persist envelope metadata (stateRoot, executionRequests, builderIndex, etc.) without the full
// execution payload body — only keep the blockHash reference. The EL already stores the payload.
// See https://github.com/ChainSafe/lodestar/issues/5671

Lodestar also already has EL payload body fetch methods:

• packages/beacon-node/src/execution/engine/http.ts
  • getPayloadBodiesByHash()
  • getPayloadBodiesByRange()

Requested work / acceptance criteria

Before testnets, we should make sure Lodestar either implements or explicitly tests the full serve path for blinded envelope storage:

• Persist execution payload envelope / block metadata in blinded form without duplicating the full EL body
• Reconstruct missing execution payload body data from EL when serving full data back out
• Verify beacon_blocks_by_range serving works
• Verify beacon_blocks_by_root serving works
• Verify Gloas-specific body fields (including BAL) are restored correctly before serving
• Add a mixed-client interop / regression test so we do not repeat Prysm's late unblinding/BAL serve bug
• Retroactively dedupe payloads that were archived while their block was still optimistic — once the EL is synced, sweep the archive and blind the envelopes that were stored full (a deep back-sync, or the EL catching up slowly, leaves un-deduped full SignedExecutionPayloadEnvelope entries; see the #10089 review thread and the executionPayloadEnvelopeArchive.ts line-35 comment)

Related

• De-duplicate payload from persisted beacon blocks #5671 De-duplicate payload from persisted beacon blocks

[avdheshcharjan]

Hi! I'd like to take a swing at this if no one is already working on it.

Disclosure per CONTRIBUTING.md: I'm using Claude Code to assist me with codebase exploration and research. The decisions and posted analysis below are mine; I've read every file I cite below, plus the relevant SSZ shapes in packages/types/src/gloas/. CC was also used to write/format this issue.

What I think is in scope from inside Lodestar

After reading the relevant files I think the issue's six checkboxes naturally split into two PRs:

PR1 — storage shape + reconstructor (no behavior change for peers):

• New SSZ types in packages/types/src/gloas/sszTypes.ts: BlindedExecutionPayload, BlindedExecutionPayloadEnvelope, SignedBlindedExecutionPayloadEnvelope (matching the existing Signed{Blinded}{X} pattern, e.g. SignedBlindedBeaconBlock).
• Pure helpers in packages/state-transition/src/util/blindedEnvelope.ts (mirroring the existing blindedBlock.ts convention).
• New packages/beacon-node/src/chain/blocks/reconstructPayloadEnvelope.ts that wraps engine_getPayloadBodiesByHashV1 / …ByRangeV1, recomputes transactionsRoot + withdrawalsRoot, and throws on mismatch (security invariant — exactly the Prysm late-bug class).
• writePayloadEnvelopeInputToDb writes blinded form (gloas-gated). Hot DB only — migrateExecutionPayloadEnvelopesFromHotToColdDb unblinds before writing to the archive bucket so historical serve stays a byte-stream. Note this adds ~32 Engine API calls per finalization to the archive migrator, batched into a single …ByRangeV1 per epoch rather than one per block. Happy to alternatively keep the archive blinded too if the perf trade is preferred — that path is also fine, it just pushes the EL fetches to serve time.
• chain.getExecutionPayloadEnvelope splits into getBlindedEnvelope() (DB only) and getFullEnvelope() (DB + reconstruct). getParentExecutionRequests uses the blinded variant since executionRequests lives outside the body.
• Unit tests including mandatory tx-root and withdrawals-root mismatch cases.

PR2 — wire into serve + interop:

• executionPayloadEnvelopesByRoot/Range handlers call getFullEnvelope. Range handler batches via engine_getPayloadBodiesByRangeV1.
• Range-serve partial-failure policy: my read of the existing strict-throw at executionPayloadEnvelopesByRange.ts:60-64 is that it should stay strict — a peer that can't reconstruct because its EL is down genuinely is a worse peer until EL is back, and silently skipping risks downstream sync-state confusion. Will go with strict throw + warn log unless you'd prefer otherwise.
• Two-node E2E in packages/beacon-node/test/e2e/sync/ (alongside finalizedSync.test.ts) validating byte-identical serve.

Happy to land it as one PR or split PR1 further (storage shape vs. reconstructor as separate PRs) if smaller pieces are preferred — two felt like the right balance between bisectability and avoiding mid-sequence dead-code PRs, but that's your call.

One blocker I want to flag before working on this issue

engine_getPayloadBodiesByHashV1 (and the …ByRangeV1 variant) currently return only {transactions, withdrawals} — see packages/beacon-node/src/execution/engine/types.ts:168-176:

export type ExecutionPayloadBodyRpc = {
  transactions: DATA[];
  withdrawals: WithdrawalV1[] | null | undefined;
};

But the Gloas ExecutionPayload (sszTypes.ts:169-176) gained two new fields:

blockAccessList: BlockAccessList, // New in GLOAS:EIP-7928
slotNumber: Slot,                 // New in GLOAS:EIP-7843

Neither is in the V1 body shape. So a Lodestar reconstructor calling existing engine_getPayloadBodies* has nothing to put in the BAL field — this is the exact unblinding/BAL bug the issue references from Prysm. slotNumber has the same gap.

Three options for the missing fields, in order of ship-from-inside-Lodestar viability:

1. Keep BAL + slotNumber inline in the "blinded" form — only transactions and withdrawals get replaced with roots. Storage win shrinks (BAL is bounded by MAX_BYTES_PER_TRANSACTION but typical sizes are small; slotNumber is 8 bytes so trivially cheap to keep) but the work is shippable today without coordinating an Engine API spec change. My default if you're OK with it.
2. Block on engine_getPayloadBodiesByHashV2 with BAL + slotNumber added — needs cross-client agreement (Geth, Reth, Besu, Nethermind) and is not a Lodestar PR.
3. Recompute BAL by re-executing transactions against the parent state (effectively running the EL inside the CL). Listed for completeness; obviously not viable.

Is anyone already working on (2) on the EL side? If so I'll align with that; if not, I'll proceed with (1) and we can swap later when the V2 Engine API arrives.

Side questions

• Does the existing TODO comment in packages/beacon-node/src/chain/blocks/writePayloadEnvelopeInputToDb.ts pointing at De-duplicate payload from persisted beacon blocks #5671 belong to anyone in particular? Don't want to step on toes.
• Any preference between landing this on unstable directly vs a feature branch given this is pre-testnet Gloas work?
• Is there an EPF cohort or other group working on Gloas pre-testnet items that this should coordinate with? Happy to join a Discord thread or tracking issue if there's one I should know about.

Thanks!

[lodekeeper]

Thanks — this is a very good read of the problem, and the split sounds reasonable to me.

A few concrete reactions from my side after checking both local Prysm code and Lodestar’s current glamsterdam branch:

1. Yes, the broad shape matches what I was trying to capture in this reminder issue.

   • Prysm already has the relevant pattern in-tree today:
     • beacon-chain/db/kv/execution_payload_envelope.go stores the envelope in blinded form
     • beacon-chain/execution/payload_body.go reconstructs the body from EL via engine_getPayloadBodiesByHashV1 with ...ByRangeV1 fallback
     • beacon-chain/rpc/eth/beacon/handlers.go reconstructs full blocks on serve/API paths

2. Your blocker looks real, and I think option (1) is the pragmatic path for now.
   I re-checked Lodestar’s current glamsterdam-devnet-0 branch and the payload really does include:

   • blockAccessList in packages/types/src/gloas/sszTypes.ts:172
   • slotNumber in packages/types/src/gloas/sszTypes.ts:173

   Meanwhile the current EL payload-body API surface in Lodestar is still just transactions + withdrawals for the body fetch path.

   So unless there is already active EL-side work for a V2 payload-bodies API, keeping BAL + slotNumber inline in the blinded form, while de-duplicating transactions and withdrawals, sounds like the most shippable pre-testnet direction.

3. The two-PR split sounds fine to me.
   The way you framed it makes sense:

   • PR1: storage shape + reconstruction helpers/path
   • PR2: serve/range-sync wiring + interop/regression coverage

   That feels like a good balance between reviewability and not leaving a weird half-dead intermediate state.

4. Target unstable directly.
   I’m not aware of a separate branch/owner for this specific item.

5. On the existing TODO pointing to De-duplicate payload from persisted beacon blocks #5671:
   I don’t know of a specific owner for it. I opened this issue mostly as a narrower reminder for the Gloas-specific serve/range-sync/BAL-restore path, because that can get missed even if the general payload-dedup direction is understood.

6. On strict serve failure:
   your read sounds reasonable to me for the initial implementation. If reconstruction from EL fails, I’d rather have that fail loudly than silently serve malformed/incomplete data.

So from my side: if you want to take a swing at it, go for it, and I’d currently bias toward your option (1) + the two-PR split unless someone from the EL side chimes in with an imminent V2 payload-bodies path.

[nflaig]

cc @markolazic01 (need to ping so I can assign)

[nflaig]

cc @markolazic01 (need to ping so I can assign)

ok nvm, that doesn't work, github doesn't allow me to assign this issue unless you comment here yourself, but it's fine, consider yourself assigned

[nflaig]

@lodekeeper is there anything remaining in this issue after we merge #10089?

[lodekeeper]

@lodekeeper is there anything remaining in this issue after we merge #10089?

Yes — one real item, one optional.

1. Mixed-client interop / regression test (last box). It's not in feat: de-duplicate execution payload from archived Gloas envelopes #10089 (the description defers it and the diff is unit tests only), and it's the one that catches a Prysm-style unblind/BAL bug on the wire: a Lodestar node with a blinded archive serving execution_payload_envelopes_by_range/root to a different-client peer that syncs from it. It needs an EL with engine_getPayloadBodiesByHashV2 + BAL retention (Update EIP-7928: align BAL retention with the history expiry window ethereum/EIPs#12347 is the retention-gap discussion). The handler-level tests flagged in review on feat: de-duplicate execution payload from archived Gloas envelopes #10089 (short return after a yield / RESOURCE_UNAVAILABLE / by-root omit) are the same gap at unit level.
2. Hot-DB blinding (optional). feat: de-duplicate execution payload from archived Gloas envelopes #10089 only blinds the finalized archive; writePayloadEnvelopeInputToDb.ts still writes the full envelope and keeps the TODO GLOAS … #5671. The hot DB only holds the unfinalized window, so the saving is small — I'd drop the TODO (noting hot stays full by design) rather than build it. Either way the TODO needs updating since it points at De-duplicate payload from persisted beacon blocks #5671, which feat: de-duplicate execution payload from archived Gloas envelopes #10089 closes for the Gloas scope. Your call.

The other boxes are covered by #10089: blinded archive storage, EL reconstruction, by-range/by-root serve, and transactions / withdrawals / BAL restored and root-verified on every read.

I'd keep this issue open for item 1 (retitle it to the interop test if you agree); #10089 says "relates to", so it won't auto-close it.

[lodekeeper]

Decomposed this umbrella per the #dedupe-payloads discussion. The serve path shipped in #10089; the two remaining follow-ups are now focused issues:

• Retroactively dedupe execution payload envelopes archived while the block was optimistic #10235 — retroactively dedupe envelopes archived while the block was still optimistic (the executionPayloadEnvelopeArchive.ts line-35 / feat: de-duplicate execution payload from archived Gloas envelopes #10089 review-thread item)
• Cross-client interop/regression test for blinded payload-envelope serve + BAL restore #10236 — cross-client interop/regression test for the serve path + BAL restore (blocked on an EL with engine_getPayloadBodiesByHashV2 + BAL retention, Update EIP-7928: align BAL retention with the history expiry window ethereum/EIPs#12347)

Hot-DB / import-time blinding stays as the TODO GLOAS in writePayloadEnvelopeInputToDb.ts rather than its own issue.

Closing as decomposed — reopen if you'd rather keep this as a parent tracker.