builderapi refactor

[pk910]

Builder API dialect separation, shared reveal/inclusion services, generic stores

TL;DR

This PR restructures the builder around a clean pre-Gloas / post-Gloas
boundary and fixes a spec-violation in the reveal path: payload envelopes
submitted via the Builder API were revealed immediately in the HTTP
handler instead of inside the reveal window (ideal 50–75% of the slot, hard
deadline PAYLOAD_DUE_BPS = 75%) — and could be published twice when p2p
bidding was active at the same time.

Why the diff is large (+10.8k / −3.7k)

Roughly half of the insertions are not hand-written logic:

Category	Lines
New/extended tests (previously near-zero coverage on these paths)	~3,300
Generated SSZ codecs (dynssz-gen, builder bid types)	~2,200
Refactored production code	~5,300

The production delta is dominated by moving code into its correct owner
(dialects, shared services) rather than rewriting it — but since packages were
renamed and split, git shows much of it as add+delete. Every commit on the
branch builds and passes the full test suite individually, so it can be
reviewed commit by commit.

The bug this fixes

submitSignedBeaconBlock (post-Gloas Builder API) built and published the
execution payload envelope inline, right after broadcasting the proposer's
block. Two problems:

1. Too early. The Gloas spec expects the reveal in the reveal window;
   revealing at block-submission time (~0–2s into the slot) exposes the
   payload before attesters commit to the block.
2. Double publish. The ePBS head-event path independently detects our
   payload in the new head and reveals it too — with both flows active, the
   same envelope went out twice.

Now all reveals go through one shared RevealService that publishes at
RevealTime, dedupes per slot (whichever flow requests first wins; the second
request is a no-op), retries ×3, and runs independently of epbs_enabled — a
Builder-API-won block reveals on time even with p2p bidding disabled.

Related fixes that fell out of the same restructuring:

• Double won_blocks rows: legacy Builder API wins were persisted twice
  (once at delivery, once by the ungated head-event path), and the head path
  recorded bogus pre-Gloas pending payments. Win detection now has a single
  owner with correct fork gating.
• Fork boundary bids: getExecutionPayloadBid resolved the fork from the
  chain head instead of the requested slot's epoch — bids for the first
  Gloas/Heze slot were signed with the wrong domain or rejected outright.
• Builder preferences lost on restart: max_execution_payment was
  memory-only; it now persists.

What changed

1. Builder API split into dialects (pkg/builderapi)

The 1,150-line server.go monolith mixed two fundamentally different flows.
It is now a thin host (route table, stats, enable fan-out) over two
self-contained dialect packages:

• builderapi/legacy — pre-Gloas flow (Electra and Fulu via the
  fork-agnostic types): registerValidators, getHeader,
  submitBlindedBlockV2. Fully fork-agnostic now: fork-agnostic builder-bid
  types with dynssz-generated SSZ (spec-resolved list limits instead of a
  hardcoded 4096), SSZ request/response content negotiation per builder-specs,
  Eth-Consensus-Version honored on requests and set on responses, block
  submission via the generic SubmitProposal.
• builderapi/epbs — post-Gloas flow (Gloas/Heze+):
  getExecutionPayloadBid, submitSignedBeaconBlock,
  submitBuilderPreferences. Block decode and proposal mapping are
  fork-agnostic; the bid fork resolves per the requested slot's epoch.

Fork guards on both sides: legacy endpoints degrade gracefully once Gloas is
active; epbs endpoints reject cleanly before it.

2. Shared Gloas+ domain services (pkg/payload_bidder)

Reveal, inclusion, and payment logic used to live inside the p2p bidding
module even though the Builder API flow needs exactly the same behavior. Now:

• RevealService — the only envelope publisher (see above). Own loop,
  channel + timer driven, no polling.
• InclusionTracker — own head-event loop; detects inclusion of our
  payloads (all forks, both flows), owns won-block records, requests the
  reveal, runs the follow-up orphan check.
• PaymentTracker — pending payments + live balance adjustments, fed by
  the two services above, consumed by lifecycle and the WebUI.
• ProposerPreferencesService — gossip proposer preferences (moved from
  the standalone pkg/proposerpreferences), pruning in its own epoch loop.

3. pkg/epbs → pkg/p2p_bidder

What remains is exactly what the name says: the active p2p bidding flow (bid
windows/tick, bid submission, competitor tracking, registration state). It has
no reveal/inclusion/payment logic and no subscriptions to the shared services
— after bidding closes for a slot, the bidder is done. epbs_enabled now
gates only this module. Config keys and CLI flags are unchanged.

4. Generic store + persistence layer (pkg/memstore, db.kv_store)

Four hand-rolled thread-safe map stores (proposer preferences, validator
registrations, builder preferences, won blocks) each duplicated the same
mutex + write-through + rehydrate pattern — one per-write DB hit each. They
are replaced by one generic memstore.Store[K,V] with buffered
write-behind persistence (dirty-set, debounced flush, one transaction per
batch, final flush on shutdown) over a single generic
kv_store(namespace, key, value BLOB) table. Flavor (codecs, validation,
namespaces) lives with each owning module. The three bespoke tables are
dropped; none of them was ever queried by column.

5. Decoupling (enforced by the import graph)

• payload_builder no longer imports Builder API code: proposer settings
  resolve through an ordered ProposerSettingsResolver chain (gossip
  preferences for Gloas+, validator registrations pre-Gloas), each
  implementation self-scoped by fork.
• builderapi and p2p_bidder do not import each other in either direction;
  lifecycle and chain no longer import feature modules.

6. go-eth2-client: fork-agnostic api/v1 types

The remaining fork-hardcoding (apiv1fulu.SignedBlockContents,
gloas.SignedBeaconBlock decodes) needed agnostic api/v1 wire types that the
library did not have. Added upstream as api/v1/all
(ethpandaops/go-eth2-client#37); this PR pins the pseudo-version.

Behavior changes

• Builder-API reveals happen at RevealTime (~58% of slot), not at block
  submission; exactly one reveal per won slot.
• "Bids Won" entries appear at inclusion (head event), not delivery;
  orphaned deliveries no longer count as wins; p2p wins now show without a
  state-db. Durable history capped at 1000 entries (same as the old UI cap).
• Legacy getHeader responds with the actual active fork version instead of
  hardcoded fulu.
• State-db migration drops the three typed store tables (best-effort caches;
  they repopulate within a slot/epoch of restart).

Testing

• Unit tests across all new/refactored packages (memstore, kv persistence,
  reveal timing/dedup/retry, inclusion + orphan detection, payment
  accounting, dialect handlers incl. SSZ negotiation, codecs, resolvers).
• make test, go vet, golangci-lint --new-from-rev green on every commit.
• Not covered by tests: end-to-end reveal timing against real CLs — needs a
  devnet run (make devnet && make devnet-run).

[Copilot]

Pull request overview

This PR refactors the builder system around a pre-Gloas vs post-Gloas boundary by splitting the Builder API into dialects, extracting shared Gloas+ services (reveal/inclusion/payment/preferences), and replacing several bespoke in-memory+DB stores with a generic buffered memstore persisted via a single kv_store table. It also updates the WebUI/API surface and documentation to align with the new shared inclusion/won-block ownership and service wiring.

Changes:

• Split Builder API into pkg/builderapi/legacy (pre-Gloas) and pkg/builderapi/epbs (post-Gloas) dialect handlers and types.
• Introduce shared pkg/payload_bidder services and move p2p-only bidding into pkg/p2p_bidder, with WebUI/API wiring updated accordingly.
• Add generic pkg/memstore + pkg/db/kv_store persistence and migrate schema to a namespaced kv_store table (dropping prior typed tables).

Reviewed changes

Copilot reviewed 97 out of 117 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pkg/webui/webui.go	Wires new services/stores into WebUI server startup and API handler construction.
pkg/webui/handlers/docs/swagger.yaml	Updates won-block schema refs/descriptions to shared inclusion-tracker ownership.
pkg/webui/handlers/docs/swagger.json	Regenerates swagger JSON to match updated won-block schema/description.
pkg/webui/handlers/docs/docs.go	Updates embedded swagger doc template to match new won-block schema/description.
pkg/webui/handlers/api/overview.go	Switches registration-state naming and pending-payments source to new modules/services.
pkg/webui/handlers/api/handler.go	Extends API handler dependencies to include shared reveal/inclusion/payment services and new validator store type.
pkg/webui/handlers/api/builder_preferences_test.go	Updates API handler construction calls for new parameter list.
pkg/webui/handlers/api/api.go	Moves bids-won ownership to inclusion tracker; updates validator listing and pending-payment computations.
pkg/rpc/beacon/client.go	Removes fork-specific submit helper; uses generic proposal submission path.
pkg/proposerpreferences/service.go	Removes old proposer-preferences SSE service (moved into payload_bidder).
pkg/proposerpreferences/cache.go	Removes old proposer-preferences cache/persistence (replaced by memstore+kv_store).
pkg/payload_builder/service.go	Replaces direct validator/prop-pref dependencies with ordered proposer-settings resolvers.
pkg/payload_builder/proposer_settings.go	Adds resolver interface and registration method for proposer settings.
pkg/payload_builder/payload_builder.go	Resolves proposer settings via resolver chain; removes fork-specific direct lookups.
pkg/payload_builder/payload_builder_test.go	Updates payload builder ctor tests for new signature.
pkg/payload_bidder/won_blocks.go	Introduces shared won-block type and kv_store codec.
pkg/payload_bidder/proposer_prefs.go	Adds new proposer-preferences SSE service and implements proposer-settings resolver for Gloas+.
pkg/payload_bidder/proposer_prefs_test.go	Adds unit tests for proposer-preferences caching/pruning/resolution and codec behavior.
pkg/payload_bidder/proposer_prefs_codec.go	Adds kv_store SSZ codec for proposer preferences.
pkg/payload_bidder/payment_tracker.go	Renames/refactors bid tracker into shared payment tracker for both flows.
pkg/payload_bidder/payment_tracker_test.go	Adds unit tests for shared payment tracker.
pkg/payload_bidder/mockchain_test.go	Adds stub chain service + helpers for payload_bidder tests.
pkg/p2p_bidder/bid.go	Renames package from epbs to p2p_bidder.
pkg/p2p_bidder/bid_tracker.go	Extracts p2p-only bid tracking into p2p_bidder module.
pkg/p2p_bidder/bid_tracker_test.go	Adds unit tests for p2p bid tracker.
pkg/p2p_bidder/bid_creator.go	Renames package from epbs to p2p_bidder.
pkg/lifecycle/manager.go	Switches lifecycle balance plumbing from epbs bid tracker to shared payment tracker.
pkg/lifecycle/balance.go	Switches effective-balance adjustments to use shared payment tracker.
pkg/epbs/reveal_handler.go	Removes old epbs reveal handler (reveal moved to shared RevealService).
pkg/epbs/payload_store.go	Removes old epbs payload store (payload retention/ownership moved).
pkg/db/won_blocks.go	Removes typed won_blocks table accessors (replaced by kv_store namespace).
pkg/db/validator_registrations.go	Removes typed validator_registrations table accessors (replaced by kv_store namespace).
pkg/db/schema/00002_kv_store.sql	Adds kv_store table and drops legacy typed store tables in migration.
pkg/db/proposer_preferences.go	Removes typed proposer_preferences table accessors (replaced by kv_store namespace).
pkg/db/kv_store.go	Adds generic kv_store persistence adapter used by memstore.
pkg/db/kv_store_test.go	Adds tests for kv_store persistence adapter behavior.
pkg/db/database_test.go	Removes tests for deleted typed tables and adjusts disabled-DB no-op test focus.
pkg/chain/service.go	Removes chain-owned proposer-preferences pruning hook (now owned by proposer-prefs service).
pkg/builderapi/validators/verify.go	Removes legacy validators package verify helper (moved into legacy dialect).
pkg/builderapi/validators/testdata_example.go	Removes old embedded example location (moved into legacy dialect test).
pkg/builderapi/validators/store.go	Removes old validator registration store (replaced by memstore+kv_store).
pkg/builderapi/mockchain_test.go	Updates mock chain service to expose configurable current fork and removes proposer-pref cache plumbing.
pkg/builderapi/legacy/types/generate.yaml	Adds dynssz-gen config for fork-agnostic legacy builder bid types.
pkg/builderapi/legacy/types/generate.go	Adds go:generate entrypoint for dynssz-gen outputs.
pkg/builderapi/legacy/types/builderbidviews.go	Adds per-fork view schemas for legacy builder bid containers.
pkg/builderapi/legacy/types/builderbid_json.go	Adds fork-aware JSON marshal/unmarshal for legacy builder bids.
pkg/builderapi/legacy/testdata/signed_validator_registrations.json	Adds embedded builder-specs example data for tests.
pkg/builderapi/legacy/submit_blinded_block.go	Adds fork-agnostic blinded-block submission with JSON/SSZ decoding and fork gating.
pkg/builderapi/legacy/settings_resolver.go	Adds pre-Gloas proposer-settings resolver backed by validator registrations.
pkg/builderapi/legacy/registrations.go	Adds kv_store SSZ codec for validator registrations.
pkg/builderapi/legacy/register_validators.go	Adds validator registration handler and signature verification in legacy dialect.
pkg/builderapi/legacy/register_validators_test.go	Updates/relocates validator registration verification tests into legacy dialect.
pkg/builderapi/legacy/http.go	Adds shared legacy dialect HTTP helpers (error writing, accept negotiation, parsing).
pkg/builderapi/legacy/handler.go	Adds legacy dialect handler struct and wiring.
pkg/builderapi/legacy/get_header.go	Adds fork-aware getHeader handler with SSZ/JSON response negotiation and registration gating.
pkg/builderapi/legacy/build_test.go	Updates legacy bid-building tests to new fork-aware APIs.
pkg/builderapi/fulu/unblind.go	Removes old fulu unblinding implementation (replaced by fork-agnostic types/paths).
pkg/builderapi/fulu/header.go	Removes old fulu header building helper (replaced by new legacy types).
pkg/builderapi/fulu/execution_payload.go	Removes old fulu payload view conversion helpers (replaced by fork-agnostic library types).
pkg/builderapi/fulu/build.go	Removes old fulu builder-bid build/sign implementation (replaced by new legacy types).
pkg/builderapi/fulu/bid.go	Removes old fulu bid types and manual SSZ implementation (replaced by dynssz views).
pkg/builderapi/epbs/types/requestauthv1.go	Adds post-Gloas request-auth type (vendored) for Builder API auth.
pkg/builderapi/epbs/types/requestauthv1_json.go	Adds JSON encoding for request-auth type.
pkg/builderapi/epbs/types/requestauthv1_yaml.go	Adds YAML encoding for request-auth type.
pkg/builderapi/epbs/types/requestauthv1_ssz.go	Adds generated SSZ encoding for request-auth type.
pkg/builderapi/epbs/types/signedrequestauthv1.go	Adds signed request-auth wrapper type.
pkg/builderapi/epbs/types/signedrequestauthv1_json.go	Adds JSON encoding for signed request-auth type.
pkg/builderapi/epbs/types/signedrequestauthv1_yaml.go	Adds YAML encoding for signed request-auth type.
pkg/builderapi/epbs/types/signedrequestauthv1_ssz.go	Adds generated SSZ encoding for signed request-auth type.
pkg/builderapi/epbs/types/builderpreferencesv1.go	Adds builder preferences type for post-Gloas builder API.
pkg/builderapi/epbs/types/builderpreferencesv1_json.go	Adds JSON encoding for builder preferences type.
pkg/builderapi/epbs/types/builderpreferencesv1_yaml.go	Adds YAML encoding for builder preferences type.
pkg/builderapi/epbs/types/builderpreferencesv1_ssz.go	Adds generated SSZ encoding for builder preferences type.
pkg/builderapi/epbs/types/builderpreferencesrequestv1.go	Adds preferences request wrapper type for submitBuilderPreferences.
pkg/builderapi/epbs/types/builderpreferencesrequestv1_json.go	Adds JSON encoding for preferences request wrapper.
pkg/builderapi/epbs/types/builderpreferencesrequestv1_yaml.go	Adds YAML encoding for preferences request wrapper.
pkg/builderapi/epbs/types/builderpreferencesrequestv1_ssz.go	Adds generated SSZ encoding for preferences request wrapper.
pkg/builderapi/epbs/types/generate.yaml	Adds dynssz-gen config for post-Gloas builder API types.
pkg/builderapi/epbs/types/generate.go	Adds go:generate entrypoint for post-Gloas type generation.
pkg/builderapi/epbs/preferences.go	Adds submitBuilderPreferences handler with request-auth verification and URL binding.
pkg/builderapi/epbs/preferences_store.go	Adds persistent builder preferences store backed by memstore+kv_store.
pkg/builderapi/epbs/preferences_store_test.go	Adds tests for builder preferences store and codec behavior.
pkg/builderapi/epbs/http.go	Adds post-Gloas dialect HTTP helpers.
pkg/builderapi/epbs/handler.go	Adds post-Gloas dialect handler struct and wiring to shared services.
pkg/builderapi/epbs/encoding.go	Adds JSON/SSZ decoding helpers for post-Gloas request bodies with content-type normalization.
pkg/builderapi/epbs/encoding_test.go	Adds tests for post-Gloas JSON/SSZ decoding helpers.
pkg/builderapi/epbs/auth.go	Adds request-auth signature verification utilities for post-Gloas builder API.
pkg/builderapi/builder_preferences.go	Removes old in-memory-only builder preferences store (replaced by persistent store).
pkg/builderapi/builder_preferences_test.go	Updates tests to use new epbs dialect constants/stores.
pkg/builderapi/bids_won.go	Removes old Builder-API-only won-block buffer (replaced by shared inclusion tracker + kv_store).
go.mod	Adds/adjusts dependencies (notably go-eth2-client pseudo-version).
go.sum	Updates module checksums for dependency changes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.