feat(client): spawn the builder preferences service

[shane-moore]

Closes #1280. Stacked on #1282; followed by #1283. GitHub's base remains epbs, so the full diff includes #1282. Review this service-wiring change against #1282 head 508d64714fe36ff876c622583b86d4a24bb99249. No release or deployment from an intermediate head.

Problem, Evidence, and Context

Produce-time builder authentication is too late to supply preferences reliably ahead of proposals. Without the builder-preferences service, the request-auth cache also lacks its pruning owner.

Change Overview

Start Lighthouse's BuilderPreferencesService under Anchor's existing Gloas-scheduled gate. Share the same Arc-backed BuilderStore and RequestAuthCache with block production. This keeps ahead-of-time signing, produce-time reuse and cache lifetime consistent without adding an Anchor-specific scheduler or cache.

The merged Lighthouse pin now skips elapsed duties, uses the target epoch's fork header, resolves per-validator configuration and signs builders within a proposer concurrently. Proposers are still sequential and publication follows collection. Therefore a below-quorum auth can delay other preferences; Anchor's two-slot future collection bound remains necessary. Successful-only marking and chunk retries are retained. Builders must tolerate replay, including after restart.

Validation

• cargo check --workspace --all-targets --locked: passed at this local stack boundary.
• Integrated stack formatting and lint: passed.
• Four Anchor operators, quorum three, ran with Lighthouse fdcc8658. Initial runs observed ahead-of-time preferences across the Gloas boundary and reuse of reconstructed auths.
• A common auth on all four operators plus a distinct second auth on only two operators delivered common preferences before all ten managed duties checked in epochs 53/54, including adjacent slots 1711/1712 and 1729/1730. The second auth produced no preference receipts. This demonstrates bounded progress in that scenario, not universal scheduling fairness.
• Restart scenarios included past/current/future duties. Early startup auth timeouts were observed and not attributed solely to head-of-line blocking.
• Empty configuration produced zero preference receipts over 75 seconds while managed attestations continued.
• All original configurations were restored byte-identically. Host clamshell sleep interrupted the longer observation and post-restoration health interval; no uninterrupted soak claim is made.

The receiver was a threaded HTTP stub. It proves receipt timing but does not verify BLS signatures, identify individual sending operators or return bids. Whole-chunk retries produced extra receipt records during transient forwarding timeouts. The runtime image preceded final diagnostic, inactive-URL, test and documentation changes; the exercised signing, service and populated valid-config paths are unchanged. The interrupted run did not establish post-sleep finality recovery. August's old-pin run remains historical evidence only.

Risks, Trade-offs, and Mitigations

The fixed pin can echo a locally produced candidate's builder URL after Anchor selects another candidate. SIP section 4 treats forwarding as a latency optimization with gossip fallback, not a liveness prerequisite. This refresh does not add a new publication subsystem. Real-builder signature acceptance and bid selection remain separate runtime evidence gaps. Hosted CI must pass on this refreshed head. These runs did not include go-ssv, so mixed-client readiness remains unproven.

Rollback

Reverting this service alone removes ahead-of-time publication and the auth cache's pruning owner. Do not deploy that state with direct builders enabled. Roll back the complete builder stack to a validated build and compatible configuration; no new database or wire format is introduced by this PR.

Blockers / Dependencies

Depends on #1282 and must be followed by #1283 before deployment. The merged Lighthouse pin clears the old upstream-stack blocker. #1302's batch receiver and the future stateless/envelope-fold integration remain separate.

[shane-moore]

@claude, please take a look

[claude-code-actions-sigp]

Claude finished @shane-moore's task in 3m 0s —— View job

---

Review: PR #1285 — spawn builder preferences service

• Read the diff against origin/epbs
• Review anchor/client/src/lib.rs (spawn + shared instances)
• Review validator_store changes (request_auth signing + tests)
• Review new test file testing/request_auth.rs
• Post review feedback

Scope note: This is a stacked PR. The only commit owned here is 89d01ca (anchor/client/src/lib.rs, +23/-7). The validator_store signing code, metrics, data_dir, and the 568-line test file are #1282's (e4f1fea), showing in the diff until it merges — I reviewed them for context but they're not this PR's to change.

Head commit (lib.rs) — the actual change

The wiring is correct:

• Move/clone semantics are sound. configured_builders and request_auth_cache are constructed once (lib.rs:732–735), .clone()d into the BlockServiceBuilder (lib.rs:743–744), and the originals moved into BuilderPreferencesService (lib.rs:884–885). No use-after-move; both types are Arc-backed so the clones genuinely share one store and one cache — which is the whole point (a second RequestAuthCache::default() would leave the block service's cache prune-less, i.e. insert-only). The rewritten comment at lib.rs:725–731 states this accurately.
• Gating is consistent with siblings. The spawn sits in the same is_gloas_scheduled() block as PayloadAttestationService and ProposerPreferencesService (lib.rs:854–890) and follows their new(...).start_update_service().map_err(...) shape one-for-one.
• No leak on non-Gloas networks. When Gloas is unscheduled, the two originals are simply dropped at scope end (the block service still got its clones). Clean.

One minor observation, not a blocker: configured_builders/request_auth_cache are constructed unconditionally (lib.rs:732–735) even on networks where is_gloas_scheduled() is false and neither the service nor any cache-insert path exists. This matches #1282's existing placement (the block service builder needs them regardless) and BuilderStore::open_or_create is cheap, so hoisting them into the Gloas gate would be churn for no real gain. Fine as-is.

I could not statically verify the BuilderPreferencesService::new argument order against the Lighthouse pin (the vendored source isn't in this checkout), but the PR reports cargo check --workspace --all-targets clean, which exercises that signature. Argument order there is Lighthouse's, not Anchor's to choose.

Stacked #1282 content (context only)

The sign_request_auth_v1 implementation and its tests are high quality and I have no correctness concerns:

• The slot-aware collection bound (request_auth_collection_bound, validator_store/src/lib.rs) uses a clean RequestAuthCollectionBound enum rather than a nested Option<Result<..>>, matching the repo's control-flow-clarity rule.
• Opaque request_auth.data is logged only by length (auth_data_len), never raw — good, honors the no-sensitive-payload rule.
• The known-answer vector test (request_auth.rs:531–568) independently re-derives the signing root with a hand-rolled Python SSZ script rather than round-tripping Lighthouse's own hashing, which makes it a genuine cross-check of the domain constant and field layout.
• The metric-reading tests correctly serialize on a tokio::sync::Mutex (METRIC_TEST_LOCK) because they assert deltas on the process-global prometheus counter — the right call to avoid flakiness.

Verdict

The head commit is a minimal, correct wiring change and is safe to review on its own merits. No changes requested. The degraded modes the description enumerates (elapsed-duty log noise, head-of-line blocking, whole-chunk resubmit) are all upstream/pin behavior confined to a feature that is off by default (empty builders file), and the PR correctly stays draft behind the Lighthouse re-pin gate. I did not run make lint/make test here; the PR reports fmt + lint + cargo check clean, and the wiring-level ACs are verified by reading the pin as the description states.
· feat/1280-wire-builder-preferences

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 14 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (epbs@8ea165f). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
anchor/client/src/lib.rs	0.00%	14 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             epbs    #1285   +/-   ##
=======================================
  Coverage        ?   80.34%           
=======================================
  Files           ?      180           
  Lines           ?    41011           
  Branches        ?        0           
=======================================
  Hits            ?    32952           
  Misses          ?     8059           
  Partials        ?        0           

Flag	Coverage Δ
rust	80.34% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[shane-moore]

Reviewed only the top commit, 89d01ca5, against its direct parent, d1a56cdd.

The Anchor wiring itself is correct: the block service and builder-preferences service share one Arc-backed store and request-auth cache, and the new service starts under the existing Gloas gate. I found one upstream behavior worth noting during soak:

1. The pinned service derives one current_fork, gathers duties from both the current and next epochs, then submits every entry with that current fork. In epoch gloas_fork_epoch - 1, it skips the current Fulu duties, gathers next-epoch Gloas duties, and sends their Gloas-only bodies with Eth-Consensus-Version: fulu. The merged validator API says the header names the version the submitted preferences belong to, while the builder spec defines a Gloas request body and says the header names the fork of the body's type. A builder that dispatches decoding by this required header can reject every pre-fork attempt, so the first Gloas epoch loses its intended ahead-of-time publication window. This behavior is in the pinned dependency, not the Anchor wiring reviewed here.

All CI checks, including run-local-testnet, pass at this head. The existing draft and soak gates remain.

Reviewed by gpt-5.6-sol max.