Add SSZ response support for validators and validator_balances endpoints

[Zyra-V21]

Motivation

The /eth/v1/beacon/states/{state_id}/validators endpoint currently returns JSON only. On mainnet today this is roughly 1 GB per state for the full validator set, while the equivalent SSZ payload is around an order of magnitude smaller and trivially cheaper to serialize and parse on both sides. The same applies to validator_balances (around 85 MB JSON per state).

For analytics and indexing tooling that snapshots validators frequently — for example Migalabs' goteth and similar consensus-layer indexers — JSON parsing and beacon-node CPU time during state snapshots are a real bottleneck. Exposing SSZ on these endpoints reduces both ends of the cost.

This proposal does not introduce a new design: it implements the agreement reached in the discussion of #333 (which has been open and unactioned since 2023).

Scope

Adds application/octet-stream (SSZ) response support to:

• GET /eth/v1/beacon/states/{state_id}/validators
• POST /eth/v1/beacon/states/{state_id}/validators
• GET /eth/v1/beacon/states/{state_id}/validators/{validator_id}
• GET /eth/v1/beacon/states/{state_id}/validator_balances
• POST /eth/v1/beacon/states/{state_id}/validator_balances

Each affected endpoint also gains a 406 Not Acceptable response, matching the pattern used by other SSZ-enabled endpoints in the spec (debug/state.v2, pending_partial_withdrawals, validator_identities, etc.).

Validator status encoding

ValidatorResponse is wire-encoded as the SSZ container:

{ index: uint64, balance: uint64, status: byte, validator: Validator }

Validator is the consensus-spec object. status is a single byte. The byte mapping is documented in the ValidatorStatus schema and is the one proposed by @mcdee in #333:

Value	Status
0x00	unknown
0x01	pending_initialized
0x02	pending_queued
0x03	active_ongoing
0x04	active_exiting
0x05	active_slashed
0x06	exited_unslashed
0x07	exited_slashed
0x08	withdrawal_possible
0x09	withdrawal_done

byte is used rather than uint8 per the discussion in #333 (cc @arnetheduck, @michaelsproul) to avoid implying the status is a numeric value with arithmetic semantics.

ValidatorBalanceResponse SSZ encoding is the simpler { index: uint64, balance: uint64 } container; no new type definitions are required.

What this PR does not change

• Existing JSON responses are untouched. Clients that do not advertise application/octet-stream continue to receive the same JSON they get today.
• No changes to request schemas, query parameters, or POST request bodies.
• No new endpoints, no deprecations.
• The existing ValidatorStatus JSON enum (active_ongoing, etc.) remains the canonical string form. The byte mapping is only relevant on the SSZ wire.

Lint

redocly lint beacon-node-oapi.yaml reports the same baseline of pre-existing warnings/errors before and after this patch (84 errors, 15 warnings, all unrelated to the touched files). No new lint findings are introduced.

Background and credits

The endpoint design, the choice of byte over uint8, and the status mapping all come from the discussion in #333. This PR is the implementation of that agreement; the credit for the design belongs to:

• @mcdee for opening Support SSZ for validators endpoint #333, proposing the SSZ container shape, and the status byte mapping.
• @arnetheduck and @michaelsproul for the byte vs uint8 clarification.
• @rolfyone for the consistency review around the embedded Validator object.
• @dapplion for the framing on debug/non-debug load semantics.

validator_balances was added because it shares the same indexer use case and avoids leaving an obviously asymmetric gap.

Related prior art that did not result in a merged spec change:

• feat: add validator status ssz type ChainSafe/lodestar#6059 and feat: support ssz response for validators endpoint ChainSafe/lodestar#6060 (closed; were waiting on the spec change this PR addresses).
• POST endpoints for validators and validator_balances #367 added POST variants for these endpoints but kept JSON-only responses; the SSZ side was left open.

Closes #333.

[etan-status]

mmh, would be cleaner to defer until SSZ receives proper enum support.

until then, would align validators/builders endpoints to match the same form (and keep JSON-only for now).

SSZ support doesn't need to be coordinated around a hardfork, client can indicate preferences via Accept header and server can just start shipping SSZ response once we decided on how to tackle enums.

[Zyra-V21]

Hey @etan-status, thanks for taking a look 🙏

I think we might actually be on the same page already: the PR doesn't use an SSZ enum type for status. It's encoded as a plain byte with the mapping just documented in the schema, exactly because @arnetheduck / @michaelsproul pushed back on uint8 in #333 to avoid implying any numeric/arithmetic semantics. So in that sense it's already "not depending on proper SSZ enum support" -> it's an opaque byte we agreed on out-of-band, not something the SSZ spec has to model.

So I want to make sure I understand what you'd prefer:

1. keep the opaque byte as-is (no SSZ enum machinery involved), or
2. drop status from the SSZ payload entirely for now and revisit once there's a proper enum story, or
3. hold the whole validators SSZ response as JSON-only until then?

I'd push back a bit on (3) : the main driver here is that the full validator set is ~1 GB of JSON per state on mainnet, which is a real bottleneck for indexers like goteth, so getting SSZ out is kind of the whole point. The byte was meant to be the minimal, non-controversial way to carry status without waiting on an enum decision.

One thing that might help unblock us regardless: validator_balances is just { index: uint64, balance: uint64 } : no enum, no status, nothing to defer. Happy to split that out so it can land independently while we settle the status question on the validators side.

And yeah, fully agree the validators/builders endpoints should end up consistent ; I'll follow #610/#614 so whatever we land on for enums applies the same way across both. Lmk which direction you'd like and I'll adjust.

[nflaig]

I would prefer (3) because (1) would mean we add the format as proposed here into the spec, if the enum ssz type has a different byte mapping, it would be a breaking change and we need to do a v2, there is also a cost to implementing this is in a non-standard way, fun fact, I did this almost 3 years ago here ChainSafe/lodestar#6059 but we never shipped it. But there was a lodestar branch that was able to return validators response in ssz format here ChainSafe/lodestar#6060 and it worked. But even after almost 3 years, I don't think the solution as proposed in this PR is correct as much as I want the validator response to be ssz encoded. And lastly, regarding (2), this is obviously a breaking change, and not in favor of this, we discussed removing status before (many times actually) and there was never a rough consensus on it.

[nflaig]

Also @Zyra-V21 as a side note, can you please stop using claude to comment for you, it's fine to use AI to generate code, but responding to comments and forcing others to read through the AI slop I find personally a bit direspectul if I didn't know you this PR would be closed already.

[Zyra-V21]

quick note on provenance: the 0x00 unknown … 0x09 withdrawal_done mapping here isn't new, its @mcdee's from #333, and its the same byte encoding your own ChainSafe/lodestar#6059 used, with an explicit "update to canonical once the spec defines it" todo. that's basically the posture im proposing too: ship the agreed byte form now, align if/when a canonical enum lands.

and that "if/when" is the whole point. the canonical-json work this waits on (@arnetheduck's consensus-specs#2983) was open from 2022 and closed unmerged in jan 2025. so (3) isn't "defer briefly", its json-only indefinitely, for a ~1 gb/state payload. the ssz form is opt-in via accept and purely additive, so if a canonical encoding ever supersedes this byte it's a versioning question for an opt-in content type, not a break to anything shipping today.

couple of concrete things: can validator_balances land now? it's just { index, balance }, no status, no enum, nothing your concern touches. and for validators, tell me the exact encoding you'd accept and i'll match it and align it with the builders endpoints (#610/#614) so we don't fragment this.

p.s: slop means low-quality, mass-produced content. this is a single sourced comment, and it's accurate, show me one wrong line if it isn't. saying a technically-sound pr would already be closed over how it's written rather than what it argues is shooting the messenger. let's stay on the encoding.

[nflaig]

adding ssz to the validator_balances is a no brainer, @lodekeeper can you check lodestar, we should support this since ~2 years already, it makes sense to document that in the spec, it's also backward compatible to add ssz to an existing endpoint (due to content-type negotiation) so this seems overdue

[lodekeeper]

Confirmed — Lodestar already supports SSZ for validator_balances.

GET /eth/v1/beacon/states/{state_id}/validator_balances has no onlySupport: json restriction in its route definition, so it does content-type negotiation and returns SSZ on Accept: application/octet-stream. The encoding is List[Container{index: uint64, balance: uint64}] (ValidatorBalanceListType) — which matches the shape proposed here exactly, so documenting it just standardizes what we already serve.

It's been in since #6749 ("provide first-class ssz support on api", merged 2024-06-10), shipped in v1.20.0 — so ~2 years, as you remembered.

For contrast, validators (getStateValidators/postStateValidators) is explicitly onlySupport: WireFormat.json on our side — JSON-only — which lines up with leaving that endpoint at status quo per the earlier discussion.

So +1 to documenting the validator_balances SSZ format in the spec — it's backward-compatible (Accept-header opt-in) and already implemented.

[Zyra-V21]

I'll take a read at everything carefully and will be back by Monday! Thanks everyone for your collaboration, have a good weekend

[Zyra-V21]

thanks all, just went through all the comments above and im happy where this landed

i'll pull validator_balances out into its own pr. since @lodekeeper confirmed lodestar has been serving ssz on it since #6749 (v1.20.0) with the same {index, balance} shape, this is basically just writing down what's already shipping. opt in via the accept header so nothing changes for json clients.

fwiw on @etan-status's point about whether anyone actually struggles with the json size vs just assuming it: that's literally us (@migalabs) . this is the exact spot goteth hits it. we snapshot balances every epoch and backfill big historical ranges, and at full set size it's the json encode/parse on both ends that hurts, not really the wire bytes (gzip helps there, but not with the cpu cost). ssz makes wide backfills a lot cheaper.

and yeah, fine to leave validators at status quo for now. i didn't realise #449 had already tried the v2-without-status shape and couldn't get consensus, so no point re-running that here. if ssz for validators ever happens it should go through the proper enum process like @etan-status described, and stay consistent with builders. happy to help on that when it's a thing.

thanks @nflaig @lodekeeper @etan-status for working through it

[lodekeeper]

Sounds good 👍 The concrete goteth/migalabs angle (encode/parse CPU on epoch-by-epoch balance backfills, not just wire bytes) is a useful data point to have on record. And since Lodestar already serves the {index, balance} SSZ shape, the split PR should be pure spec documentation — nothing new to implement on our side. I'll keep an eye out for it and happy to review. Thanks all.