Prototype for SSZ-REST Engine

[syjn99]

ethereum/execution-apis#793

Note

This work assumes the devnet will start from Fusaka due to reducing boilerplate code. If a devnet starts the fork prior to Fusaka, the engine API won't work.

This PR implements SSZ-REST Engine API, which will replace current JSON-RPC engine_* methods that require hex-encoded JSON encoding between EL and CL. Adding a flag(--engine-ssz-http) will enable SSZ-REST engine.

As the following panels show, this will bring a lot of benefit including:

• Payload body size: For example, a request for engine_getPayloadBodiesByHashV1 becomes much smaller (4.7x improvement) in average.
• Latency: For example, a req-resp roundtrip has improved 17x for engine_getBlobsV2 in average.

Current caveat on implementation:

• engine_forkchoiceUpdated* method often shows poor latency. This is because 1) now CL is responsible for serialising fork choice update (See the rationale.) and 2) the current implementation in this PR uses mutex for this. We might expect better latency using singleflight.

See Appendix for config and scripts.

Design Principal

Files and packages

• beacon-chain/execution/enginehttp: Implements h2c client and wires SSZ-REST APIs.
• engine_client.go: Now only holds interfaces like Reconstructor and EngineCaller. Didn't move reconstructor methods in this file though.
• engine_transport.go: Defines core interface (engineTransport) with a wrapper for observability (instrumentedEngine).
• engine_jsonrpc.go: All legacy JSON-RPC method callers are located. jsonrpc_error.go has util functions for handling error from JSON-RPC Engine.
• engine_ssz.go: All SSZ-REST method callers are located.
• execution_block.go: All eth namespace callers are located.

Additional notes

• engineTransport interface: Abstracts the wire transport for the engine namespace. This enables us to decouple the execution service from rpcClient. The service now holds two transport: sszTransport and jsonTransport. Note that jsonTransport is a fallback for sszTransport in case the EL doesn't support SSZ-REST Engine API.
  • Abstraction for Engine transport makes us achieve the goal without much code change outside of beacon-chain/execution package.
• An execution service originally holds rpcClient and calls it with context. rpcClient cannot be fully removed as methods under eth namespace are still widely used. jsonTransport has a pointer to s.rpcClient as a primary client.
  • Directly calling rpcClient for engine methods is now discouraged: See refactoring for reconstructing block and payload.
• instrumentedEngine is for observability: It observes the latency for each call.

Plan

✅ Implementation Plan

• ✅ Build a HTTP/2 client
• ✅ Wire newly added SSZ containers for communicating with EL
• ✅ Add a feature flag --engine-ssz-http
• ✅ Abstraction: leverage EngineCaller/Reconstructor interface. New transport will implement both interfaces, and the caller will call on the interface.
• ✅ Endpoints implementation
• ✅ Interop: Currently, it seems Erigon, Nethermind and Ethrex has ongoing implementation. Interop was done with Erigon at this moment.

Appendix

Kurtosis config (with Erigon#21729)

participants:
  - el_type: erigon
    el_image: test/erigon:engine-ssz-793
    cl_type: prysm
    cl_image: prysm-bn-custom-image:engine-ssz
    vc_image: prysm-vc-custom-image:engine-ssz
    supernode: true
    count: 2
    cl_extra_params:
      - --subscribe-all-subnets
      - --verbosity=debug
    vc_extra_params:
      - --verbosity=debug

  - el_type: erigon
    el_image: test/erigon:engine-ssz-793
    cl_type: prysm
    cl_image: prysm-bn-custom-image:engine-ssz
    vc_image: prysm-vc-custom-image:engine-ssz
    supernode: false
    count: 1
    prometheus_config:
      scrape_interval: 15s
      labels:
        variant: ssz-rest
    cl_extra_params:
      - --subscribe-all-subnets
      - --verbosity=debug
      - --engine-ssz-http
    vc_extra_params:
      - --verbosity=debug

  - el_type: erigon
    el_image: test/erigon:engine-ssz-793
    cl_type: prysm
    cl_image: prysm-bn-custom-image:engine-ssz
    vc_image: prysm-vc-custom-image:engine-ssz
    supernode: false
    count: 1
    prometheus_config:
      scrape_interval: 15s
      labels:
        variant: no-ssz-rest
    cl_extra_params:
      - --subscribe-all-subnets
      - --verbosity=debug
    vc_extra_params:
      - --verbosity=debug

network_params:
  fulu_fork_epoch: 0
  genesis_delay: 40

additional_services:
  - dora
  - spamoor
  - prometheus
  - grafana

# spamoor generates load so payloads are non-empty: eoatx for normal EL txs, and blobs for blob transactions
spamoor_params:
  spammers:
    - scenario: eoatx
      config:
        throughput: 10
    - scenario: blobs
      config:
        throughput: 2

global_log_level: debug

• 2 Supernodes + 1 Normal Node with --engine-ssz-http enabled + 1 Normal Node without --engine-ssz-http.
• spamoor simulates network with transactions and blobs.
• This devnet will start from Fusaka, and never fork to Glamsterdam.

Build a docker image for Prysm

#!/usr/bin/env bash
set -euo pipefail

PLATFORM="@io_bazel_rules_go//go/toolchain:linux_arm64_cgo"
CONFIG="release"
TAG="engine-ssz"

echo "Building Prysm beacon-node image..."
bazel build //cmd/beacon-chain:oci_image_tarball \
  --platforms="${PLATFORM}" \
  --config="${CONFIG}"

docker load -i bazel-bin/cmd/beacon-chain/oci_image_tarball/tarball.tar
docker tag gcr.io/offchainlabs/prysm/beacon-chain "prysm-bn-custom-image:${TAG}"

echo "Building Prysm validator image..."
bazel build //cmd/validator:oci_image_tarball \
  --platforms="${PLATFORM}" \
  --config="${CONFIG}"

docker load -i bazel-bin/cmd/validator/oci_image_tarball/tarball.tar
docker tag gcr.io/offchainlabs/prysm/validator "prysm-vc-custom-image:${TAG}"

echo "Built images:"
docker images | grep -E 'prysm-(bn|vc)-custom-image|REPOSITORY'

Grafana Dashboard

See hack/grafana/engine_transport_dashboard.json.