feat(l1): add experimental REST/SSZ /new-payload-with-witness endpoint

[developeruche]

This PR introduces an experimental POST /new-payload-with-witness REST endpoint to Ethrex.

Motivation
Witness retrieval today requires two round-trips , engine_newPayload followed by debug_executionWitness, with the response serialized as JSON. For large witnesses, this encoding alone inflates payload size by roughly 2×. On Ethrex, this combined flow currently takes ~8 seconds for a 500 MB witness block.

By serving the endpoint over HTTP with SSZ serialization and merging both operations into one, we reduce that to ~1.2 seconds, a ~6.5× improvement.

Description

• Adds a REST-based POST /new-payload-with-witness route outside the JSON-RPC layer
• Uses SSZ for both inbound payload and outbound witness encoding
• Validates the payload and produces the execution witness in a single atomic operation
• Marked as experimental, consistent with how debug_executionWitness was introduced

It is also important to note that there are ongoing discussions around introducing SSZ + HTTP to the Engine API more broadly, replacing JSON-RPC entirely, including proposals to combine "new-payload" execution and witness retrieval into a single operation. Once these discussions mature, we can standardize around a unified approach. However, this is a significant architectural change and will likely take time to reach consensus.

Engine: add Rest-SSZ spec
ethereum/execution-apis#793
Add SSZ to Engine API
ethereum/execution-apis#764
Proposal: opt-in witness retrieval via flag on existing engine methods
ethereum/execution-apis#799

[greptile-apps]

Greptile Summary

This PR adds an experimental POST /new-payload-with-witness REST endpoint to the Engine API that accepts a JSON-encoded ExecutionPayload, executes the block, and returns an SSZ-encoded witness in a single round-trip — replacing the existing two-call flow and cutting reported witness retrieval time from ~8 s to ~1.2 s.

• A new witness_block_executor thread and add_block_pipeline_with_witness_inner handle block execution and background storage, with a store_lock mutex used to prevent the next witness call from reading stale trie state until the previous block's store completes.
• The nonce mismatch check in default_hook.rs is commented out with a TODO tag, disabling a core EVM transaction validation guard for the entire node.
• The v5 REST handler directly indexes params[0] before the later parse_params length check, and background store failures are silently swallowed after the caller has already received a Valid response.

Confidence Score: 1/5

Not safe to merge — the nonce validation removal in default_hook.rs breaks a fundamental EVM security invariant for the entire node, not just the new endpoint.

The commented-out nonce mismatch check in default_hook.rs means every transaction processed by this node now bypasses the guard that ensures a transaction's nonce matches the sender's current account nonce. This is a node-wide regression unrelated to the experimental endpoint's own correctness. Additionally, the witness path's background store can silently fail after returning Valid to the consensus client, and the store_lock coordination gap means the regular block executor can read stale trie state if both execution paths are ever active simultaneously.

crates/vm/levm/src/hooks/default_hook.rs (nonce check removed), crates/blockchain/blockchain.rs (background store error handling and store_lock scope), crates/networking/rpc/engine/rest.rs (params bounds check)

Security Review

• Transaction nonce validation disabled (crates/vm/levm/src/hooks/default_hook.rs): The nonce mismatch check is commented out entirely. All EVM transaction processing now accepts transactions regardless of whether the sender nonce matches the account nonce, enabling replay and out-of-order transaction attacks across the entire node.

Important Files Changed

Filename	Overview
crates/vm/levm/src/hooks/default_hook.rs	Nonce mismatch check is fully commented out with a TODO; this disables a critical EVM transaction validation guard for all transaction processing.
crates/blockchain/blockchain.rs	Adds add_block_pipeline_with_witness_inner that stores blocks in a background thread and returns the witness early; storage errors are silently swallowed, and the store_lock does not coordinate with the regular block executor path.
crates/networking/rpc/engine/rest.rs	New REST endpoint for SSZ-encoded witness responses; params[0] is accessed without a prior length check in the v5 handler, which panics on an empty JSON array body.
crates/networking/rpc/engine/payload.rs	Adds PayloadStatusWithWitness, witness-aware handler variants, and add_block_with_witness; logic mirrors the existing non-witness handlers closely.
crates/networking/rpc/rpc.rs	Wires up the new REST routes and spawns a dedicated witness_block_executor thread; the witness worker channel is always started, making the experimental endpoint unconditionally active.
crates/networking/rpc/engine/mod.rs	Advertises rest_engine_newPayloadWithWitness in exchangeCapabilities response unconditionally.

Sequence Diagram

sequenceDiagram
    participant CL as Consensus Client
    participant REST as REST Handler
    participant WE as witness_block_executor
    participant BC as Blockchain
    participant BG as Background Store Thread
    participant DB as Storage

    CL->>REST: POST /new-payload-with-witness (JWT + JSON)
    REST->>REST: authenticate + parse + validate
    REST->>WE: send(oneshot_tx, block, bal)
    WE->>BC: add_block_pipeline_with_witness(block, bal)
    BC->>BC: acquire+drop store_lock (wait for prev BG)
    BC->>BC: execute_block_pipeline
    BC->>BC: generate_witness
    BC->>BG: spawn(store_lock held by BG)
    BC-->>WE: Ok(Some(rpc_witness))
    WE-->>REST: Ok(Some(rpc_witness))
    REST-->>CL: 200 OK SSZ response
    Note over BG,DB: Async — errors only logged
    BG->>DB: store_witness
    BG->>DB: store_block_static
    BG->>BG: release store_lock

Loading

Comments Outside Diff (2)

1. crates/blockchain/blockchain.rs, line 222-259 (link)

   Store failure is silently swallowed; endpoint returns Valid on storage error

   store_block_static is called on a detached background thread, and its Err path is only logged — it is never surfaced back to the caller. The witness and Valid response have already been returned to the REST caller before the background thread even starts, so a storage error leaves the node in a state where it confirmed block acceptance to the CL but never persisted the block.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: crates/blockchain/blockchain.rs
   Line: 222-259
   
   Comment:
   **Store failure is silently swallowed; endpoint returns `Valid` on storage error**
   
   `store_block_static` is called on a detached background thread, and its `Err` path is only logged — it is never surfaced back to the caller. The witness and `Valid` response have already been returned to the REST caller before the background thread even starts, so a storage error leaves the node in a state where it confirmed block acceptance to the CL but never persisted the block.
   
   How can I resolve this? If you propose a fix, please make it concise.

2. crates/blockchain/blockchain.rs, line 136-142 (link)

   store_lock does not coordinate with the regular block executor path

   The store_lock is acquired and immediately dropped at the top of add_block_pipeline_with_witness_inner to wait for the previous iteration's background store to finish. However, the lock is never acquired by add_block_pipeline_inner (the regular block_executor thread). If both execution paths are active concurrently, the regular executor will read trie state that may not yet reflect the previous block's writes, silently producing a corrupt state root or an execution failure against stale state.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: crates/blockchain/blockchain.rs
   Line: 136-142
   
   Comment:
   **`store_lock` does not coordinate with the regular block executor path**
   
   The `store_lock` is acquired and immediately dropped at the top of `add_block_pipeline_with_witness_inner` to wait for the previous iteration's background store to finish. However, the lock is never acquired by `add_block_pipeline_inner` (the regular `block_executor` thread). If both execution paths are active concurrently, the regular executor will read trie state that may not yet reflect the previous block's writes, silently producing a corrupt state root or an execution failure against stale state.
   
   How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

Fix the following 4 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 4
crates/vm/levm/src/hooks/default_hook.rs:82-92
**Nonce validation silently removed**

The nonce mismatch check has been commented out entirely. This check is the core EVM guard that prevents accepting transactions whose nonce doesn't match the current account nonce — without it, replayed transactions, out-of-order transactions, and already-consumed nonces all pass validation. This affects every transaction processed by the EVM node, not just the new witness endpoint, and the `TODO:DEVELOPERUCHE` tag signals it was intentionally deferred rather than fixed.

### Issue 2 of 4
crates/networking/rpc/engine/rest.rs:285-298
**Panic on empty or non-array JSON body**

`params[0]` is indexed directly before any length check. If the request body is a valid JSON array with fewer than 1 element (e.g., `[]`), this panics with an index-out-of-bounds. The `parse_params` call that does validate `params.len() != 4` comes later — after this direct indexing. An authenticated request that sends `[]` as the body will panic the handler task.

```suggestion
    if params.is_empty() {
        return json_error_response(-32602, "Parse error: Expected 4 params, got 0");
    }
    let Ok(raw_bal_hash) = params[0]
        .get("blockAccessList")
```

### Issue 3 of 4
crates/blockchain/blockchain.rs:222-259
**Store failure is silently swallowed; endpoint returns `Valid` on storage error**

`store_block_static` is called on a detached background thread, and its `Err` path is only logged — it is never surfaced back to the caller. The witness and `Valid` response have already been returned to the REST caller before the background thread even starts, so a storage error leaves the node in a state where it confirmed block acceptance to the CL but never persisted the block.

### Issue 4 of 4
crates/blockchain/blockchain.rs:136-142
**`store_lock` does not coordinate with the regular block executor path**

The `store_lock` is acquired and immediately dropped at the top of `add_block_pipeline_with_witness_inner` to wait for the previous iteration's background store to finish. However, the lock is never acquired by `add_block_pipeline_inner` (the regular `block_executor` thread). If both execution paths are active concurrently, the regular executor will read trie state that may not yet reflect the previous block's writes, silently producing a corrupt state root or an execution failure against stale state.

_{Reviews (1): Last reviewed commit: "update default kurtosis yaml" | Re-trigger Greptile}

[ElFantasma]

Cross-PR: #6710 (iovoid's first engine REST/SSZ API) was closed and replaced by the now-open #6770 — both attempt the broader SSZ-REST surface with proper SSZ extractors, axum-middleware JWT auth, and a Content-Type guard. This PR adds a parallel REST handler that duplicates JWT auth (line 173-176) + SSZ response building (line 381-392) inline instead of riding on top of that infrastructure. If #6770 lands first, you'll want to rebase this onto its engine_rest module rather than maintain a second REST stack. Worth syncing with @iovoid on which lands first and whether the foundations should merge.

[ElFantasma]

The duplication concern: handle_new_payload_v3_with_witness, handle_new_payload_v4_with_witness, handle_new_payload_v1_v2_with_witness, try_execute_payload_with_witness, and validate_ancestors_with_witness are near-line-by-line copies of the non-witness counterparts above. Every future change to one path now has to be mirrored in the other or the two will silently drift.

Two shapes to think about:

1. Parameterize the existing path with compute_witness: bool (you've already added that flag to add_block_pipeline_inner's signature, this PR adds a parallel function instead of consolidating). The pipeline returns Option<RpcExecutionWitness> and the callers thread the flag through. Eliminates ~250 lines of duplicate code.
2. Wrap PayloadStatus in a trait with the witness as an associated type, so both code paths share the executor and only diverge at the response-building step.

(1) is simpler. The cost is the existing JSON-RPC path's hot path grows a never-true branch, but it's a single if compute_witness && status == Valid { ... } at the response-building site and the rest is identical. Non-blocking but the maintenance cliff at the next engine-API spec change is real.

[ElFantasma]

Panicking in an axum handler is a sharper edge than returning a 500 — it crashes the per-request task (and depending on tower configuration, can poison the connection). Response::builder() only fails if the header value is invalid, which for application/octet-stream it never is — but expect here makes that an unenforceable invariant rather than a checked one.

Map to an error response instead:

Response::builder()
    .status(StatusCode::OK)
    .header(header::CONTENT_TYPE, "application/octet-stream")
    .body(Body::from(bytes))
    .unwrap_or_else(|_| json_error_response(-32603, "failed to build SSZ response"))

Or factor through IntoResponse so the type system enforces it. Non-blocking — today's input is fixed and safe — but it's an unsafe-shaped surface in safe code.

[ElFantasma]

JWT auth is replicated inline at each handler (here at line 185, and again at line 271 in the v5 handler). #6710 / #6770 use an axum middleware (engine_auth_middleware) so every route in the engine-REST sub-router gets auth uniformly, without each handler having to remember.

For a 2-route PR this is fine. But if you keep the two-stacks-in-parallel design, the next REST endpoint added by someone unfamiliar with the auth contract is one missing line away from an unauthenticated endpoint on the engine API port. A Router::new().route(...).route(...).layer(middleware::from_fn_with_state(secret, auth_middleware)) shape would close that gap. Same argument as #6710's auth design. Non-blocking.