Support all zkevm tests via both blockchain and engine by hudem1 · Pull Request #12177 · NethermindEth/nethermind

hudem1 · 2026-06-30T14:01:06Z

Support all zkevm tests via both blockchain and engine using the latest zkevm release https://github.com/ethereum/execution-specs/releases/tag/tests-zkevm%40v0.4.1

Rebased on new payload with witness. We should first merge that one.

Types of changes

What types of changes does your code introduce?

Bugfix (a non-breaking change that fixes an issue)
New feature (a non-breaking change that adds functionality)
Breaking change (a change that causes existing functionality not to work as expected)
Optimization
Refactoring
Documentation update
Build-related changes
Other: Pyspec tests

Testing

Requires testing

Yes
No

If yes, did you write tests?

Yes
No

Notes on testing

Optional. Remove if not applicable.

Documentation

Requires documentation update

Yes
No

If yes, link the PR to the docs update or the issue with the details labeled docs. Remove if not applicable.

Requires explanation in Release Notes

Yes
No

If yes, fill in the details here. Remove if not applicable.

Remarks

Optional. Remove if not applicable.

…-witness`

WitnessCapturingWorldStateProxy implemented only the explicit IWorldState methods, leaving IsNonZeroAccount, IsStorageEmpty, HasCode, GetNonce, and IsDelegatedCode to fall through to the default interface implementations. Those defaults route via TryGetAccount and see only the committed AccountStruct, while the real WorldState overrides them to consult the in-flight persistent storage provider. The mismatch broke EIP-7610 checks on SELFDESTRUCT-then-CREATE within a single block, causing the Pyspec test_recreate fixtures on Paris and Shanghai to fail with a header gas-used mismatch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Restore codeHash-first lookup and the parallel-BAL comment. The proxy already captures the address via GetCodeHash(Address) earlier in InternalGetCodeInfo, so the swap is not required for witness capture. Also drop the verbose comment on the new default-method forwarders and update the misleading note on GetCode(in ValueHash256). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Spec MAX_REQUEST_BODY_SIZE per execution-apis#764 is 16 MiB; revert the undocumented bump to 64 MiB. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Witness owns ArrayPoolList buffers. On the JSON-RPC success path the result is wrapped in ResultWrapper -> JsonRpcSuccessResponse, whose Dispose() calls Result.TryDispose(). Implementing IDisposable here routes that into Witness.Dispose() so the pool buffers are returned instead of GC'd. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

When two engine_newPayloadWithWitness calls land for the same blockHash, ArmCapture cancels the first TCS. The first caller's await captureTask then throws OperationCanceledException. The block itself executed successfully, so return VALID with a null witness rather than letting the cancellation propagate as a 500. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The INVALID_BLOCK_HASH constant and its SSZ byte mapping (4) were added but no code path produces them; Nethermind returns INVALID with a descriptive error for block-hash mismatches. Drop both until/unless we wire the status through the actual mismatch path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

BranchProcessor tracked the witness-capture lifecycle with a pair of booleans and a manual finally block to disarm on exception. Replace with a `using` session struct that arms in its factory, drains on success, and disarms on Dispose if not drained. Removes the inner try/finally and the ArmWitnessCapture / DrainWitnessCapture helpers; restores the minor cosmetic edits that crept into the file. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Replaces the Func<...> indirection the handler used to break the construction cycle. Now it takes `Lazy<IEngineRpcModule>` directly and calls `Value.engine_newPayloadV5(...)`, which simplifies the DI wiring to a plain type registration. Tests substitute IEngineRpcModule on the builder. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

newPayloadWithWitness already has both a JSON-RPC and SSZ-REST surface gated on the same fork, so route it through the same Configure(...) helper the other Amsterdam entries use rather than two parallel dict writes. Trim the comment in SszWireTypes that explains why NewPayloadWithWitnessResponseV1 is hand-rolled in SszCodec — keep the relevant why, drop the prose. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Restore the blank line after the opening brace and the inline body of engine_getClientVersionV1 — neither was needed for the PR's witness work. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Three near-identical tests differing only in the engine module factory collapse into a single [TestCaseSource]-driven case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

WitnessGeneratingWorldState.GetWitness and WitnessCapturingWorldStateProxy.BuildWitness both ran the same per-(address, slots) AccountProofCollector tree-walk loop. Pull the loop into WitnessProofCollector.CollectAccountProofs so both call sites share it. Also: - Swap Arm() order in the proxy to allocate the tracking dictionaries before flipping the armed flag; the prior order was safe only on the single-threaded ProcessOne path but inverted is a free correctness improvement. - Update WitnessCaptureRegistry xmldoc; the registry tolerates multiple concurrent armed entries for distinct block hashes (and cancels-and-replaces on duplicates). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…dispatch SszMiddleware.InvokeAsync (versioned) and DispatchWitnessAsync (non-versioned witness path) duplicated the entire post-route pipeline: PipeReader body read, request-bytes metric, handler invocation, status-code bucketing, and the 413/400/500 error mapping. Pull the shared work into DispatchAsync(handler, version, extra) — the witness path now adds only the path-specific method/content-type/handler-not-null checks. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- Use Utf8JsonReader(ReadOnlySequence<byte>) in NewPayloadWithWitnessSszHandler instead of `body.ToArray()` on multi-segment bodies — saves an LOH allocation on every request. - Promote the encode-length Debug.Assert in EncodeNewPayloadWithWitnessResponse to a runtime check so a future header-size bug fails fast in Release rather than emitting silently-wrong bytes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- PatriciaTrieWitnessGenerator: fold upsert detection into the leaf/extension walks and each walked child's result; scan the bucket for an upsert only for an absent (null) child, removing the per-occupied-child scan from the hot path (behaviour-equivalent to the previous per-branch scan). - WitnessGeneratingWorldState: tag surviving accounts Upsert (not Read) so a newly-created account occupies its post-state state-trie slot and a sibling deletion in the same branch does not falsely collapse it (over-capture). - ZkEvmMutatedWitnessIndex: key the mutated index on Category::Name (not name alone) to avoid cross-dir collisions, and throw on RLP-block/engine-payload count mismatch to lock down the blocks[i] == engineNewPayloads[i] assumption. - NewPayloadWithWitnessHandler: drain the produced witness up front so the non-Success early return disposes its pooled buffers instead of leaking them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…sts/witness-comparison

LukaszRozmej

verbose comments - as always

The witness block-processing envs hand-built BlockAccessListManager with a null transactionProcessorFactory, so the block-access list fell back to the plain Ethereum factory. On AuRa the chain overrides ITransactionProcessorFactory, and its contract-based withdrawal system calls touch the system-sender account that the plain factory omits, so the witness-generated BAL diverged from the main pipeline's and newPayloadWithWitness returned INVALID. Pass the container's ITransactionProcessorFactory while deliberately leaving the parallel parent-reader factories out, so the BAL still runs sequentially and every access is recorded into the witness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…to hudem1/feat/zkevm-tests/witness-comparison # Conflicts: # src/Nethermind/Nethermind.Evm/CodeInfoRepository.cs

The base branch was changed.

…ests/witness-comparison # Conflicts: # src/Nethermind/Nethermind.Consensus/Stateless/WitnessCapturingBlockProcessor.cs # src/Nethermind/Nethermind.Consensus/Stateless/WitnessGeneratingWorldState.cs # src/Nethermind/Nethermind.Merge.Plugin.Test/EngineModuleTests.V1.cs # src/Nethermind/Nethermind.Merge.Plugin/Handlers/NewPayloadWithWitnessHandler.cs # src/Nethermind/Nethermind.Merge.Plugin/SszRest/Handlers/SszRestPaths.cs # src/Nethermind/Nethermind.State.Test/PatriciaTrieWitnessGeneratorTests.cs # src/Nethermind/Nethermind.Trie/PatriciaTrieWitnessGenerator.cs

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Dyslex7c and others added 30 commits May 15, 2026 21:02

feat(engine): introduce REST based + SSZ serialized `new-payload-with…

0c62196

…-witness`

register handler in DI

45710c8

fix test and formatting

f5631ea

address deep review comments

c4028c5

add JSON-RPC method

1e31a1f

fix taiko tests

6671f99

address review comments

1d382f4

address review comments

5ded051

fix taiko tests and build

8d10573

remove double block execution

2997bad

merge with upstream

56551a1

fix CI build error

bf1ccb1

address prev review comments

792a296

address claude review comments

13a70ee

address claude review

6ec0663

bug fixes

5e43a8f

restore SSZ MaxBodySize to spec value (16 MiB)

c9af935

Spec MAX_REQUEST_BODY_SIZE per execution-apis#764 is 16 MiB; revert the undocumented bump to 64 MiB. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

revert cosmetic refactor in EngineRpcModule

8556efa

Restore the blank line after the opening brace and the inline body of engine_getClientVersionV1 — neither was needed for the PR's witness work. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

parameterize three Handler_calls_DisarmCapture tests

bbb6876

Three near-identical tests differing only in the engine module factory collapse into a single [TestCaseSource]-driven case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

hudem1 and others added 2 commits July 2, 2026 15:28

Merge branch 'new-payload-with-witness-ssz' into hudem1/feat/zkevm-te…

c339882

…sts/witness-comparison

hudem1 force-pushed the hudem1/feat/zkevm-tests/witness-comparison branch from 9bbd562 to c339882 Compare July 2, 2026 07:30

fix: Compare witnesses as-is (arrays) instead of sets

85315d7

asdacap previously approved these changes Jul 2, 2026

View reviewed changes

LukaszRozmej reviewed Jul 2, 2026

View reviewed changes

Comment thread src/Nethermind/Ethereum.Blockchain.Pyspec.Test/ZkEvmFixtures/Constants.cs

Comment thread src/Nethermind/Ethereum.Blockchain.Pyspec.Test/ZkEvmFixtures/ZkEvmMutatedWitnessIndex.cs Outdated

hudem1 and others added 5 commits July 3, 2026 12:51

fix: Trim comments

5457cb2

chore: trim more comments

55e7858

Merge branch 'master' into new-payload-with-witness-ssz

36aca55

Merge remote-tracking branch 'origin/new-payload-with-witness-ssz' in…

48d1aea

…to hudem1/feat/zkevm-tests/witness-comparison # Conflicts: # src/Nethermind/Nethermind.Evm/CodeInfoRepository.cs

LukaszRozmej previously approved these changes Jul 5, 2026

View reviewed changes

Base automatically changed from new-payload-with-witness-ssz to master July 5, 2026 12:03

LukaszRozmej requested review from AnkushinDaniil, LukaszRozmej and asdacap July 5, 2026 12:05

github-actions Bot added rpc taiko related to the taiko alethia rollup and removed rpc taiko related to the taiko alethia rollup labels Jul 5, 2026

AnkushinDaniil approved these changes Jul 5, 2026

View reviewed changes

Comment thread src/Nethermind/Nethermind.Trie/PatriciaTrieWitnessGenerator.cs

wurdum approved these changes Jul 5, 2026

View reviewed changes

docs: clarify AccessType.Read vs Upsert occupancy

eddff71

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

benaadams approved these changes Jul 6, 2026

View reviewed changes

hudem1 merged commit d6b155c into master Jul 6, 2026
699 of 701 checks passed

hudem1 deleted the hudem1/feat/zkevm-tests/witness-comparison branch July 6, 2026 07:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support all zkevm tests via both blockchain and engine#12177

Support all zkevm tests via both blockchain and engine#12177
hudem1 merged 136 commits into
masterfrom
hudem1/feat/zkevm-tests/witness-comparison

hudem1 commented Jun 30, 2026 •

edited

Loading

Uh oh!

LukaszRozmej left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

hudem1 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Types of changes

What types of changes does your code introduce?

Testing

Requires testing

If yes, did you write tests?

Notes on testing

Documentation

Requires documentation update

Requires explanation in Release Notes

Remarks

Uh oh!

LukaszRozmej left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

hudem1 commented Jun 30, 2026 •

edited

Loading