[parallel exec] eth mainnet invalid block: could not apply tx 25203874:81 [0xed4738fe081d00a22996dfe8b20b59e72fecfd596dee51789f7197ce087ffcca]: too many validator-invalid retries: 251, expected: 250

[taratorio]

EXEC3_PARALLEL=true
Prysm CL
git commit a1f9699

[WARN] [05-29|22:01:01.014] [5/5 Execution] rw exit                  err="invalid block: could not apply tx 25203874:81 [0xed4738fe081d00a22996dfe8b20b59e72fecfd596dee51789f7197ce087ffcca]: too many validator-invalid retries: 251, expected: 250" stack="[exec3_parallel.go:278 exec3_parallel.go:653 exec3_parallel.go:653 exec3.go:253 stage_execute.go:395 default_stages.go:328 sync.go:500 sync.go:331 stageloop.go:685 executor.go:308 fork_validator.go:297 fork_validator.go:259 exec_module.go:520 chain_reader.go:284 engine_server.go:1020 engine_server.go:475 engine_api_methods.go:216 value.go:586 value.go:369 service.go:236 handler.go:673 handler.go:572 handler.go:513 handler.go:276 handler.go:426]"

[VBulikov]

yperbasis added Imp0 as Importance

[mh0lt]

Looked at the block / tx contents to see if a focused unit test would close this without a 25M-block from-0 repro. Findings:

Block 25,203,874 (248 txs) — tx 81 calls createForwarder(address,address,bytes32) (selector 0x13b2f75c) on factory 0x37996e762fa8b671869740c79eb33f625b3bf92a. CREATE2 deploys a forwarder at fresh address 0x8524661e81e30103483e5e481f12c45207242c4e (pre-block code = 0 bytes, NOT a recreate of a previously-destroyed address — so the metamorphic-SD pattern from #20711 isn't the trigger here).

Tx 82 in the same block calls the same factory with a different salt (deploys to a different forwarder). So the factory's own state (nonce, internal storage) is touched by both 81 and 82 in the same block — read-version invalidation between the two is plausible.

Error site is exec3_parallel.go:2606 — validator-invalid retry exhaustion in the OCC scheduler. Same code class as #21590's targeted fixes:

• intra_block_state.go CreateAccount: defaults incSource, incVersion / balSource, balVersion to (StorageRead, UnknownVersion) instead of inheriting an outer refreshVersionedAccount-promoted (MapRead, V_subfield), which previously stamped IncarnationPath / BalancePath with a cell coordinate the versionMap never wrote — directly triggering "non-converging validator-invalid retry loop" per the in-code comment.
• versionedio.go versionedStateReader.ReadAccountData: matching AddressPath at TxIdx >= destructTxIndex check that mirrors IBS.
• intra_block_state.go accountRead: drops the (MapRead, V) promotion when vm.Read(AddressPath) returns None, to avoid the same non-convergence.

The TestCreateAccount_SyntheticIncarnationStamp_DoesNotInvalidate unit test in execution/state/versionedio_test.go pins the BalancePath-promotion variant of this. Without the specific reads tx 81 makes (depends on factory source) I can't be 100% sure the variant is identical, but the symptom is the same and the candidate fixes target precisely that surface.

Action: rebase / merge #21590 into your branch and rerun mainnet sync to 25.2M. If it still hits the retry budget, please grab the [5/5 Execution] rw exit stack on the new run and the preceding [5/5 Execution] lines around tx 80-82 — that will tell us whether it's the same stamping path or a different validator-invalid variant.

[mh0lt]

Explicitly checked on current main vs. an unpatched branch — this no longer reproduces on main, but still reproduces on a branch that predates the parallel-exec fix cluster (#21590).

main — does NOT reproduce ✅

• Commit b2329dc55b (current main, includes execution/state, stagedsync: parallel-exec SD-revival and metamorphic-CREATE2 fixes #21590).
• Fresh mainnet datadir, EXEC3_PARALLEL=true, embedded Caplin, serial commitment.
• Parallel-exec batch from=25203000 to=25203999 — which contains block 25203874 — executed and committed cleanly ([4/6 Execution] parallel done blk=25203999).
• Block 25203874 tx 81 converged normally (repeat%=17, ordinary Block-STM churn); zero too many validator-invalid retries, zero invalid-block, across the whole run.

unpatched branch — DOES reproduce ❌

• A branch based off BAL cache lineage (PR #6 of the perf stack) #20893 (merge-base with main = 2026-05-25, i.e. before execution/state, stagedsync: parallel-exec SD-revival and metamorphic-CREATE2 fixes #21590 merged on 2026-06-03).
• Same datadir/flags, parallel exec from the snapshot boundary.
• Deterministically hits the exact error from this issue:

  invalid block: could not apply tx 25203874:81 [0xed4738fe081d00a22996dfe8b20b59e72fecfd596dee51789f7197ce087ffcca]: too many validator-invalid retries: 251, expected: 250
  

Conclusion

#21590 (the SD-revival / metamorphic-CREATE2 validator-stamping fixes) resolves the 25203874:81 non-convergence. The earlier "reproduces on plain main" observation predated #21590. Suggest closing once a full sync past 25.2M on main confirms it end-to-end (sync currently running).

[taratorio]

confirmed fixed