Skip to content

ci, db/state: matrix-test serial vs parallel commitment across the test workflows #22141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mh0lt wants to merge 12 commits into
base: main
Choose a base branch
from
+271 −215
Open

ci, db/state: matrix-test serial vs parallel commitment across the test workflows #22141

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
4d1d24e
ci, db/state: matrix-test serial vs parallel commitment across the te…
mh0lt Jul 1, 2026
536fc68
Merge branch 'main' into mh/ci-commitment-mode-matrix
mh0lt Jul 2, 2026
f1633dc
Merge branch 'main' into mh/ci-commitment-mode-matrix
mh0lt Jul 2, 2026
f86ceab
ci: switch EEST spec shards from exec-parallel to commitment-parallel
mh0lt Jul 2, 2026
e7308d8
Merge branch 'main' into mh/ci-commitment-mode-matrix
AskAlexSharov Jul 3, 2026
f6b548f
Merge branch 'main' into mh/ci-commitment-mode-matrix
mh0lt Jul 3, 2026
c923faa
ci: default suffix-less EEST devnet shards to serial commitment
mh0lt Jul 3, 2026
aa048ec
Merge branch 'main' into mh/ci-commitment-mode-matrix
mh0lt Jul 4, 2026
0cbedfc
Merge remote-tracking branch 'origin/main' into mh/ci-commitment-mode…
awskii Jul 6, 2026
d0a9774
Merge branch 'main' into mh/ci-commitment-mode-matrix
AskAlexSharov Jul 7, 2026
3a60ced
Merge branch 'main' into mh/ci-commitment-mode-matrix
awskii Jul 7, 2026
1436a65
execution/commitment: defer parallel trie selection to EnableParaTrieDB
awskii Jul 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions .github/workflows/qa-rpc-integration-tests-latest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:

jobs:
mainnet-rpc-integ-tests-latest:
name: mainnet-rpc-integ-tests-latest (${{ matrix.exec_mode }})
name: mainnet-rpc-integ-tests-latest (${{ matrix.commitment_mode }})
concurrency:
group: >-
${{
Expand All @@ -42,7 +42,7 @@ jobs:
# just one after the other (~2× wall-clock).
max-parallel: 1
matrix:
exec_mode:
commitment_mode:
- serial
- parallel
env:
Expand All @@ -53,10 +53,10 @@ jobs:
ERIGON_ASSERT: true
RPC_PAST_TEST_DIR: /opt/rpc-past-tests
CHAIN: mainnet
# Toggle dbg.Exec3Parallel from CI without code changes. envLookup
# in common/dbg/dbg_env.go auto-prepends ERIGON_, so this maps to
# the EXEC3_PARALLEL flag declared in common/dbg/experiments.go.
ERIGON_EXEC3_PARALLEL: ${{ matrix.exec_mode == 'parallel' && 'true' || 'false' }}
# Toggle statecfg.ExperimentalParallelCommitment from CI without code
# changes. envLookup in common/dbg/dbg_env.go auto-prepends ERIGON_, so
# this maps to the COMMITMENT_PARALLEL flag read in db/state/statecfg.
ERIGON_COMMITMENT_PARALLEL: ${{ matrix.commitment_mode == 'parallel' && 'true' || 'false' }}

steps:
- name: Check out repository
Expand Down Expand Up @@ -123,7 +123,7 @@ jobs:
if: failure() && steps.preparing_step.outcome == 'failure'
uses: actions/upload-artifact@v7
with:
name: preparing-step-logs-${{ matrix.exec_mode }}
name: preparing-step-logs-${{ matrix.commitment_mode }}
path: ${{ env.ERIGON_REFERENCE_DATA_DIR }}/logs/

- name: Pause the Erigon instance dedicated to db maintenance
Expand Down Expand Up @@ -180,7 +180,7 @@ jobs:
if: failure() && steps.pre_test_step.outcome == 'failure'
uses: actions/upload-artifact@v7
with:
name: pre-test-logs-${{ matrix.exec_mode }}
name: pre-test-logs-${{ matrix.commitment_mode }}
path: ${{ env.ERIGON_TESTBED_DATA_DIR }}/logs/

- name: Run RPC Integration Tests
Expand Down Expand Up @@ -260,7 +260,7 @@ jobs:
if: always() && steps.test_step.outputs.test_executed == 'true'
uses: actions/upload-artifact@v7
with:
name: test-results-${{ matrix.exec_mode }}
name: test-results-${{ matrix.commitment_mode }}
path: |
${{ env.TEST_RESULT_DIR }}
${{ env.ERIGON_TESTBED_DATA_DIR }}/logs/
Expand All @@ -280,7 +280,7 @@ jobs:
--repo erigon \
--commit $(git rev-parse HEAD) \
--branch ${{ github.ref_name }} \
--test_name rpc-integration-tests-latest${{ matrix.exec_mode == 'parallel' && '-parallel' || '' }} \
--test_name rpc-integration-tests-latest${{ matrix.commitment_mode == 'parallel' && '-parallel' || '' }} \
--chain $CHAIN \
--runner ${{ runner.name }} \
--db_version $db_version \
Expand Down
26 changes: 13 additions & 13 deletions .github/workflows/qa-rpc-performance-comparison-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ on:
type: boolean
required: false
default: false
exec_mode:
description: 'Erigon execution mode ("default" leaves ERIGON_EXEC3_PARALLEL unset)'
commitment_mode:
description: 'Erigon commitment mode ("default" leaves ERIGON_COMMITMENT_PARALLEL unset)'
type: choice
required: false
default: default
Expand Down Expand Up @@ -194,22 +194,22 @@ jobs:
id: erigon_running_step
working-directory: ${{ github.workspace }}/build/bin
env:
EXEC_MODE: ${{ github.event.inputs.exec_mode }}
EXEC_MODE: ${{ github.event.inputs.commitment_mode }}
run: |
set +e # Disable exit on error
echo "Starting Erigon..."

# Only set ERIGON_EXEC3_PARALLEL on an explicit choice; otherwise ensure it's
# Only set ERIGON_COMMITMENT_PARALLEL on an explicit choice; otherwise ensure it's
# unset so erigon uses its built-in default (and doesn't inherit runner env).
if [ "$EXEC_MODE" = "parallel" ]; then
export ERIGON_EXEC3_PARALLEL=true
echo "Set ERIGON_EXEC3_PARALLEL=true (parallel mode)"
export ERIGON_COMMITMENT_PARALLEL=true
echo "Set ERIGON_COMMITMENT_PARALLEL=true (parallel mode)"
elif [ "$EXEC_MODE" = "serial" ]; then
export ERIGON_EXEC3_PARALLEL=false
echo "Set ERIGON_EXEC3_PARALLEL=false (serial mode)"
export ERIGON_COMMITMENT_PARALLEL=false
echo "Set ERIGON_COMMITMENT_PARALLEL=false (serial mode)"
else
unset ERIGON_EXEC3_PARALLEL
echo "Leaving ERIGON_EXEC3_PARALLEL unset (default behavior)"
unset ERIGON_COMMITMENT_PARALLEL
echo "Leaving ERIGON_COMMITMENT_PARALLEL unset (default behavior)"
fi

./erigon --prune.mode=minimal --datadir $ERIGON_TESTBED_DATA_DIR --http.api admin,debug,eth,parity,erigon,trace,web3,txpool,ots,net --ws > erigon.log 2>&1 &
Expand Down Expand Up @@ -378,7 +378,7 @@ jobs:
--repo $client \
--branch $branch_name \
--commit $commit_hash \
--test_name rpc-performance-test-latest${{ (matrix.client == 'erigon' && github.event.inputs.exec_mode == 'parallel' && '-parallel') || (matrix.client == 'erigon' && github.event.inputs.exec_mode == 'serial' && '-serial') || '' }}-$method \
--test_name rpc-performance-test-latest${{ (matrix.client == 'erigon' && github.event.inputs.commitment_mode == 'parallel' && '-parallel') || (matrix.client == 'erigon' && github.event.inputs.commitment_mode == 'serial' && '-serial') || '' }}-$method \
--chain $CHAIN \
--runner ${{ runner.name }} \
--db_version $db_version \
Expand Down Expand Up @@ -451,7 +451,7 @@ jobs:
if: (matrix.client == 'erigon' || needs.setup.outputs.run_geth == 'true') && steps.test_step.outputs.test_executed == 'true'
uses: actions/upload-artifact@v7
with:
name: test-results-${{ env.CHAIN }}-${{ matrix.client }}${{ (matrix.client == 'erigon' && github.event.inputs.exec_mode == 'parallel' && '-parallel') || (matrix.client == 'erigon' && github.event.inputs.exec_mode == 'serial' && '-serial') || '' }}
name: test-results-${{ env.CHAIN }}-${{ matrix.client }}${{ (matrix.client == 'erigon' && github.event.inputs.commitment_mode == 'parallel' && '-parallel') || (matrix.client == 'erigon' && github.event.inputs.commitment_mode == 'serial' && '-serial') || '' }}
path: ${{ env.past_test_dir }}

- name: Stop Erigon
Expand All @@ -471,7 +471,7 @@ jobs:
if: matrix.client == 'erigon' && steps.test_step.outputs.test_executed == 'true'
uses: actions/upload-artifact@v7
with:
name: erigon-logs-${{ env.CHAIN }}${{ (github.event.inputs.exec_mode == 'parallel' && '-parallel') || (github.event.inputs.exec_mode == 'serial' && '-serial') || '' }}
name: erigon-logs-${{ env.CHAIN }}${{ (github.event.inputs.commitment_mode == 'parallel' && '-parallel') || (github.event.inputs.commitment_mode == 'serial' && '-serial') || '' }}
path: ${{ github.workspace }}/build/bin/erigon.log

- name: Delete Erigon Testbed Data Directory
Expand Down
26 changes: 13 additions & 13 deletions .github/workflows/qa-rpc-performance-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ on:
type: boolean
required: false
default: false
exec_mode:
description: 'Erigon execution mode ("default" leaves ERIGON_EXEC3_PARALLEL unset)'
commitment_mode:
description: 'Erigon commitment mode ("default" leaves ERIGON_COMMITMENT_PARALLEL unset)'
type: choice
required: false
default: default
Expand Down Expand Up @@ -208,21 +208,21 @@ jobs:
if: matrix.client == 'erigon'
working-directory: ${{ github.workspace }}/build/bin
env:
EXEC_MODE: ${{ github.event.inputs.exec_mode }}
EXEC_MODE: ${{ github.event.inputs.commitment_mode }}
run: |
echo "Starting RpcDaemon..."

# Only set ERIGON_EXEC3_PARALLEL on an explicit choice; otherwise ensure it's
# Only set ERIGON_COMMITMENT_PARALLEL on an explicit choice; otherwise ensure it's
# unset so erigon uses its built-in default (and doesn't inherit runner env).
if [ "$EXEC_MODE" = "parallel" ]; then
export ERIGON_EXEC3_PARALLEL=true
echo "Set ERIGON_EXEC3_PARALLEL=true (parallel mode)"
export ERIGON_COMMITMENT_PARALLEL=true
echo "Set ERIGON_COMMITMENT_PARALLEL=true (parallel mode)"
elif [ "$EXEC_MODE" = "serial" ]; then
export ERIGON_EXEC3_PARALLEL=false
echo "Set ERIGON_EXEC3_PARALLEL=false (serial mode)"
export ERIGON_COMMITMENT_PARALLEL=false
echo "Set ERIGON_COMMITMENT_PARALLEL=false (serial mode)"
else
unset ERIGON_EXEC3_PARALLEL
echo "Leaving ERIGON_EXEC3_PARALLEL unset (default behavior)"
unset ERIGON_COMMITMENT_PARALLEL
echo "Leaving ERIGON_COMMITMENT_PARALLEL unset (default behavior)"
fi

./rpcdaemon --datadir $ERIGON_REFERENCE_DATA_DIR --http.api admin,debug,eth,parity,erigon,trace,web3,txpool,ots,net > erigon.log 2>&1 &
Expand Down Expand Up @@ -371,7 +371,7 @@ jobs:
--repo $client \
--branch $branch_name \
--commit $commit_hash \
--test_name rpc-performance-test-$client${{ (matrix.client == 'erigon' && github.event.inputs.exec_mode == 'parallel' && '-parallel') || (matrix.client == 'erigon' && github.event.inputs.exec_mode == 'serial' && '-serial') || '' }}-$method \
--test_name rpc-performance-test-$client${{ (matrix.client == 'erigon' && github.event.inputs.commitment_mode == 'parallel' && '-parallel') || (matrix.client == 'erigon' && github.event.inputs.commitment_mode == 'serial' && '-serial') || '' }}-$method \
--chain $CHAIN \
--runner ${{ runner.name }} \
--db_version $db_version \
Expand Down Expand Up @@ -434,7 +434,7 @@ jobs:
if: always() && matrix.client == 'erigon' && steps.rpcdaemon_running_step.outputs.rpc_daemon_started == 'true'
uses: actions/upload-artifact@v7
with:
name: rpcdaemon-logs${{ (github.event.inputs.exec_mode == 'parallel' && '-parallel') || (github.event.inputs.exec_mode == 'serial' && '-serial') || '' }}
name: rpcdaemon-logs${{ (github.event.inputs.commitment_mode == 'parallel' && '-parallel') || (github.event.inputs.commitment_mode == 'serial' && '-serial') || '' }}
path: ${{ github.workspace }}/build/bin/erigon.log

- name: Restore Erigon Chaindata Directory
Expand Down Expand Up @@ -470,7 +470,7 @@ jobs:
if: (matrix.client == 'erigon' || needs.setup.outputs.run_geth == 'true') && steps.test_step.outputs.test_executed == 'true'
uses: actions/upload-artifact@v7
with:
name: test-results-${{ env.CHAIN }}-${{ matrix.client }}${{ (matrix.client == 'erigon' && github.event.inputs.exec_mode == 'parallel' && '-parallel') || (matrix.client == 'erigon' && github.event.inputs.exec_mode == 'serial' && '-serial') || '' }}
name: test-results-${{ env.CHAIN }}-${{ matrix.client }}${{ (matrix.client == 'erigon' && github.event.inputs.commitment_mode == 'parallel' && '-parallel') || (matrix.client == 'erigon' && github.event.inputs.commitment_mode == 'serial' && '-serial') || '' }}
path: ${{ env.past_test_dir }}

- name: Action to check failure condition
Expand Down
38 changes: 19 additions & 19 deletions .github/workflows/qa-stage-exec.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,48 +17,48 @@ concurrency:

jobs:
stage-exec-test:
name: stage-exec-test (${{ matrix.mode_name }}, ${{ matrix.exec_mode }})
name: stage-exec-test (${{ matrix.mode_name }}, ${{ matrix.commitment_mode }})
runs-on: [self-hosted, qa, Ethereum, tip-tracking]
strategy:
fail-fast: false
matrix:
# Each (mode_name, exec_mode) pair runs separately. The testbed
# Each (mode_name, commitment_mode) pair runs separately. The testbed
# data dir already disambiguates by mode_name; we extend it with
# exec_mode so serial+parallel entries don't clobber each other.
# commitment_mode so serial+parallel entries don't clobber each other.
include:
- mode_name: resume-nonchaintip
extra_flags: ""
test_name: stage_exec_resume_nonchaintip
exec_mode: serial
commitment_mode: serial
- mode_name: resume-nonchaintip
extra_flags: ""
test_name: stage_exec_resume_nonchaintip
exec_mode: parallel
commitment_mode: parallel
- mode_name: from-0
extra_flags: "--rm-state-all"
test_name: stage_exec_from_0
exec_mode: serial
commitment_mode: serial
- mode_name: from-0
extra_flags: "--rm-state-all"
test_name: stage_exec_from_0
exec_mode: parallel
commitment_mode: parallel
- mode_name: chaintip
extra_flags: "--chaintip"
test_name: stage_exec_resume_chaintip
exec_mode: serial
commitment_mode: serial
- mode_name: chaintip
extra_flags: "--chaintip"
test_name: stage_exec_resume_chaintip
exec_mode: parallel
commitment_mode: parallel
env:
ERIGON_TESTBED_DATA_DIR: /opt/erigon-testbed/datadir-${{ matrix.mode_name }}-${{ matrix.exec_mode }}
ERIGON_TESTBED_DATA_DIR: /opt/erigon-testbed/datadir-${{ matrix.mode_name }}-${{ matrix.commitment_mode }}
ERIGON_QA_PATH: /home/qarunner/erigon-qa
TIMEOUT_SECONDS: 360
CHAIN: mainnet
# Toggle dbg.Exec3Parallel from CI without code changes. envLookup
# in common/dbg/dbg_env.go auto-prepends ERIGON_, so this maps to
# the EXEC3_PARALLEL flag declared in common/dbg/experiments.go.
ERIGON_EXEC3_PARALLEL: ${{ matrix.exec_mode == 'parallel' && 'true' || 'false' }}
# Toggle statecfg.ExperimentalParallelCommitment from CI without code
# changes. envLookup in common/dbg/dbg_env.go auto-prepends ERIGON_, so
# this maps to the COMMITMENT_PARALLEL flag read in db/state/statecfg.
ERIGON_COMMITMENT_PARALLEL: ${{ matrix.commitment_mode == 'parallel' && 'true' || 'false' }}

steps:
- name: Check out repository
Expand Down Expand Up @@ -114,7 +114,7 @@ jobs:

python3 $ERIGON_QA_PATH/test_system/qa-tests/stage-exec/run_and_check_stage_exec.py \
${{ github.workspace }}/build/bin $ERIGON_TESTBED_DATA_DIR $TIMEOUT_SECONDS $CHAIN \
--result-file ${{ github.workspace }}/result-$CHAIN-${{ matrix.mode_name }}-${{ matrix.exec_mode }}.json \
--result-file ${{ github.workspace }}/result-$CHAIN-${{ matrix.mode_name }}-${{ matrix.commitment_mode }}.json \
${{ matrix.extra_flags }}

test_exit_status=$?
Expand Down Expand Up @@ -148,20 +148,20 @@ jobs:
--runner ${{ runner.name }} \
--db_version $db_version \
--outcome $TEST_RESULT \
--result_file ${{ github.workspace }}/result-$CHAIN-${{ matrix.mode_name }}-${{ matrix.exec_mode }}.json
--result_file ${{ github.workspace }}/result-$CHAIN-${{ matrix.mode_name }}-${{ matrix.commitment_mode }}.json

- name: Upload test results
if: ${{ always() && steps.test_step.outputs.test_executed == 'true' }}
uses: actions/upload-artifact@v7
with:
name: test-results-${{ matrix.mode_name }}-${{ matrix.exec_mode }}
path: ${{ github.workspace }}/result-${{ env.CHAIN }}-${{ matrix.mode_name }}-${{ matrix.exec_mode }}.json
name: test-results-${{ matrix.mode_name }}-${{ matrix.commitment_mode }}
path: ${{ github.workspace }}/result-${{ env.CHAIN }}-${{ matrix.mode_name }}-${{ matrix.commitment_mode }}.json

- name: Upload erigon logs
if: ${{ always() && steps.test_step.outputs.test_executed == 'true' }}
uses: actions/upload-artifact@v7
with:
name: erigon-logs-${{ matrix.mode_name }}-${{ matrix.exec_mode }}
name: erigon-logs-${{ matrix.mode_name }}-${{ matrix.commitment_mode }}
path: ${{ env.ERIGON_TESTBED_DATA_DIR }}/logs/

- name: Delete Erigon Testbed Data Directory
Expand Down
30 changes: 15 additions & 15 deletions .github/workflows/qa-txpool-performance-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,23 +18,23 @@ on:

jobs:
tx_pool_assertoor_test:
name: tx_pool_assertoor_test (${{ matrix.exec_mode }})
name: tx_pool_assertoor_test (${{ matrix.commitment_mode }})
runs-on: [self-hosted, qa, X64, long-running]
strategy:
fail-fast: false
# Self-hosted long-running pool is small; matrix entries serialize.
max-parallel: 1
matrix:
exec_mode:
commitment_mode:
- serial
- parallel
env:
ERIGON_QA_PATH: /home/qarunner/erigon-qa
ENCLAVE_NAME: "kurtosis-run-${{ github.run_id }}-${{ matrix.exec_mode }}"
# Toggle dbg.Exec3Parallel from CI without code changes. envLookup
# in common/dbg/dbg_env.go auto-prepends ERIGON_, so this maps to
# the EXEC3_PARALLEL flag declared in common/dbg/experiments.go.
ERIGON_EXEC3_PARALLEL: ${{ matrix.exec_mode == 'parallel' && 'true' || 'false' }}
ENCLAVE_NAME: "kurtosis-run-${{ github.run_id }}-${{ matrix.commitment_mode }}"
# Toggle statecfg.ExperimentalParallelCommitment from CI without code
# changes. envLookup in common/dbg/dbg_env.go auto-prepends ERIGON_, so
# this maps to the COMMITMENT_PARALLEL flag read in db/state/statecfg.
ERIGON_COMMITMENT_PARALLEL: ${{ matrix.commitment_mode == 'parallel' && 'true' || 'false' }}

steps:
- name: Fast checkout git repository
Expand All @@ -50,14 +50,14 @@ jobs:
run: |
docker build -t test/erigon:current-base .

- name: Bake exec_mode env into the runtime image
# Add ERIGON_EXEC3_PARALLEL as an ENV layer on top of the base image
# so the matrix entries differ in exec mode. ENV layers are cheap and
# don't invalidate earlier layer caches.
- name: Bake commitment_mode env into the runtime image
# Add ERIGON_COMMITMENT_PARALLEL as an ENV layer on top of the base
# image so the matrix entries differ in commitment mode. ENV layers are
# cheap and don't invalidate earlier layer caches.
run: |
docker build -t test/erigon:current - <<EOF
FROM test/erigon:current-base
ENV ERIGON_EXEC3_PARALLEL=${ERIGON_EXEC3_PARALLEL}
ENV ERIGON_COMMITMENT_PARALLEL=${ERIGON_COMMITMENT_PARALLEL}
EOF

- name: Run self hosted Kurtosis + assertoor tests
Expand Down Expand Up @@ -147,7 +147,7 @@ jobs:
id: upload_outputs_kurtosis_json
uses: actions/upload-artifact@v7
with:
name: outputs_kurtosis-${{ matrix.exec_mode }}.json
name: outputs_kurtosis-${{ matrix.commitment_mode }}.json
path: outputs_kurtosis.json

- name: Create HDR plots PNG artifacts
Expand Down Expand Up @@ -254,7 +254,7 @@ jobs:
id: upload_latency_plots
uses: actions/upload-artifact@v7
with:
name: tx-pool-latency-plots-${{ matrix.exec_mode }}
name: tx-pool-latency-plots-${{ matrix.commitment_mode }}
path: tx_pool_latency_analysis_*.png
if-no-files-found: ignore

Expand All @@ -263,7 +263,7 @@ jobs:
id: upload_throughput_plots
uses: actions/upload-artifact@v7
with:
name: tx-pool-throughput-plots-${{ matrix.exec_mode }}
name: tx-pool-throughput-plots-${{ matrix.commitment_mode }}
path: tx_pool_throughput_analysis_*.png
if-no-files-found: ignore

Expand Down
Loading
Loading