[DO NOT MERGE] bench: switch foundry-capable scenarios to forge commands by Wodann · Pull Request #8324 · NomicFoundation/hardhat

Wodann · 2026-05-21T21:52:05Z

Because this PR includes a bug fix, relevant tests have been included.
Because this PR includes a new feature, the change was previously discussed on an Issue or with someone from the team.
I didn't do anything of this.

This is a draft PR, merely intended to run the performance regression benchmark suite for Foundry, to create a baseline for our performance comparisons.

⚠️ This should not be merged!

changeset-bot · 2026-05-21T21:52:11Z

⚠️ No Changeset found

Latest commit: 130fea1

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copilot

Pull request overview

This PR updates several end-to-end benchmark scenarios to run Foundry-native workloads by switching their benchmark commands from npx hardhat ... to forge ..., and adds scenario preinstall hooks to install Foundry before running benchmarks.

Changes:

Switch benchmark phases (cold/warm compile + tests) to forge build / forge test for Foundry-capable scenarios.
Add per-scenario preinstall-foundry.sh scripts and ensure PATH includes $HOME/.foundry/bin so forge is available to benchmark commands.
Introduce a shared end-to-end/_shared/foundry-install.sh helper to install Foundry via foundryup.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
end-to-end/uniswap-x/scenario.json	Adds Foundry PATH + preinstall; switches benchmark commands to `forge`.
end-to-end/uniswap-x/preinstall-foundry.sh	Installs Foundry via shared helper before existing scenario-specific setup.
end-to-end/uniswap-v4-core/scenario.json	Adds Foundry PATH + preinstall; switches benchmark commands to `forge`.
end-to-end/uniswap-v4-core/preinstall-foundry.sh	New preinstall script to install a pinned Foundry version.
end-to-end/lidofinance-dual-governance/scenario.json	Adds Foundry PATH + preinstall; switches benchmark commands to `forge` (including test filtering).
end-to-end/lidofinance-dual-governance/preinstall-foundry.sh	New preinstall script to install a pinned Foundry version.
end-to-end/ens-verifiable-factory/scenario.json	Adds Foundry PATH + preinstall; switches benchmark commands to `forge`.
end-to-end/ens-verifiable-factory/preinstall-foundry.sh	New preinstall script to install a pinned Foundry version.
end-to-end/aave-v4/scenario.json	Adds Foundry PATH + preinstall; switches benchmark commands to `forge`.
end-to-end/aave-v4/preinstall-foundry.sh	New preinstall script to install a pinned nightly Foundry build.
end-to-end/1inch-swap-vm/scenario.json	Adds Foundry PATH + preinstall; switches benchmark commands to `forge`.
end-to-end/1inch-swap-vm/preinstall-foundry.sh	New preinstall script to install a pinned Foundry version.
end-to-end/1inch-cross-chain-swap/scenario.json	Adds Foundry PATH + preinstall; switches benchmark commands to `forge`.
end-to-end/1inch-cross-chain-swap/preinstall-foundry.sh	New preinstall script to install a pinned Foundry version.
end-to-end/1inch-aqua/scenario.json	Adds Foundry PATH + preinstall; switches benchmark commands to `forge`.
end-to-end/1inch-aqua/preinstall-foundry.sh	New preinstall script to install a pinned Foundry version.
end-to-end/_shared/foundry-install.sh	New shared helper that installs Foundry via `foundryup` (currently using a remote installer script).

+  if ! command -v foundryup >/dev/null 2>&1; then
+    curl -L https://foundry.paradigm.xyz | bash
+  fi


Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

+  local version="$1"
+  local foundry_bin_dir="${FOUNDRY_DIR:-$HOME/.foundry}/bin"
+  local foundryup_path="$foundry_bin_dir/foundryup"
+


+  if [ ! -x "$foundryup_path" ]; then
+    mkdir -p "$foundry_bin_dir"
+    curl -sSfL \
+      "https://raw.githubusercontent.com/foundry-rs/foundry/HEAD/foundryup/foundryup" \
+      -o "$foundryup_path"
+    chmod +x "$foundryup_path"


Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

+    curl -sSfL \
+      "https://raw.githubusercontent.com/foundry-rs/foundry/HEAD/foundryup/foundryup" \
+      -o "$foundryup_path"


+    curl -sSfL \
+      "https://raw.githubusercontent.com/foundry-rs/foundry/HEAD/foundryup/foundryup" \
+      -o "$foundryup_path"
+    chmod +x "$foundryup_path"


Wodann · 2026-05-26T12:54:38Z

Closing this as we have generated the Foundry baseline. If we need to generate Foundry baseline numbers for additional scenarios, we can re-use this PR.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Wodann · 2026-06-12T15:32:30Z

Closing this as we have generated the Foundry baseline. If we need to generate Foundry baseline numbers for additional scenarios, we can re-use this PR.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

end-to-end/_shared/foundry-install.sh:33

foundryup is downloaded from a moving HEAD URL. That makes benchmark runs non-reproducible (the installer script can change between runs) and increases supply-chain risk for CI. Consider pinning this download to a specific commit or release ref and (ideally) verifying the downloaded script (checksum/signature) before executing it.

  if [ ! -x "$foundryup_path" ]; then
    mkdir -p "$foundry_bin_dir"
    curl -sSfL \
      "https://raw.githubusercontent.com/foundry-rs/foundry/HEAD/foundryup/foundryup" \
      -o "$foundryup_path"

@@ -44,6 +44,18 @@ export function installDependencies(
      ...process.env,
      ...env,
      COREPACK_ENABLE_DOWNLOAD_PROMPT: "0",
+      // actions/setup-node (with `registry-url`) writes $RUNNER_TEMP/.npmrc
+      // with an `_authToken` referencing ${NODE_AUTH_TOKEN}, and the CI clone
+      // dir ($RUNNER_TEMP/hardhat-e2e-clones) lives right under it. Yarn
+      // Classic registers every .npmrc in the cwd's ancestor directories and
+      // hard-fails on any unset variable in a config it reads (npm only
+      // warns). The variable is only set in publish jobs; setup-node ≤v6
+      // exported a placeholder that kept the substitution working, v7
+      // stopped, so provide the fallback here. The registry these installs
+      // actually use is unaffected: the Verdaccio config outranks any
+      // ambient npmrc.
+      NODE_AUTH_TOKEN:
+        env?.NODE_AUTH_TOKEN ?? process.env.NODE_AUTH_TOKEN ?? "",
      npm_config_minimum_release_age: "0",


    needs: tests
-    timeout-minutes: 180
+    timeout-minutes: 900
    steps:


Replace each Foundry-capable scenario's benchmark.commands with the forge equivalents so that pnpm bench:regression measures baseline Foundry performance on the same external repos and the same benchmark shape used for Hardhat regression. This mirrors main's per-scenario "compile sequence" (reset/clean -> cold compile -> edit & compile the min-deps test, max-deps test, and a contract) plus the warm-compile and test-solidity commands, keeping identical command/step names and run counts so the Hardhat and Foundry numbers line up entry-for-entry: npx hardhat clean -> forge clean npx hardhat compile -> forge build (cold, warm, and every edit & compile) hardhat test solidity --no-compile -> forge test ... --grep="^(...)" -> forge test --match-test "^(...)" (lidofinance) Each scenario now ships a preinstall-foundry.sh (sourcing the shared _shared/foundry-install.sh helper) that pins a specific Foundry release, and prepends $HOME/.foundry/bin to PATH via scenario.json env so forge is on PATH for the benchmark commands. aave-v4 keeps main's preinstall.sh fuzz-runs reduction and additionally installs Foundry there; uniswap-x's preinstall-foundry.sh also builds the calibur submodule and patches foundry.toml. defaultCommand is unchanged so pnpm e2e continues to run Hardhat. The four non-Foundry scenarios are not touched.

popescuoctavian · 2026-08-28T08:52:46Z

/bench

github-actions · 2026-08-28T08:52:57Z

🚀 Starting regression benchmark for 771ab69971de.

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

end-to-end/_shared/foundry-install.sh:33

Downloading the foundryup installer from the HEAD ref makes benchmark runs non-reproducible and creates a supply-chain risk (the script’s contents can change without any change to this repo). Since install_foundry already takes a specific version, fetch the installer from a stable ref (e.g. the same tag) instead of HEAD.

    curl -sSfL \
      "https://raw.githubusercontent.com/foundry-rs/foundry/HEAD/foundryup/foundryup" \
      -o "$foundryup_path"

    # against the configured alert-threshold.
    runs-on: hardhat-linux-amd64-self-hosted
-    timeout-minutes: 180
+    timeout-minutes: 900


github-actions · 2026-08-28T11:53:25Z

❌ Regression benchmark failed for 771ab69971de. This is either a detected performance regression or an infrastructure failure — see the run for details.

View workflow run

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Suppressed comments (2)

end-to-end/_shared/foundry-install.sh:34

The Foundry installer downloads and executes foundryup from the HEAD of the foundry repo. This makes benchmark runs non-reproducible and introduces a supply-chain risk (a change to HEAD can alter what gets executed even when scenarios pin a Foundry version). Consider pinning the raw URL to a specific commit/tag and (ideally) verifying a checksum before executing it.

  if [ ! -x "$foundryup_path" ]; then
    mkdir -p "$foundry_bin_dir"
    curl -sSfL \
      "https://raw.githubusercontent.com/foundry-rs/foundry/HEAD/foundryup/foundryup" \
      -o "$foundryup_path"
    chmod +x "$foundryup_path"

.github/workflows/regression-benchmark.yml:96

timeout-minutes was increased from 180 to 900 (15h). On a self-hosted runner this can tie up the machine for long periods if something hangs or run-counts increase further. Consider keeping a tighter timeout and/or splitting the Foundry-baseline run into a separate, explicitly long-running workflow so normal regression runs can fail fast.

    # Use a self-hosted runner for stable benchmark measurements; the shared
    # GitHub runners produce noisy results that will generate false positives
    # against the configured alert-threshold.
    runs-on: hardhat-linux-amd64-self-hosted
    timeout-minutes: 900
    permissions:

Copilot AI lite review requested due to automatic review settings May 21, 2026 21:52

Wodann added the ci:perf label May 21, 2026

Wodann self-assigned this May 21, 2026

Copilot started reviewing on behalf of Wodann May 21, 2026 21:52 View session

Copilot AI reviewed May 21, 2026

View reviewed changes

Comment thread end-to-end/_shared/foundry-install.sh Outdated

Comment on lines +15 to +17

if ! command -v foundryup >/dev/null 2>&1; then

curl -L https://foundry.paradigm.xyz | bash

fi

Wodann had a problem deploying to github-action-benchmark May 21, 2026 22:43 — with GitHub Actions Failure

Wodann force-pushed the test/bench-foundry-baseline branch from 7ca9d7d to 018c7cd Compare May 22, 2026 18:04

Copilot AI review requested due to automatic review settings May 23, 2026 02:36

Copilot started reviewing on behalf of Wodann May 23, 2026 02:36 View session

Copilot AI reviewed May 23, 2026

View reviewed changes

Wodann force-pushed the test/bench-foundry-baseline branch from 057c70c to 149bcde Compare May 23, 2026 06:02

Wodann added no changeset needed This PR doesn't require a changeset no docs needed This PR doesn't require links to documentation no peer bump needed labels May 23, 2026

Copilot AI review requested due to automatic review settings May 23, 2026 16:32

Copilot started reviewing on behalf of Wodann May 23, 2026 16:33 View session

Copilot AI reviewed May 23, 2026

View reviewed changes

Wodann changed the title ~~{DO NOT MERGE] bench: switch foundry-capable scenarios to forge commands~~ [DO NOT MERGE] bench: switch foundry-capable scenarios to forge commands May 23, 2026

Wodann force-pushed the test/bench-foundry-baseline branch from 8493bcb to 570d091 Compare May 24, 2026 04:27

Wodann temporarily deployed to github-action-benchmark May 24, 2026 04:34 — with GitHub Actions Inactive

Wodann temporarily deployed to github-action-benchmark May 24, 2026 15:37 — with GitHub Actions Inactive

Wodann had a problem deploying to github-action-benchmark May 25, 2026 02:42 — with GitHub Actions Failure

Wodann had a problem deploying to github-action-benchmark May 25, 2026 13:43 — with GitHub Actions Failure

Wodann temporarily deployed to github-action-benchmark May 25, 2026 15:08 — with GitHub Actions Inactive

Wodann force-pushed the test/bench-foundry-baseline branch from 570d091 to 68d3bae Compare May 25, 2026 17:10

Wodann temporarily deployed to github-action-benchmark May 25, 2026 17:17 — with GitHub Actions Inactive

Wodann closed this May 26, 2026

Wodann reopened this Jun 11, 2026

Copilot AI review requested due to automatic review settings June 11, 2026 21:11

Wodann force-pushed the test/bench-foundry-baseline branch from 68d3bae to d948195 Compare June 11, 2026 21:11

Copilot started reviewing on behalf of Wodann June 11, 2026 21:12 View session

Wodann had a problem deploying to github-action-benchmark June 11, 2026 21:19 — with GitHub Actions Error

Copilot AI reviewed Jun 11, 2026

Wodann had a problem deploying to github-action-benchmark June 12, 2026 04:07 — with GitHub Actions Failure

Wodann closed this Jun 12, 2026

popescuoctavian reopened this Aug 28, 2026

Copilot AI review requested due to automatic review settings August 28, 2026 08:38

Copilot started reviewing on behalf of popescuoctavian August 28, 2026 08:39 View session

Copilot AI reviewed Aug 28, 2026

View reviewed changes

Wodann and others added 3 commits August 28, 2026 08:46

ci: increase timeout temporarily

6075632

bench: update aave-v4

771ab69

popescuoctavian force-pushed the test/bench-foundry-baseline branch from dff1b3f to 771ab69 Compare August 28, 2026 08:49

Copilot AI review requested due to automatic review settings August 28, 2026 08:49

Copilot started reviewing on behalf of popescuoctavian August 28, 2026 08:49 View session

Copilot AI reviewed Aug 28, 2026

View reviewed changes

Comment thread .github/workflows/regression-benchmark.yml

# against the configured alert-threshold.

runs-on: hardhat-linux-amd64-self-hosted

timeout-minutes: 180

timeout-minutes: 900

popescuoctavian had a problem deploying to github-action-benchmark August 28, 2026 17:15 — with GitHub Actions Failure

always upload regression report

130fea1

popescuoctavian had a problem deploying to github-action-benchmark August 31, 2026 07:19 — with GitHub Actions Error

Copilot AI review requested due to automatic review settings August 31, 2026 07:44

popescuoctavian had a problem deploying to github-action-benchmark August 31, 2026 07:45 — with GitHub Actions Failure

Copilot started reviewing on behalf of popescuoctavian August 31, 2026 07:45 View session

Copilot AI reviewed Aug 31, 2026

View reviewed changes

Navigation Menu

Uh oh!

[DO NOT MERGE] bench: switch foundry-capable scenarios to forge commands - #8324

[DO NOT MERGE] bench: switch foundry-capable scenarios to forge commands#8324
Wodann wants to merge 4 commits into
mainfrom
test/bench-foundry-baseline

Wodann commented May 21, 2026

Uh oh!

changeset-bot Bot commented May 21, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Wodann commented May 26, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Wodann commented Jun 12, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

popescuoctavian commented Aug 28, 2026

Uh oh!

github-actions Bot commented Aug 28, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Aug 28, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Wodann commented May 21, 2026

Uh oh!

changeset-bot Bot commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Wodann commented May 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Wodann commented Jun 12, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

popescuoctavian commented Aug 28, 2026

Uh oh!

github-actions Bot commented Aug 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented Aug 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

changeset-bot Bot commented May 21, 2026 •

edited

Loading