perf: reduce call-trace memory retention in Solidity test runner - #1612
Draft
Wodann wants to merge 15 commits into
Draft
perf: reduce call-trace memory retention in Solidity test runner#1612Wodann wants to merge 15 commits into
Wodann wants to merge 15 commits into
Conversation
Replace the per-request `spawn_blocking` + blocking channel round trip with a completion callback: `handle_request` now deserializes the request on the calling thread, enqueues it on the provider's background thread, and settles the napi deferred from that thread once the response is available. This cuts the thread handoffs per request from four to two (~7x fewer futex syscalls), making the provider benchmark suite ~2x faster than main. Queued requests are settled with the new `ProviderError::UnexpectedTermination` when the provider shuts down, so pending promises always resolve.
…bosity Runs each (repo, verbosity) pair in a fresh child process, records peak RSS via /usr/bin/time -v cross-checked with process.resourceUsage(), and reports a markdown table. Repo setup now also initializes submodules pinned by the benchmark commit and tolerates tool-dependent npm lifecycle scripts (morpho-blue's prepare step requires forge). Children are capped at 4 rayon threads: the unoptimized runner OOMs a 16 GiB machine at verbosity >= 3 on solady with unbounded parallelism, which would leave nothing to compare optimisations against.
Previously the failing arena was implicit: get_stack_trace folded over an
iterator and converted whichever arena came last, with a doc comment
('assumes last trace is the error one') carrying the whole contract.
Every other arena was walked only to seed the creation/runtime code maps.
Make that split explicit in the signature:
- failing_trace: the arena that is converted and analysed,
- code_sources: arenas walked only for their CREATE nodes, which
therefore never need recorded EVM steps,
- executed_code: pre-computed creation/runtime code mappings, seeded
before the arenas are walked (replaces the runtime-code-only third
parameter; the provider now passes ExecutedCode { runtime: Some(..) }).
The stronger invariant - only the failing arena's steps are ever read -
is what allows later commits to strip steps from retained arenas.
No behavioural change.
Previously every test's SparsedTraceArenas stayed resident in TestResult.execution_traces until the entire suite completed and the napi conversion filtered them - regardless of include_traces. With CollectStackTraces::Always (Hardhat -vvv and above) each arena carries per-opcode EVM steps, which is the difference between megabytes and gigabytes: solady at -vvv peaks at 4.7 GiB and at -vvvv OOMs a 16 GiB machine. Introduce TraceRetentionPolicy, applied at the single point where each finished test's result is produced (after stack-trace generation, the only consumer of EVM steps). It frees: - execution_traces, when include_traces will never surface them (always at None; for passing tests at Failing), - counterexample arenas, which have no consumer at all after the test finishes (not decoded, not exposed over napi), - gas_report_traces, when no gas report was requested (also fixes the leak where the decode loop's continue skipped the clear for passing tests at IncludeTraces::Failing).
Recorded steps (one entry per executed opcode) are only consumed by stack-trace generation, which has already run by the time a test finishes; the remaining consumers - trace decoding, the gas report and the napi conversion - only read the call tree, logs and their ordering (the napi conversion explicitly discards TraceMemberOrder::Step). Add SparsedTraceArena::strip_steps, resolving ignored ranges first because their bookkeeping refers to step indices, and apply it to every arena the retention policy keeps. This is the fix that matters at IncludeTraces::All (Hardhat -vvvv), where every test's arenas survive until the suite completes.
Only the most recent arena in a collection can still be named as the failing trace of a stack-trace computation; every earlier one is only walked for its CREATE nodes, which don't involve steps. Roll the steps off on push, so at most one step-laden arena is alive per test at any time - this bounds the transient peak within a test (beforeTestSetup chains, invariant replay sequences) rather than waiting for the retention policy at test end.
InvariantTestRun::run_traces is only ever consumed by end_run, which samples it into gas_report_traces while fewer than gas_report_samples entries exist and discards it otherwise. Apply the same condition at collection time: without a gas report (gas_report_samples = 0, as the napi layer configures) an invariant run no longer accumulates up to 'depth' arenas per run only to throw them away.
A passing invariant test replays its last run purely to collect logs and identify created contracts, but replay_run also pushed one arena per replayed call (up to 'depth', plus the invariant and afterInvariant calls) into TestResult.execution_traces. Those arenas are only ever surfaced as call traces, so gate the pushes on the retention policy; they are still kept whenever stack-trace generation needs them as code sources (generate_stack_trace). The per-call failure stack trace inside replay_run is now only computed when the caller asked for one - the success-path caller discards the ReplayResult, and without retained arenas the computation would have nothing to work from.
The napi conversion cloned the suite's setup arenas into every included test result, so a suite with N tests at IncludeTraces::All materialized N copies. Materialize the surfaced subset once per suite (moving it out of the suite result instead of cloning) and share it as Arc<[_]>; the call_traces() output is unchanged.
…up arenas Stack-trace decoding only needs the setup arenas for the creation and runtime code of the contracts deployed during setup. Capture that into ExecutedCodeMaps as setup traces are recorded, and use it everywhere the setup arenas were previously walked as code sources: - collect_stack_trace seeds get_stack_trace with the map instead of chaining the setup arenas; - the invariant replay paths take ExecutedCode instead of borrowing the setup traces; - the re-run-on-failure helpers no longer re-trace setup when the original setup was traced. When it wasn't (OnFailure with IncludeTraces::None, Hardhat's default, traces nothing during setup), they keep re-tracing to rebuild the mapping - covered by a new regression test that fails without the conditional. With the arenas' last in-run consumer gone, a suite's setup traces are dropped at suite end when include_traces is None instead of surviving until the napi conversion.
IncludeCallTraces controls whether call trace arenas are included in test results, mirroring the provider's includeCallTraces observability option; CollectStackTraces separately controls whether a stack trace is computed for failing tests. The old name suggested it governed trace collection as a whole. BREAKING: the napi enum is now IncludeCallTraces and the SolidityTestRunnerConfigArgs property is now includeCallTraces. The in-repo Hardhat patch is updated accordingly; the same rename is needed in Hardhat when it adopts this EDR version. The benchmark keeps a temporary shim mapping the old property until then.
🦋 Changeset detectedLatest commit: a178475 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Wodann
temporarily deployed
to
github-action-benchmark
August 11, 2026 14:42 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
Wodann
changed the title
Wodann
temporarily deployed
to
github-action-benchmark
August 11, 2026 15:38 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Wodann
temporarily deployed
to
github-action-benchmark
August 11, 2026 15:38 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Base automatically changed from
refactor/provider-on-background-thread
to
main
August 31, 2026 20:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Hardhat 3 runs out of memory in EDR at verbosity
-vvvand above. Hardhat maps verbosity to two EDR options:-vvv→collectStackTraces: Always+includeTraces: Failing,-vvvv+ →Always+All(default-vvisOnFailure+None). InAlwaysmode the runner records EVM steps (one entry per executed opcode, ~150–200 bytes each) for every call of every test, and every resulting trace arena stayed resident inTestResult.execution_tracesuntil the entire test suite finished — regardless of whether anything would ever consume it. On a 16 GiB machine, solady at-vvvvneeds over 10 GiB and gets OOM-killed; with unbounded parallelism it already dies at-vvv.The fix is a per-test retention policy: the moment a test finishes (and its stack trace, the only consumer of EVM steps, has been computed), every arena — or part of one — that has no remaining consumer is freed.
Benchmark methodology
New
pnpm soltestsMemorybenchmark (first commit): runs each (repo × verbosity) cell in a fresh child process and records peak RSS via/usr/bin/time -v, cross-checked againstprocess.resourceUsage(). Children are capped at 4 rayon threads because the unoptimized baseline OOMs the 16 GiB benchmark machine with unbounded parallelism, which would leave nothing to compare against. Repos: solady (1557 tests, fuzz-heavy), uniswap-v4-core (598 tests), morpho-blue (145 tests incl. 6 invariant functions, runs=16 × depth=256). All benchmark tests pass, which matters for reading the tables:-vvv(Failing, nothing surfaced) exercises the "drop everything" path,-vvvv(All) the "keep call traces" path.Each commit was benchmarked stacked on the previous ones (the later commits depend on the earlier ones), so the Δ columns below are vs. baseline including all prior commits; a commit's own contribution is the change relative to the previous table. Run-to-run noise on step-recording cells is ±10–15 %, so single-digit percentages are not meaningful.
Baseline (peak RSS / wall-clock)
-vv-vvv-vvvvPer-commit changes and results
refactor: name the failing arena explicitly in get_stack_tracePreviously
get_stack_tracefolded over an arena iterator and converted whichever came last ("assumes last trace is the error one"); every earlier arena was walked only for CREATE nodes to map contract addresses to code. The signature now says this:failing_trace(converted and analysed) vs.code_sources(walked for contract code only, therefore never need recorded steps) vs.executed_code(pre-computed code maps). No behavioural change; the point was to verify runtime neutrality and to establish the invariant — only the failing arena's steps are ever read — that makes the step-stripping commits sound by construction.-vv-vvv-vvvvNeutral, as required (the +12/+16 % readings are within the noise band of these cells and reverse sign in later runs).
perf: free unconsumed trace arenas as soon as each test finishes(O1)Introduces
TraceRetentionPolicy, applied once per finished test. Freesexecution_traceswhenincludeTraceswill never surface them (always atNone; passing tests atFailing), counterexample arenas (no consumer at all after a test finishes — not decoded, not exposed over napi), andgas_report_traceswhen no gas report was requested (also fixing a leak where passing tests atFailingskipped the existing clear).-vv-vvv-vvvvSmall on these all-passing benchmarks — per-test retained arenas turn out not to dominate them — but it is the mechanism the following commits act through, and its target scenarios (failing tests, the
Failing-mode leak) aren't exercised here.perf: strip recorded EVM steps from arenas retained for call traces(O2b)The
-vvvvfix. After stack-trace generation nothing reads steps: trace decoding, the gas report and the napi conversion only use the call tree, logs and ordering (the napi conversion explicitly discards step entries). NewSparsedTraceArena::strip_steps()(resolving pause/resume-tracing bookkeeping first, since it refers to step indices) is applied to every arena the policy keeps.-vv-vvv-vvvvperf: strip EVM steps from displaced arenas as traces accumulate(O2a)Same idea applied during a test: when a new arena is pushed onto a collection, the one it displaces as most-recent can no longer be a failing trace, so its steps are stripped immediately. Bounds the worst case within a test (
beforeTestSetupchains, invariant replay sequences) to one step-laden arena at a time.-vv-vvv-vvvvWithin noise vs. O2b on these repos (multi-arena tests are rare there); kept because it bounds the worst case at zero cost.
perf: don't collect invariant run traces past the gas-sample budget(O3a)The invariant fix. During an invariant campaign every run accumulated up to
deptharenas inrun_traces— with steps, underAlways— only forend_runto throw them away unless gas-report samples were still needed (the napi layer always setsgas_report_samples = 0). The same condition is now applied at collection time.-vv-vvv-vvvvmorpho-blue also gets ~15 % faster (10.9s → 8.8s). ¹ The uniswap
-vvvcell is this benchmark's noisiest (2.1–3.3 GiB across all builds); the +42 % is an outlier reading, not an effect of this commit — it returns to −4 % two builds later.perf: don't retain invariant replay arenas that nothing will consume(O3b)A passing invariant test replays its last run purely to collect logs and identify created contracts, but also pushed one arena per replayed call (up to
depth+2) into the test result. Pushes are now gated on the retention policy; arenas are still kept whenever stack-trace generation needs them as code sources.-vv-vvv-vvvvMarginal trim on morpho vs. O3a; mainly removes a per-test transient. ² solady
-vvvvnoise (4.4–5.8 GiB band).perf: share setup trace arenas between a suite's napi test results(O4b)The napi conversion cloned the suite's setup arenas into every included test result (N copies for an N-test suite at
All). The surfaced subset is now materialized once per suite (moved, not cloned) and shared viaArc;callTraces()output is unchanged.-vv-vvv-vvvvWithin noise here — these repos' setup arenas are step-free and small. Kept because it removes the pathological case for heavyweight
setUps at no cost.perf: derive a suite-level deployed-code map instead of retaining setup arenas(O4c)Stack-trace decoding only needs setup arenas for the code of contracts deployed during setup. That's now captured into
ExecutedCodeMapsas setup runs, used everywhere the arenas were previously walked, letting setup arenas be dropped at suite end underNoneand letting failure re-runs skip re-tracing setup. Subtlety: at Hardhat's default (OnFailure+None) setup runs untraced, so the re-run path still re-traces setup to rebuild the map — covered by a new regression test that fails without the conditional.-vv-vvv-vvvvWithin noise here; its wins need failing tests or heavyweight
setUps, which these all-passing benchmarks don't have.refactor!: rename IncludeTraces to IncludeCallTracesIncludeCallTracescontrols whether call trace arenas appear in test results (mirroring the provider'sincludeCallTracesoption);CollectStackTracesseparately controls stack-trace computation. The old name suggested it governed trace collection as a whole. Breaking for the napi API (includeCallTracesproperty); the in-repo Hardhat patch is updated, and Hardhat needs the same rename when adopting this version.What's not fixed (follow-up)
solady still peaks ~4.5 GiB at
-vvv+: that is the transient cost of step recording itself while a test executes, not retention. The stack-trace inferrer only readsstep.pcandstep.op, whileCallTraceStepstores far more per opcode — a slimmer step record in the tracer (revm-inspectors) is the natural next step.Testing
cargo test -p edr_solidity_tests(123 passing, incl. new regression tests forAlways-mode stack traces and the untraced-setup re-run path)IncludeCallTraces.All/.Failingcall-trace suites andOnFailure/Alwaysstack-trace guardrailsAll, and the policy keeps what it needs)