Benchmark the solx DWARF decode path - #1718
Draft
nebasuke wants to merge 3 commits into
Draft
Conversation
Nothing benchmarked EDR's DWARF consumption: the JS scenario benchmarks are solc-compiled and the crate's contracts_identifier bench covers the solc source-map path. Meanwhile decode_instructions re-parses hex -> ELF -> DWARF on every call and its time grows ~bytes^1.7, so dependency changes in gimli/addr2line (e.g. #1626's lazy-to-eager DIE attribute parsing) and decode changes need a standing measurement. Two pieces: - benches/dwarf_decode: criterion bench that always runs on the committed scenarios fixture (full pass + largest blob, with byte throughput), and accepts EDR_DWARF_BENCH_DIR with standard-JSON input/output pairs for large-blob corpora, since the committed fixture's 13 KB max blob is the cheap regime of the scaling curve. - examples/dwarf_ab: the A/B driver used to validate #1626, for what criterion cannot report — per-stage RSS checkpoints, per-blob timings for fitting the scaling exponent, and an order-independent output digest for cross-revision equivalence checks.
Unlike the wall-time benchmarks that compare a single sample against the last stored value on main (a design whose alert threshold sits inside its own noise band), this job benches the PR's merge commit against its first parent on the same runner in the same job, and lets criterion do the statistics. The gate fails only when the 95% confidence interval puts the mean regression above 10%, so a few percent of runner drift per side cannot fail an innocent PR; regressions subtler than that are deferred to a deterministic-counter setup (iai-callgrind + Bencher, the slang pattern). Path-filtered to the decode path's modules and fixtures, Cargo.lock, and the workflow itself — dependency bumps are the proven risk vector (a gimli major bump is what prompted the benchmark). CI only exercises the committed scenarios fixture: the large real-world corpora contain UNLICENSED third-party sources and stay local via EDR_DWARF_BENCH_DIR.
nebasuke
had a problem deploying
to
github-action-benchmark
September 1, 2026 09:56 — with 
GitHub Actions
Error
GitHub Actions
Error
|
A PR that introduces or moves the bench target (this one included) has no base side to A/B against; benching HEAD^1 would fail on the missing bench and the gate would then fail loudly on the absent comparison. Detect the bench at the base commit and fall back to a single informational run.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1718 +/- ##
==========================================
+ Coverage 79.90% 79.97% +0.06%
==========================================
Files 461 462 +1
Lines 80120 80509 +389
Branches 80120 80509 +389
==========================================
+ Hits 64023 64387 +364
- Misses 13889 13893 +4
- Partials 2208 2229 +21 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
nebasuke
temporarily deployed
to
github-action-benchmark
September 1, 2026 10:10 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
nebasuke
temporarily deployed
to
github-action-benchmark
September 1, 2026 10:35 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
nebasuke
temporarily deployed
to
github-action-benchmark
September 1, 2026 10:35 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
nebasuke
added a commit
that referenced
this pull request
Sep 1, 2026
Real projects ship vanity runs values -- aave-v4's config used optimizer.runs: 444444444444, which solc accepts (it reads the field as 64-bit, as does foundry-compilers) but overflowed our Option<u32>. The serde error then propagated out of BuildInfoBuffers::parse, failing the entire build-info parse and with it stack traces, over a field the decoder never reads. Found while building a DWARF decode benchmark corpus from aave-v4's real build-info (#1718). The regression test round-trips the value through serialization so a silent clamp would also fail it; it was proven red on the pre-fix code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Claude summary
Why
Reviewing #1626 (addr2line 0.25→0.27 + gimli 0.32→0.34, which switches DIE attribute parsing from lazy to eager) surfaced that nothing benchmarks EDR's DWARF consumption:
providerBenchmark/soltestsBenchmarkare solc-compiled, the crate'scontracts_identifierbench covers the solc source-map path, and the solx compile benchmark measures solx emitting DWARF, not EDR consuming it. The measurement for #1626 had to be done with an ad-hoc, uncommitted driver.Two properties make a standing benchmark worthwhile here:
dwarf::decode_instructionsre-parses hex → ELF → DWARF on every call, once per contract bytecode section at build-info load — so it is a load-time cost that scales with project size.What
benches/dwarf_decode.rs— criterion bench, runs unconditionally on the committed scenarios fixture (no silent env-var skip). Per corpus: afull_passover all decodable blobs and alargest_blobbench, both reporting byte throughput. Because the committed fixture tops out at ~13 KB blobs (the cheap regime), the bench also acceptsEDR_DWARF_BENCH_DIRpointing at a directory of<name>.input.json/<name>.output.jsonsolx standard-JSON pairs, each benchmarked as its own corpus.examples/dwarf_ab.rs— the A/B driver used to validate fix(deps): update rust crate addr2line to 0.27 #1626, committed so the next dependency bump or decode change doesn't need to rebuild it. It reports what criterion can't: per-stage RSS checkpoints, per-blob timings for fitting the scaling exponent, and an order-independent output digest for cross-revision equivalence checks. Its doc header carries the recipe for building a large corpus pair with solx.Numbers (this branch)
full_pass/scenarios(96 blobs, 490 KB DWARF)full_pass/aave-v4(118 blobs, 1.14 MB DWARF)full_pass/corpusB(sourcify contract, 1.36 MB DWARF)largest_blob/scenarios/13052Blargest_blob/aave-v4/145428Blargest_blob/corpusB/84160B(corpusB = sourcify
11155111_0xa270236e935b27af69535f377eeea36ee48742f3; solx 0.1.7 and 0.1.8 emit byte-identical DWARF for it. aave-v4 = the HH3 migration fork's real hardhat-slang-solx build-info input recompiled with released solx 0.1.8; its largest blob is SpokeInstance's deployed bytecode. aave's 145 KB blob decoding faster than corpusB's 84 KB blob shows per-blob cost tracks the model's per-file tables, not blob bytes alone — corpusB is a single 2.2 MB source file, aave spreads 123 sources.)Building the aave corpus also surfaced a real parse bug:
OptimizerSettings.runsisOption<u32>, but aave shipsoptimizer.runs: 444444444444(valid for solc, which reads 64-bit) — so EDR's build-info parsing hard-fails on that project's real build-info, solc and solx paths alike. Tracked separately; the bench corpus clamps the field as a workaround.CI: same-machine A/B on PRs
.github/workflows/dwarf-decode-benchmark.ymlbenches the PR's merge commit against its first parent (the base tip) on the same runner in the same job, so machine variance cancels, and lets criterion do the statistics. The gate fails only when the 95% confidence interval puts the mean regression above 10% — measured same-machine rerun noise reaches ~+6% mean (CI lower bound +2.7%), so anything gated tighter than the CI bound would false-alarm. This is deliberately not the single-sample-vs-last-stored-value pattern of the wall-time benchmarks, whose 110% threshold sits inside the soltests series' ~14% noise band.Path-filtered to
crates/edr_solidity/**,Cargo.lock, and the workflow itself — dependency bumps are the proven trigger (#1626).workflow_dispatchgives an informational single run.Scope limits, stated openly: CI only exercises the committed scenarios fixture (small blobs — the large real-world corpora embed
UNLICENSEDthird-party sources and cannot be committed; they stay local viaEDR_DWARF_BENCH_DIR), and the 10%-confident gate won't catch subtle regressions. Both are addressed by the planned follow-up: deterministic counters (iai-callgrind → Bencher, the slang pattern) plus a synthesized, owned large-DWARF fixture.Out of scope
RangeSweep/DecodeCaches, validated at 5.5× on corpus B) — separate PR; this benchmark is how its effect stays visible.