Conversation
jamescook
marked this pull request as draft
July 30, 2026 20:18
jamescook
marked this pull request as ready for review
July 31, 2026 10:40
Each pure-Ruby benchmark now runs a third subprocess variant under --zjit in addition to the existing no-JIT and --yjit runs, so BENCHMARKS.md reports native, interpreter, YJIT, and ZJIT side by side with a ZJIT-vs-YJIT speedup row as the headline comparison. Each worker subprocess is told which JIT mode it's expected to be running under via CATARACT_BENCH_JIT and asserts that RubyVM actually reports that mode before any benchmarking starts, so a Ruby build that silently can't honor --yjit/--zjit (or an inherited environment variable overriding the intended flag) fails loudly instead of quietly measuring the wrong thing under the wrong label. Also fixes format_speedup, which always labeled a comparison "faster" regardless of the ratio - misleading now that ZJIT vs YJIT can legitimately come out slower.
The harness identified a run by a single symbol - :pure_with_zjit - that
mashed two independent axes together: which Cataract backend was loaded,
and which JIT the VM was running. Every consumer took it apart again
with `sub(/_with_yjit|_without_yjit|_with_zjit/, '')`, nine copies of
that regex across the tests modules and the speedup calculator, plus
four copy-pasted case statements turning the string into a label.
Split the axes into values that own what varies by them:
RubyMode no JIT / YJIT / ZJIT - launch flags, detection,
labels, and normalized JIT statistics
Backend pure / native, each declaring the modes it runs under
Implementation a backend paired with a mode; the unit orchestrators,
workers and the doc generator all deal in
Results now carry `backend` and `jit` as separate fields, so nothing has
to recover them from a name. All nine regexes are gone.
Consequences worth noting:
- `yjit_applicable?` disappeared. "Native runs one way, pure runs three"
is now a list each backend declares, not a branch at four call sites.
- The eight worker classes became four. A worker reads its own
configuration off the VM, so nothing per-variant needed subclassing.
- run_subprocess and combine_worker_results, previously copy-pasted into
all four orchestrators, moved to BenchmarkHarness.
- In flattening and specificity, `case base_impl_type` had two branches
running identical code that differed only in the report label.
- The four speedup_config overrides were identical once test case ids
were normalized, so one default remains.
Verification is now symmetric across both axes: a worker confirms the
backend that loaded and the JIT now active both match what the parent
asked for, and refuses to measure otherwise. Previously only the JIT was
checked, and native was exempt - which is how native came to be measured
under YJIT (via RUBY_YJIT_ENABLE in the environment) while labeled and
compared as a no-JIT run. Native now launches with --disable-yjit and
asserts it.
Verification is a pure function of two values; the environment, the VM
and Cataract::IMPLEMENTATION are read once at Implementation.current and
passed as parameters below that. The results directory is likewise an
object handed to the harness rather than a constant, so a test redirects
it without touching global state.
BENCHMARKS.md rendering moved into ResultTable/OverheadTable. Column
headings come from the Implementation objects that produced the numbers,
so a heading cannot drift from its data - "Pure (no YJIT)" now reads
"Pure (no JIT)", which is what it always meant. Lookup is by whole test
case id rather than substring, fixing rows that borrowed another test
case's numbers: 'compact' matched 'bootstrap_compact' and 'shorthand'
matched 'no_shorthand', so two pairs of rows were published identical.
Tests cover the POROs directly and, importantly, the plumbing: a fake
benchmark exercises the real subprocess pipeline end to end in seconds,
and every orchestrator and worker script is checked to actually start a
benchmark when executed. The latter catches a script that defines its
class but never invokes it - which exits 0 and lets the whole rake task
succeed in silence.
ZJIT arrived in Ruby 4.0, and YJIT can be left out of a build, so on older Rubies `ruby --zjit` is an error rather than a slower run. The harness enumerated all four variants unconditionally and the plumbing test asserted it got all four back, which fails everywhere below 4.0. RubyMode now knows which VM constant backs it, so a mode can say whether this Ruby can run it at all. Implementation.available filters on that, and the harness and plumbing test work from that list instead of the full matrix. Availability is about the build, not the current switch position, so a JIT that is present but disabled still counts. Process.warmup is likewise Ruby 3.3+, so guard it: on older Rubies the run is just marginally noisier, and the compaction test that leans on it skips rather than errors. Only the parts that actually execute a JIT are affected. The tests that treat modes as data - labels, flags, ids, column headings - still cover ZJIT everywhere, because they run against a stand-in VM rather than the real one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each pure-Ruby benchmark now runs a third subprocess variant under --zjit in addition to the existing no-JIT and --yjit runs, so BENCHMARKS.md reports native, interpreter, YJIT, and ZJIT side by side with a ZJIT-vs-YJIT speedup row as the headline comparison.
Each worker subprocess is told which JIT mode it's expected to be running under via CATARACT_BENCH_JIT and asserts that RubyVM actually reports that mode before any benchmarking starts, so a Ruby build that silently can't honor --yjit/--zjit (or an inherited environment variable overriding the intended flag) fails loudly instead of quietly measuring the wrong thing under the wrong label.
Also fixes format_speedup, which always labeled a comparison "faster" regardless of the ratio - misleading now that ZJIT vs YJIT can legitimately come out slower.