Add eval/benchmark suite, cut Ruby↔C++ overhead, fix the install path, add CI

[benngarcia]

Adds a benchmark suite that converts .pt checkpoints to .pte and runs them through the gem, then fixes what it found. Also adds CI and a working ExecuTorch build script.

Measured on 4-core x86_64, Ruby 3.3.6, Rice 4.12, ExecuTorch 0.7.0. All five benchmark runs are checked into bench/results/ as the evidence; bench/FINDINGS.md is the full walkthrough.

The eval harness (bench/)

bench_models.py     model definitions
make_pt_models.py   → .pt checkpoints + golden input/output from PyTorch
pt_to_pte.py        → .pte  (torch.export → to_edge → to_executorch)
run_bench.rb        → evals vs PyTorch + per-phase timings and allocations
compare.rb          → diffs two runs

Seven models spanning three regimes, from x*2+1 (pure overhead) to resnet18. pt_to_pte.py works on any .pt, not just these. Every model's output matches PyTorch to ≤3e-6.

Timings are batched so microsecond-scale operations aren't swamped by clock overhead, and each phase stops early once it hits a time budget so resnet18 doesn't stall the suite.

Finding 1: the bindings paid an rb_protect per element

Rice routes every Array element access and every scalar conversion through detail::protect(), which is an rb_protect — a VM tag push plus setjmp. The tensor loop paid two per element in, one per element out:

float_data.push_back(static_cast<float>(
  detail::From_Ruby<double>().convert(data[i].value())));   // two rb_protects

That's ~3,000 cycles to read one float. Building a resnet18 input took 92 ms before any math ran.

Fixed by inlining the Float/Fixnum paths (falling back to Rice only for values that genuinely can raise or run Ruby code), building output arrays with rb_ary_new_capa + rb_ary_push, dropping a deep-copy of every input in forward() (the caller's Array holds them live, so the copy bought nothing), replacing exception-based type dispatch with Data_Type::is_descendant, and flattening nested arrays with Array#concat instead of a flat_map that allocated an object per leaf.

phase	model	before	after
tensor_new_flat	resnet18	92.20 ms	7.00 ms	13.2×
to_a_flat	mobilenet_v2	1.105 ms	0.017 ms	64.6×
tensor_new_nested allocs	resnet18	153,238	11

Also adds Tensor.from_bytes / #to_binary, an opt-in raw-buffer path that skips per-element conversion entirely: 92 ms → 0.53 ms.

Finding 2: the kernels were worth 658×

A default build links the portable reference kernels. resnet18 forward was 8.26 s against PyTorch eager's 25 ms.

I built the XNNPACK delegate and measured rather than assuming:

model	portable	XNNPACK		PyTorch eager
resnet18	8262 ms	12.56 ms	658×	25.3 ms
mobilenet_v2	1618 ms	3.39 ms	477×	16.7 ms

Delegated ExecuTorch ends up ~2× faster than PyTorch eager on resnet18.

I also tested KERNELS_OPTIMIZED, since it needs no re-export and sounds like the easy win — it bought nothing measurable (8204 ms vs 8262 ms). These models are all convolution, which still falls back to portable. Worth knowing before reaching for it.

Finding 3: the two halves need each other

Neither piece looks impressive alone. On portable kernels the binding fixes barely move end-to-end time — which is exactly why a 0.6 µs/element conversion cost went unnoticed.

So I rebuilt the original bindings against the XNNPACK runtime rather than extrapolating:

model	e2e	forward()	overhead
mobilenet_v2	107.9 ms	3.7 ms	96.6%
resnet18	116.4 ms	11.7 ms	90.0%

Building the mobilenet_v2 input cost 28× more than running the model. With both fixes plus from_bytes, end-to-end goes 107.9 ms → 5.26 ms (20.5×).

Fix the kernels first, because that's the 658× — but the moment you do, the bindings become the bottleneck.

Finding 4: the documented install path didn't work

cmake --install doesn't install extension/module/module.h or the runtime/executor tree, both of which this gem includes directly. rake compile fails with "module.h header not found" on a perfectly good ExecuTorch build. Anyone following the README hit this.

script/build-executorch.sh handles the whole thing: pinned clone, submodules, the matching buck2 binary (ExecuTorch's CMake shells out to it), configure, build, install, then copy the ~1300 headers the install step leaves behind. Verified end to end — a fresh prefix built by this script compiles the extension and passes all 70 tests.

CI

• ci.yml — lint (Ruby/shell syntax, gem packaging, CHANGELOG entry) plus tests on Ruby 3.1–3.4 on Linux and 3.3 on macOS. ExecuTorch is built once per OS and cached; the test matrix restores that cache rather than paying a cold build per Ruby version.
• release.yml — on a v* tag: verify the tag matches Executorch::VERSION and the CHANGELOG has an entry, build the gem, check the extension sources are actually in the package, publish via RubyGems trusted publishing (OIDC, no API key in secrets), cut a GitHub release. workflow_dispatch runs it as a dry run.

Other fixes

• extconf.rb whole-archived two operator libraries when both were present, which aborts the runtime at init on duplicate operator registration. Now picks one, with EXECUTORCH_OPS_LIB to override. Related: self-registering libraries must be whole-archived or the linker silently drops them and you get a missing-operator error at model load.
• Object input = inputs[i] no longer compiles against Rice 4.12, which lost Array::Proxy's implicit conversion. This was needed just to build.
• The CI trigger and the gemspec's changelog_uri both assumed a main branch; the default branch is master, so the changelog link would have 404'd on rubygems.org.

Testing

70 tests, 0 failures, 0 skips. add_mul.pte is wired in as test/support/models/simple.pte, so the 8 previously-skipped model tests now actually run — meaning the changed forward() path is covered. New test/binary_tensor_test.rb covers the binary API including dtype round trips and byte-length mismatches.

Before merging

• Trusted publishing needs a one-time setup on rubygems.org before release.yml can publish — repo benngarcia/executorch-ruby, workflow release.yml, environment release. Until then the release job will fail at the publish step.
• The version bump is left to you. Changes are in a ## [Unreleased] CHANGELOG section; 0.2.0 seems right given the new public API.
• The first CI run pays a cold ExecuTorch build (tens of minutes). Subsequent runs restore from cache.

---

Generated by Claude Code