ci: add release performance benchmarks

[mrousavy]

Summary

• add a dedicated Release/Hermes benchmark entrypoint to the existing example app, backed by the real generated C++, Swift/Kotlin, and TurboModule bindings
• compare base and head on the same Android emulator/iOS simulator in base → head → head → base order, with reversed case order for the second pair and one permitted noisy repeat pair
• post the rebuilt paired-comparison table in an updatable PR comment; publish validated BMF data to the existing Bencher nitro project through a trusted workflow_run reporter
• retain Harness as the Debug-only functional test lane and keep performance verdicts advisory during calibration

Benchmark protocol

• 40 boundary/control metrics across primitives, properties, strings, arrays, structs, maps, optionals, variants, HybridObjects, ArrayBuffers, callbacks, and advisory Promise metrics
• calibrated batches targeting 150 ms, five warmups, twenty measured samples, checksum validation, median/p95/MAD, and deterministic bootstrap 95% confidence intervals
• bounded iteration counts for allocation-heavy JNI/ArrayBuffer cases; progress, collection opportunities, and result transport stay outside timed regions
• mandatory Android KVM acceleration, bounded device waits, and retained native crash diagnostics

The base of this initial PR predates the benchmark infrastructure. This bootstrap run uses the head binary for both sides as an A/A validation, not as a measurement of the old main binary. Subsequent PRs build both revisions independently.

Security and rollout

• the PR workflow is unprivileged and uploads bounded artifacts only
• the default-branch reporter validates workflow/repository/PR/SHA identity and bounded raw data, then rebuilds comparisons, Markdown, and BMF without executing artifact content or PR code
• actions and the Bencher installer are pinned to immutable commits
• Bencher uploads stay disabled until the exposed credential is revoked, BENCHER_KEY is replaced with a new project-scoped key, and the repository variable NITRO_BENCHER_ENABLED is set to true
• paired PR comments do not need the Bencher credential and do not overwrite user comments or newer PR revisions
• enforcement is deliberately not enabled; after at least 30 successful main/no-change runs, a separately reviewed per-case policy must use max(5%, 1.5 × p95(abs(no-change delta))), leaving budgets over 10% and Promise metrics advisory

GitHub only loads workflow_run definitions from the default branch. This PR therefore tests the unprivileged workflow; the trusted PR-comment/Bencher reporter can run end to end only after landing on main.

Validation

• bun run test:performance-tools (22 tests, including receiver failure paths, process-level A/A drift and ±15% comparisons, run-aware confidence intervals, schema/BMF/Markdown validation, and PR-comment safeguards)
• bun run typecheck:performance-tools
• bun example typecheck and bun example lint-ci
• Android assembleBenchmark Release build with Hermes/R8
• local hardware-accelerated Android arm64: 40 validated metrics / 20 samples, Release Nitro bindings, __DEV__ === false, Hermes enabled; latest diagnostic suite completed in 78 seconds
• Xcode 26.5 CI: complete six-run iOS Release/Hermes sequence passed; generated Swift shared-pointer compatibility fixed in Nitrogen and regenerated bindings

CI investigation

The first Android measurement step took 50m40s because the emulator silently selected -accel off after a /dev/kvm permission failure. KVM is now required and verified. In the successful full performance run:

• emulator boot dropped from about 8.5 minutes to 48 seconds
• the Android emulator/benchmark step dropped from 50m40s to 11m28s
• the whole Android job dropped from 59m05s to 20m09s
• both platforms completed six suites with 40 metrics and 20 samples each, and nitro-performance passed

One accelerated Android attempt had an app-process exit that did not recur on the successful retry. Native crash capture and bounded device waits are retained for future diagnosis.

The A/A data exposed one false-positive TurboModule control classification from pooling process runs as independent batches. The final host-only fix resamples matched process runs before their batches, and includes between-run variability in the noise check and Bencher bounds. Replaying the complete CI artifact through the trusted validator produces no A/A regressions on either platform; consistent ±15% changes still pass their detection tests. A fresh CI run is running after the final reporting fix and a bounded udev-settle wait for KVM permissions. Its tooling tests and KVM preflight have passed; native app and sampler code are unchanged.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
nitro-docs	Ready	Preview	Sep 4, 2026 7:45am UTC

[mrousavy]

Bootstrap A/A preview from the successful Release/Hermes CI run, reprocessed with the final run-aware comparator. Both sides use the same binary because this PR introduces the benchmark infrastructure. The observed differences below are CI noise, not changes to Nitro. No Bencher upload was performed.

Validated paired comparison table (Android and iOS)

Nitro performance

Performance changes are advisory while the baseline is being calibrated.

Android

Benchmark	Base	Head	Δ	95% CI	Status
javascript/control/add-numbers	44.0 ns	43.8 ns	-0.45%	-34.80%…5.38%	⚪ unchanged
nitro-cpp/array-buffer/bounce-1-mib	653.8 ns	647.7 ns	-0.92%	-8.73%…19.11%	🟡 noisy
nitro-cpp/array-buffer/bounce-4-kib	698.3 ns	588.9 ns	-15.66%	-17.73%…5.89%	🟡 noisy
nitro-cpp/array-buffer/copy-1-mib	35.45 µs	37.78 µs	6.59%	0.10%…11.28%	🟡 noisy
nitro-cpp/array-buffer/copy-4-kib	1.57 µs	1.53 µs	-2.52%	-6.42%…10.82%	🟡 noisy
nitro-cpp/array/large-1024	82.50 µs	82.87 µs	0.45%	-0.23%…1.08%	⚪ unchanged
nitro-cpp/array/small-16	1.70 µs	1.69 µs	-0.71%	-2.50%…1.59%	⚪ unchanged
nitro-cpp/callback/synchronous	496.2 ns	499.7 ns	0.70%	-5.06%…4.21%	🟡 noisy
nitro-cpp/hybrid-object/create	1.71 µs	1.68 µs	-1.56%	-6.12%…8.54%	🟡 noisy
nitro-cpp/hybrid-object/return-existing	419.4 ns	478.7 ns	14.14%	-3.01%…25.22%	🟡 noisy
nitro-cpp/map/typed-eight-entries	3.55 µs	3.47 µs	-2.24%	-4.01%…-1.20%	⚪ unchanged
nitro-cpp/optional/trailing-string	271.6 ns	275.9 ns	1.60%	-1.75%…3.23%	⚪ unchanged
nitro-cpp/primitive/add-numbers	168.4 ns	167.4 ns	-0.58%	-2.06%…1.55%	⚪ unchanged
nitro-cpp/primitive/simple-func	124.1 ns	123.5 ns	-0.44%	-1.35%…2.72%	⚪ unchanged
nitro-cpp/promise/immediate	1.72 µs	1.72 µs	0.34%	-0.41%…1.41%	ℹ️ advisory
nitro-cpp/property/number-get-set	266.6 ns	263.9 ns	-1.02%	-1.78%…0.06%	⚪ unchanged
nitro-cpp/string/ascii-short	243.8 ns	244.2 ns	0.15%	-2.44%…1.35%	⚪ unchanged
nitro-cpp/string/unicode	534.5 ns	533.8 ns	-0.13%	-0.96%…1.04%	⚪ unchanged
nitro-cpp/struct/nested-car	4.55 µs	4.59 µs	0.87%	-0.78%…1.41%	🟡 noisy
nitro-cpp/variant/number-or-string	256.6 ns	257.7 ns	0.44%	-1.63%…3.48%	⚪ unchanged
nitro-platform/array-buffer/bounce-1-mib	1.91 µs	2.05 µs	7.42%	-2.61%…14.48%	🟡 noisy
nitro-platform/array-buffer/bounce-4-kib	1.92 µs	1.98 µs	2.93%	-1.83%…7.47%	🟡 noisy
nitro-platform/array-buffer/copy-1-mib	470.86 µs	446.63 µs	-5.15%	-10.51%…1.95%	🟡 noisy
nitro-platform/array-buffer/copy-4-kib	8.24 µs	8.63 µs	4.78%	-1.06%…8.82%	🟡 noisy
nitro-platform/array/large-1024	89.83 µs	88.80 µs	-1.14%	-2.07%…0.34%	⚪ unchanged
nitro-platform/array/small-16	2.22 µs	2.18 µs	-1.77%	-2.85%…0.56%	⚪ unchanged
nitro-platform/callback/synchronous	1.74 µs	1.98 µs	13.90%	-2.78%…16.03%	🟡 noisy
nitro-platform/hybrid-object/create	4.01 µs	3.86 µs	-3.81%	-6.22%…-0.98%	🟡 noisy
nitro-platform/hybrid-object/return-existing	1.50 µs	1.48 µs	-1.42%	-9.11%…7.30%	🟡 noisy
nitro-platform/map/typed-eight-entries	13.04 µs	13.17 µs	1.01%	-1.34%…2.65%	⚪ unchanged
nitro-platform/optional/trailing-string	697.6 ns	714.2 ns	2.38%	-1.58%…4.73%	⚪ unchanged
nitro-platform/primitive/add-numbers	264.6 ns	261.4 ns	-1.21%	-3.38%…1.88%	⚪ unchanged
nitro-platform/primitive/simple-func	211.4 ns	206.8 ns	-2.20%	-8.32%…-0.81%	⚪ unchanged
nitro-platform/promise/immediate	6.55 µs	5.93 µs	-9.39%	-17.85%…5.40%	ℹ️ advisory
nitro-platform/property/number-get-set	532.4 ns	517.4 ns	-2.82%	-3.38%…-2.13%	⚪ unchanged
nitro-platform/string/ascii-short	917.1 ns	907.6 ns	-1.03%	-1.74%…0.29%	⚪ unchanged
nitro-platform/string/unicode	1.50 µs	1.49 µs	-0.69%	-2.20%…0.83%	⚪ unchanged
nitro-platform/struct/nested-car	9.52 µs	9.39 µs	-1.43%	-2.73%…1.19%	⚪ unchanged
nitro-platform/variant/number-or-string	873.8 ns	884.1 ns	1.18%	-1.08%…2.64%	⚪ unchanged
turbo-module/control/add-numbers	527.9 ns	561.8 ns	6.40%	-1.75%…9.82%	🟡 noisy

iOS

Benchmark	Base	Head	Δ	95% CI	Status
javascript/control/add-numbers	34.9 ns	37.1 ns	6.45%	-10.56%…10.60%	🟡 noisy
nitro-cpp/array-buffer/bounce-1-mib	433.2 ns	468.1 ns	8.08%	-12.41%…33.62%	🟡 noisy
nitro-cpp/array-buffer/bounce-4-kib	412.0 ns	534.0 ns	29.61%	-13.01%…41.79%	🟡 noisy
nitro-cpp/array-buffer/copy-1-mib	111.03 µs	120.95 µs	8.94%	-17.55%…47.18%	🟡 noisy
nitro-cpp/array-buffer/copy-4-kib	1.73 µs	1.92 µs	10.80%	-18.89%…32.84%	🟡 noisy
nitro-cpp/array/large-1024	62.76 µs	72.77 µs	15.94%	-10.20%…25.68%	🟡 noisy
nitro-cpp/array/small-16	1.22 µs	1.44 µs	18.03%	-24.07%…39.61%	🟡 noisy
nitro-cpp/callback/synchronous	318.9 ns	334.3 ns	4.81%	-13.70%…15.55%	🟡 noisy
nitro-cpp/hybrid-object/create	793.5 ns	897.4 ns	13.10%	-20.53%…38.06%	🟡 noisy
nitro-cpp/hybrid-object/return-existing	355.6 ns	376.5 ns	5.90%	-3.97%…50.52%	🟡 noisy
nitro-cpp/map/typed-eight-entries	2.33 µs	2.33 µs	-0.10%	-6.18%…6.21%	🟡 noisy
nitro-cpp/optional/trailing-string	247.0 ns	241.2 ns	-2.36%	-8.19%…4.86%	🟡 noisy
nitro-cpp/primitive/add-numbers	133.4 ns	162.1 ns	21.50%	-7.85%…31.91%	🟡 noisy
nitro-cpp/primitive/simple-func	128.8 ns	152.2 ns	18.15%	-11.57%…39.41%	🟡 noisy
nitro-cpp/promise/immediate	1.27 µs	1.35 µs	6.35%	-25.23%…11.69%	ℹ️ advisory
nitro-cpp/property/number-get-set	229.7 ns	231.4 ns	0.76%	-5.58%…9.52%	🟡 noisy
nitro-cpp/string/ascii-short	208.0 ns	210.1 ns	1.03%	-24.55%…6.89%	🟡 noisy
nitro-cpp/string/unicode	463.7 ns	483.9 ns	4.37%	-14.93%…6.95%	🟡 noisy
nitro-cpp/struct/nested-car	3.53 µs	3.15 µs	-10.73%	-16.49%…-1.84%	🟡 noisy
nitro-cpp/variant/number-or-string	249.1 ns	220.8 ns	-11.36%	-19.82%…3.01%	🟡 noisy
nitro-platform/array-buffer/bounce-1-mib	519.0 ns	477.2 ns	-8.05%	-47.29%…18.09%	🟡 noisy
nitro-platform/array-buffer/bounce-4-kib	581.1 ns	519.8 ns	-10.55%	-18.11%…-0.19%	🟡 noisy
nitro-platform/array-buffer/copy-1-mib	108.85 µs	128.95 µs	18.46%	3.34%…41.87%	🟡 noisy
nitro-platform/array-buffer/copy-4-kib	2.00 µs	2.01 µs	0.10%	-7.81%…10.21%	🟡 noisy
nitro-platform/array/large-1024	74.73 µs	66.50 µs	-11.01%	-22.54%…-2.21%	🟡 noisy
nitro-platform/array/small-16	1.54 µs	1.31 µs	-14.89%	-27.26%…-1.91%	🟡 noisy
nitro-platform/callback/synchronous	476.1 ns	480.8 ns	0.98%	-10.85%…15.37%	🟡 noisy
nitro-platform/hybrid-object/create	48.69 µs	41.84 µs	-14.06%	-23.81%…5.91%	🟡 noisy
nitro-platform/hybrid-object/return-existing	467.9 ns	473.0 ns	1.08%	-6.39%…14.19%	🟡 noisy
nitro-platform/map/typed-eight-entries	3.43 µs	3.68 µs	7.13%	-30.37%…21.44%	🟡 noisy
nitro-platform/optional/trailing-string	260.4 ns	284.8 ns	9.37%	-25.86%…25.40%	🟡 noisy
nitro-platform/primitive/add-numbers	152.1 ns	130.5 ns	-14.22%	-23.45%…1.85%	🟡 noisy
nitro-platform/primitive/simple-func	119.0 ns	124.7 ns	4.74%	-27.97%…9.91%	🟡 noisy
nitro-platform/promise/immediate	2.41 µs	2.46 µs	2.17%	-4.54%…82.80%	ℹ️ advisory
nitro-platform/property/number-get-set	286.9 ns	239.4 ns	-16.57%	-29.63%…-1.28%	🟡 noisy
nitro-platform/string/ascii-short	256.6 ns	272.6 ns	6.27%	-33.30%…17.95%	🟡 noisy
nitro-platform/string/unicode	620.4 ns	570.9 ns	-7.98%	-41.31%…0.78%	🟡 noisy
nitro-platform/struct/nested-car	3.69 µs	3.52 µs	-4.65%	-27.05%…10.32%	🟡 noisy
nitro-platform/variant/number-or-string	363.6 ns	358.1 ns	-1.53%	-28.72%…14.92%	🟡 noisy
turbo-module/control/add-numbers	1.47 µs	1.49 µs	1.98%	-4.13%…12.50%	🟡 noisy

_{Base 94e72e6d · Head d9d52410 · lower is better}

[mrousavy]

Superseded by the atomic stack requested in review: #1582 moves the example app, and #1583 adds the standalone benchmark app and performance CI on top. The dependency-version guard will be a third follow-up PR. The old branch is retained for reference.