Skip to content

test: demonstrate a HybridFunction performance regression (DO NOT MERGE) - #1588

Draft
mrousavy wants to merge 3 commits into
ci/check-app-version-alignmentfrom
test/performance-regression-canary
Draft

test: demonstrate a HybridFunction performance regression (DO NOT MERGE)#1588
mrousavy wants to merge 3 commits into
ci/check-app-version-alignmentfrom
test/performance-regression-canary

Conversation

@mrousavy

@mrousavy mrousavy commented Sep 4, 2026

Copy link
Copy Markdown
Member

Performance regression canary — DO NOT MERGE

This draft PR intentionally adds 128 rounds of dependent unsigned arithmetic to
each HybridFunction native-state lookup. Volatile local storage keeps the work
in optimized Release builds without shared state, sleeps, allocations, or altered
return values. Closing this PR discards the entire experiment.

The base is the top of the infrastructure stack, including the version-alignment
PR. Both base and head already contain the same benchmark app and definitions.
CI must build both binaries independently and run the existing ABBA sequence.
The suite hash must remain identical; this is not a rebaseline or an A/A run.

Expected observations:

  • C++ and Swift/Kotlin primitive calls and property access get slower.
  • JS and TurboModule controls do not execute this extra work.
  • Expensive operations can show a much smaller relative change.
  • Promise metrics remain advisory. Performance verdicts overall remain advisory;
    a detected regression does not imply that the build check should turn red.

End-to-end validation checks the real GitHub Actions artifacts, confirms base
and head SHAs and matching suite hashes, inspects the paired PR comment, and
verifies Bencher reports for both platform testbeds and this PR branch. Bencher
publishing requires the rotated project key in BENCHER_KEY and
NITRO_BENCHER_ENABLED=true; the paired PR comment is independent of that key.

Reporting fixes belong in the infrastructure PR, not in this canary diff. Keep
this PR draft and never merge its intentional production slowdown.

First paired CI result

Run 33898772711
built both revisions independently and completed six base/head suites per
platform, including the one permitted noise retry. Both sides used Release
Nitro, Hermes, and the same benchmark-suite hash.

  • Android C++ addNumbers: 173.6 ns/op → 621.4 ns/op (+258%, 95% CI
    +247%…+261%). The JS control changed -0.6%; the TurboModule control remained
    inconclusive.
  • iOS C++ addNumbers: 161.5 ns/op → 450.5 ns/op (+179%, 95% CI
    +154%…+211%). The first reporter called every synchronous iOS case noisy
    because its robust CV exceeded 5%, even when the paired interval was
    decisive.

That result exposed a reporting-policy bug rather than a sampling failure. The
infrastructure PR now lets a change whose full paired confidence interval is
beyond the budget keep its regression/improvement verdict. CV still produces a
neutral result when the interval cannot decide. Rebuilding the report from the
same raw samples yields 29 Android and 25 iOS regressions while leaving the JS
and TurboModule controls neutral.

@vercel

vercel Bot commented Sep 4, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nitro-docs Ready Ready Preview Sep 4, 2026 6:19pm UTC

Request Review

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

Nitro performance

Performance changes are advisory while the baseline is being calibrated.

Android

Benchmark Base Head Δ 95% CI Status
javascript/control/add-numbers 26.1 ns 26.4 ns 1.15% -2.37%…5.02% ⚪ unchanged
nitro-cpp/array-buffer/bounce-1-mib 558.4 ns 633.2 ns 13.40% 2.25%…22.09% 🔴 regression
nitro-cpp/array-buffer/bounce-4-kib 546.7 ns 634.2 ns 16.00% 4.42%…23.42% 🔴 regression
nitro-cpp/array-buffer/copy-1-mib 15.44 µs 15.57 µs 0.86% -0.90%…1.65% ⚪ unchanged
nitro-cpp/array-buffer/copy-4-kib 961.4 ns 1.04 µs 7.76% 2.04%…12.37% 🔴 regression
nitro-cpp/array/large-1024 40.51 µs 41.17 µs 1.64% -2.90%…3.42% ⚪ unchanged
nitro-cpp/array/small-16 872.0 ns 978.3 ns 12.20% 8.75%…14.22% 🔴 regression
nitro-cpp/callback/synchronous 422.6 ns 490.8 ns 16.16% 10.67%…26.12% 🔴 regression
nitro-cpp/hybrid-object/create 696.4 ns 871.1 ns 25.09% 22.66%…30.51% 🔴 regression
nitro-cpp/hybrid-object/return-existing 247.6 ns 464.9 ns 87.77% 83.23%…90.86% 🔴 regression
nitro-cpp/map/typed-eight-entries 1.90 µs 2.03 µs 6.60% 2.23%…9.94% 🔴 regression
nitro-cpp/optional/trailing-string 160.1 ns 277.5 ns 73.39% 69.24%…76.11% 🔴 regression
nitro-cpp/primitive/add-numbers 101.0 ns 214.0 ns 111.97% 105.62%…114.95% 🔴 regression
nitro-cpp/primitive/simple-func 85.9 ns 196.2 ns 128.36% 122.17%…136.32% 🔴 regression
nitro-cpp/promise/immediate 961.8 ns 1.05 µs 9.05% 6.51%…13.18% ℹ️ advisory
nitro-cpp/property/number-get-set 176.6 ns 387.8 ns 119.55% 113.28%…126.21% 🔴 regression
nitro-cpp/string/ascii-short 157.7 ns 265.6 ns 68.49% 66.08%…71.74% 🔴 regression
nitro-cpp/string/unicode 281.7 ns 394.8 ns 40.14% 36.93%…41.59% 🔴 regression
nitro-cpp/struct/nested-car 2.34 µs 2.53 µs 8.20% 4.41%…10.63% 🔴 regression
nitro-cpp/variant/number-or-string 149.1 ns 256.9 ns 72.35% 70.67%…75.73% 🔴 regression
nitro-platform/array-buffer/bounce-1-mib 1.52 µs 1.53 µs 0.44% -3.59%…7.46% 🟡 noisy
nitro-platform/array-buffer/bounce-4-kib 1.53 µs 1.57 µs 2.90% -4.21%…11.94% 🟡 noisy
nitro-platform/array-buffer/copy-1-mib 316.75 µs 313.89 µs -0.90% -3.77%…2.53% 🟡 noisy
nitro-platform/array-buffer/copy-4-kib 5.26 µs 5.30 µs 0.75% -3.71%…5.10% 🟡 noisy
nitro-platform/array/large-1024 46.54 µs 46.20 µs -0.73% -2.85%…1.02% ⚪ unchanged
nitro-platform/array/small-16 1.23 µs 1.33 µs 8.71% 6.76%…11.64% 🔴 regression
nitro-platform/callback/synchronous 1.19 µs 1.25 µs 5.85% 2.17%…11.10% 🔴 regression
nitro-platform/hybrid-object/create 2.05 µs 2.20 µs 7.21% 1.82%…13.88% 🔴 regression
nitro-platform/hybrid-object/return-existing 649.4 ns 851.9 ns 31.18% 29.51%…35.05% 🔴 regression
nitro-platform/map/typed-eight-entries 7.99 µs 8.19 µs 2.46% -0.61%…5.71% ⚪ unchanged
nitro-platform/optional/trailing-string 388.3 ns 488.0 ns 25.70% 22.88%…28.77% 🔴 regression
nitro-platform/primitive/add-numbers 133.5 ns 240.0 ns 79.74% 77.09%…84.45% 🔴 regression
nitro-platform/primitive/simple-func 108.0 ns 215.8 ns 99.84% 96.25%…103.62% 🔴 regression
nitro-platform/promise/immediate 2.76 µs 2.85 µs 3.44% 1.60%…7.48% ℹ️ advisory
nitro-platform/property/number-get-set 252.6 ns 469.0 ns 85.69% 81.69%…89.91% 🔴 regression
nitro-platform/string/ascii-short 550.8 ns 668.4 ns 21.36% 17.61%…23.17% 🔴 regression
nitro-platform/string/unicode 873.5 ns 979.1 ns 12.08% 4.61%…18.38% 🔴 regression
nitro-platform/struct/nested-car 5.33 µs 5.45 µs 2.23% 0.64%…4.17% ⚪ unchanged
nitro-platform/variant/number-or-string 477.7 ns 589.0 ns 23.31% 19.17%…25.98% 🔴 regression
turbo-module/control/add-numbers 263.5 ns 258.9 ns -1.74% -4.63%…5.43% ⚪ unchanged

iOS

Benchmark Base Head Δ 95% CI Status
javascript/control/add-numbers 42.9 ns 39.1 ns -9.01% -11.99%…23.14% 🟡 noisy
nitro-cpp/array-buffer/bounce-1-mib 463.0 ns 771.1 ns 66.54% 46.92%…97.81% 🔴 regression
nitro-cpp/array-buffer/bounce-4-kib 507.0 ns 857.1 ns 69.03% 46.01%…93.14% 🔴 regression
nitro-cpp/array-buffer/copy-1-mib 103.87 µs 117.01 µs 12.65% 2.77%…54.26% 🔴 regression
nitro-cpp/array-buffer/copy-4-kib 1.92 µs 2.41 µs 25.40% -8.41%…60.46% 🟡 noisy
nitro-cpp/array/large-1024 75.55 µs 79.47 µs 5.18% -15.89%…23.52% 🟡 noisy
nitro-cpp/array/small-16 1.60 µs 1.91 µs 19.84% 6.62%…31.52% 🔴 regression
nitro-cpp/callback/synchronous 384.4 ns 712.8 ns 85.45% 44.88%…174.67% 🔴 regression
nitro-cpp/hybrid-object/create 747.4 ns 1.52 µs 103.08% 41.66%…127.38% 🔴 regression
nitro-cpp/hybrid-object/return-existing 347.3 ns 1.08 µs 209.83% 139.31%…235.80% 🔴 regression
nitro-cpp/map/typed-eight-entries 2.90 µs 3.85 µs 32.78% 7.60%…57.19% 🔴 regression
nitro-cpp/optional/trailing-string 287.0 ns 653.1 ns 127.51% 111.88%…150.47% 🔴 regression
nitro-cpp/primitive/add-numbers 167.8 ns 425.5 ns 153.59% 143.13%…193.96% 🔴 regression
nitro-cpp/primitive/simple-func 143.0 ns 441.8 ns 209.00% 189.90%…235.80% 🔴 regression
nitro-cpp/promise/immediate 1.62 µs 1.89 µs 16.91% -10.40%…66.01% ℹ️ advisory
nitro-cpp/property/number-get-set 290.7 ns 869.9 ns 199.26% 182.71%…252.50% 🔴 regression
nitro-cpp/string/ascii-short 260.3 ns 452.8 ns 73.95% 65.27%…131.52% 🔴 regression
nitro-cpp/string/unicode 512.8 ns 863.7 ns 68.42% 35.23%…78.13% 🔴 regression
nitro-cpp/struct/nested-car 4.08 µs 4.49 µs 9.95% 0.08%…17.02% 🔴 regression
nitro-cpp/variant/number-or-string 261.0 ns 508.1 ns 94.65% 58.19%…146.22% 🔴 regression
nitro-platform/array-buffer/bounce-1-mib 526.4 ns 819.7 ns 55.71% 33.72%…71.33% 🔴 regression
nitro-platform/array-buffer/bounce-4-kib 456.1 ns 848.1 ns 85.95% 35.27%…95.78% 🔴 regression
nitro-platform/array-buffer/copy-1-mib 122.46 µs 94.94 µs -22.47% -33.54%…24.12% 🟡 noisy
nitro-platform/array-buffer/copy-4-kib 1.99 µs 2.48 µs 24.54% -1.98%…51.57% 🟡 noisy
nitro-platform/array/large-1024 70.32 µs 84.49 µs 20.15% -10.74%…28.47% 🟡 noisy
nitro-platform/array/small-16 1.51 µs 2.07 µs 37.40% 0.68%…47.95% 🔴 regression
nitro-platform/callback/synchronous 514.7 ns 811.6 ns 57.69% 22.44%…76.59% 🔴 regression
nitro-platform/hybrid-object/create 37.19 µs 45.34 µs 21.89% 6.60%…34.28% 🔴 regression
nitro-platform/hybrid-object/return-existing 435.6 ns 1.01 µs 131.82% 118.39%…163.91% 🔴 regression
nitro-platform/map/typed-eight-entries 3.55 µs 4.80 µs 35.41% -15.75%…59.44% 🟡 noisy
nitro-platform/optional/trailing-string 274.7 ns 624.4 ns 127.32% 84.40%…163.18% 🔴 regression
nitro-platform/primitive/add-numbers 141.0 ns 455.7 ns 223.13% 125.74%…247.24% 🔴 regression
nitro-platform/primitive/simple-func 118.4 ns 390.8 ns 229.93% 137.52%…302.17% 🔴 regression
nitro-platform/promise/immediate 2.52 µs 2.61 µs 3.62% -26.17%…42.87% ℹ️ advisory
nitro-platform/property/number-get-set 260.3 ns 867.3 ns 233.15% 95.68%…274.13% 🔴 regression
nitro-platform/string/ascii-short 292.7 ns 585.5 ns 100.01% 61.00%…145.07% 🔴 regression
nitro-platform/string/unicode 615.1 ns 1.04 µs 68.76% 13.13%…89.97% 🔴 regression
nitro-platform/struct/nested-car 3.55 µs 4.52 µs 27.51% -18.73%…34.67% 🟡 noisy
nitro-platform/variant/number-or-string 330.9 ns 641.8 ns 93.97% 40.91%…140.49% 🔴 regression
turbo-module/control/add-numbers 1.79 µs 1.80 µs 0.19% -10.53%…17.71% 🟡 noisy

Base 8ed1f9be · Head 144e2c61 · lower is better

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant