ci: add dedicated Release benchmark app and performance reporting

[mrousavy]

Stack

Depends on #1582 (example move), which depends on #1585 (isolated Xcode compatibility fix). Version alignment stays in the separate follow-up #1584.

Summary

• Add apps/benchmark: a minimal Release/Hermes app without Harness or navigation dependencies. apps/example keeps demos and Harness correctness tests.
• Reuse the real C++, Swift/Kotlin, and TurboModule bindings for 40 metrics. No production Nitro API or implementation changes in this PR diff.
• Calibrate each sample toward 150 ms of timed work (roughly 100–200 ms), with round iteration counts to two significant digits. Recheck calibration after five warmups, then freeze the count for twenty measured samples. Keep measured outliers.
• Replace allocation-limited tiny samples with bounded chunks whose timed durations are summed. GC, native cleanup, checks, and transport stay outside timing. Kotlin buffer-copy and Promise cases also collect Java references through a synchronous benchmark-only helper.
• Install each binary once per suite and start a fresh process for each case, releasing runtime-scoped JSI reference bookkeeping between cases. Startup is not timed. Preserve raw samples, iteration counts, and chunk sizes.
• Keep paired ABBA comparisons, run-aware bootstrap uncertainty, raw artifacts, and fork-safe Bencher/PR reporting. Version-2 definitions require a new baseline.
• Keep CV visible as a noise signal, but do not let high absolute variance hide a change whose full paired confidence interval is already beyond the budget. The canary exposed and now covers this policy edge case.
• Same-repository paired PRs now publish before merge from a separate clean job using pinned, reviewed reporting code. Forks remain secret-free and use the default-branch reporter after merge.
• Upload both measured platform baselines under an exact-SHA baseline branch before either PR head; this supports an empty Bencher project and stacked PRs. Do not reset the branch per platform, which would discard the first platform's reports. Pin the Bencher action and CLI version, and verify the reviewed Linux executable SHA-256 before exposing the API key.

Validation

• Android Release APK with Hermes/R8 and iOS Release simulator build pass.
• Final local iOS run: all 40 cases completed, with all median timed batches between 102 and 163 ms. 784/800 individual samples were in 100–200 ms; outliers were retained. In-app suite work totaled 228 seconds, excluding process startup.
• Final local Android run: all 40 cases completed, including the previously failing Java buffer/Promise cases, with all median timed batches between 115 and 157 ms. 794/800 individual samples were in 100–200 ms; outliers were retained. In-app suite work totaled 240 seconds, excluding process startup.
• These are local smoke checks, not CI baselines. Android used API 34 arm64 with HVF rather than CI API 36 x86_64 with KVM; iOS used Xcode 26.6 and the iOS 26.5 simulator.
• 50 performance tooling tests, performance-tool TypeScript, benchmark-app TypeScript/ESLint, and diff checks pass. The pinned Bencher CLI accepts real result JSON in dry-run mode; the downloaded Linux binary matches the pinned digest.
• Draft test: demonstrate a HybridFunction performance regression (DO NOT MERGE) #1588, stacked above ci: check app dependency version alignment #1584, adds an intentional native hot-path slowdown for actual end-to-end reporting validation. Its first paired run completed on both platforms, produced validated artifacts, and posted the combined PR comment. The raw data exposed the high-CV verdict issue fixed here. Redundant infrastructure-only performance runs were cancelled to prioritize it; correctness checks were left alone.

Rollout

The initial PR uses the head binary on both sides (A/A), because its base predates the benchmark app. It does not measure this PR versus main; the current BASE label/SHA on bootstrap A/A artifacts is misleading and still needs correction. The pre-merge publisher deliberately excludes these bootstrap runs. Later PRs, including #1588, build each revision independently and publish true paired results.

Verdicts remain advisory. Same-repository paired PR comments can publish now; fork reporting activates once workflow_run is on main. Rotate the exposed credential before setting NITRO_BENCHER_ENABLED=true; no build/device job or fork job receives that secret. Bencher activation still awaits confirmation that the exposed credential has been replaced.

Supersedes #1581.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
nitro-docs	Ready	Preview	Sep 4, 2026 6:18pm UTC

[mrousavy]

The version-alignment guard is isolated in #1584. Local validation also completed all 40 benchmark cases × 20 samples on the standalone iOS Release/Hermes app; the receiver validated release metadata and checksums. Full paired GitHub CI is running.