Fixture apps for the Stim agent
benchmark. A coding agent gets one of these apps and a short task (launch it,
make a native change, or fix a launch error), with Stim, with the platform's
own tools, or with Expo's @expo/agent-cli, and the benchmark records how long
it takes to prove the result on a simulator or emulator. The harness and the
protocol live in the Stim repository:
docs/agent-benchmark.md
and
docs/benchmark-fixtures.md.
| Directory | App |
|---|---|
expo/ |
A calculator on Expo SDK 58 with a development build (expo-dev-client) and Expo Router |
bare/ |
The same calculator on bare React Native 0.88 with React Navigation, no Expo packages |
exact/stim-bench-exact |
An Exact app with a counter, iOS only |
The calculator has three tabs: Calculator, History (results saved with MMKV)
and Settings (two switches). expo/ and bare/ share their src/ code except
src/platform.tsx (haptics, icons, version) and the root component. They carry
the native modules a small production app has, each used by the UI:
Reanimated with worklets, Gesture Handler, Screens, Safe Area Context, SVG,
MMKV with Nitro Modules, an icon font, and haptics. expo/ adds
expo-dev-client, expo-router, expo-font, expo-splash-screen,
expo-constants, expo-status-bar and expo-haptics; bare/ uses
@react-native-vector-icons/ionicons and react-native-haptic-feedback. The
apps make no network requests, collect nothing, and need no account.
Both React Native apps install as com.appandflow.stimbench (product
StimBench), so only one of them is on a device at a time. Their ios/ and
android/ projects are committed: expo/ was generated by expo prebuild,
and the benchmark's native-change task edits these files.
The standalone native fixtures (an Xcode app, a SwiftPM macOS app and a Gradle
Android app) are not here: the harness generates them with
scripts/agent-benchmark/native-fixtures.mjs in the Stim repository.
Each React Native directory has a fixture.json the harness reads instead of
hard-coding names:
kind:expoorbare.name: names control devices and the native-change marker (StimBench <run-id>).ios.bundleId,ios.product,android.applicationId.ios.nativeChange,android.nativeChange: the file and the line the native-change task edits. iOS sets the window's accessibility identifier inSceneDelegate.swift; Android changesapp_name.rootComponent: the root component's file, name and declaration. The error-fix task inserts athrowas the first line of that function, so the app fails on launch with an error that names the file and component.proof: the Settings row the proof waits for (Key haptics), its subtitle, and the subtitle the app-launch task changes it to.
exact/fixture.json describes the Exact app: the folder it lives in, the file
with the native-revision-one text the native task replaces, and the line after
which the error-fix task inserts a throw. Exact app code cannot end the
process at launch, so that error shows The app failed to start. on screen and
its message only in the device log.
FIXTURES_COMMIT: the commit of this repository.FIXTURE_DIR:expoorbarefor the React Native rows; Exact uses its own root and readsexact/fixture.json.- Node.js 22, Xcode with an iOS 27 simulator runtime, CocoaPods 1.15.2 from
each app's
Gemfile.lock(bundle exec pod installinios/), and the Android SDK with an arm64 system image. - Exact: Bun 1.4.2 at
~/.bun-1.4.2, the exact2 checkout at~/.exact/exact2/4756891bc2a0440eadd9a8e3ec67e2a8c04802cf(EXACT2overrides it), and Rust 1.97.0 withaarch64-apple-ios-simthrough rustup. Becauseexact.mjsfinds exact2 through the home directory rather than a literal path, Stim builds the Exact app on the machine that runs it and does not offload that build.
Clone this repository on its own (not as a worktree of another repository),
run npm ci in expo/ and bare/, then bundle install and
bundle exec pod install in their ios/ directories.
MIT, see LICENSE. exact/stim-bench-exact was created with exact new and
keeps Exact's MIT license in its folder.