Skip to content

About

Fixture apps for the Stim agent benchmark

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

6 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

stim-benchmarks

Fixture apps for the Stim agent benchmark. A coding agent gets one of these apps and a short task (launch it, make a native change, or fix a launch error), with Stim, with the platform's own tools, or with Expo's @expo/agent-cli, and the benchmark records how long it takes to prove the result on a simulator or emulator. The harness and the protocol live in the Stim repository: docs/agent-benchmark.md and docs/benchmark-fixtures.md.

Fixtures

Directory App
expo/ A calculator on Expo SDK 58 with a development build (expo-dev-client) and Expo Router
bare/ The same calculator on bare React Native 0.88 with React Navigation, no Expo packages
exact/stim-bench-exact An Exact app with a counter, iOS only

The calculator has three tabs: Calculator, History (results saved with MMKV) and Settings (two switches). expo/ and bare/ share their src/ code except src/platform.tsx (haptics, icons, version) and the root component. They carry the native modules a small production app has, each used by the UI: Reanimated with worklets, Gesture Handler, Screens, Safe Area Context, SVG, MMKV with Nitro Modules, an icon font, and haptics. expo/ adds expo-dev-client, expo-router, expo-font, expo-splash-screen, expo-constants, expo-status-bar and expo-haptics; bare/ uses @react-native-vector-icons/ionicons and react-native-haptic-feedback. The apps make no network requests, collect nothing, and need no account.

Both React Native apps install as com.appandflow.stimbench (product StimBench), so only one of them is on a device at a time. Their ios/ and android/ projects are committed: expo/ was generated by expo prebuild, and the benchmark's native-change task edits these files.

The standalone native fixtures (an Xcode app, a SwiftPM macOS app and a Gradle Android app) are not here: the harness generates them with scripts/agent-benchmark/native-fixtures.mjs in the Stim repository.

The fixture contract

Each React Native directory has a fixture.json the harness reads instead of hard-coding names:

  • kind: expo or bare.
  • name: names control devices and the native-change marker (StimBench <run-id>).
  • ios.bundleId, ios.product, android.applicationId.
  • ios.nativeChange, android.nativeChange: the file and the line the native-change task edits. iOS sets the window's accessibility identifier in SceneDelegate.swift; Android changes app_name.
  • rootComponent: the root component's file, name and declaration. The error-fix task inserts a throw as the first line of that function, so the app fails on launch with an error that names the file and component.
  • proof: the Settings row the proof waits for (Key haptics), its subtitle, and the subtitle the app-launch task changes it to.

exact/fixture.json describes the Exact app: the folder it lives in, the file with the native-revision-one text the native task replaces, and the line after which the error-fix task inserts a throw. Exact app code cannot end the process at launch, so that error shows The app failed to start. on screen and its message only in the device log.

Pins a benchmark root needs

  • FIXTURES_COMMIT: the commit of this repository.
  • FIXTURE_DIR: expo or bare for the React Native rows; Exact uses its own root and reads exact/fixture.json.
  • Node.js 22, Xcode with an iOS 27 simulator runtime, CocoaPods 1.15.2 from each app's Gemfile.lock (bundle exec pod install in ios/), and the Android SDK with an arm64 system image.
  • Exact: Bun 1.4.2 at ~/.bun-1.4.2, the exact2 checkout at ~/.exact/exact2/4756891bc2a0440eadd9a8e3ec67e2a8c04802cf (EXACT2 overrides it), and Rust 1.97.0 with aarch64-apple-ios-sim through rustup. Because exact.mjs finds exact2 through the home directory rather than a literal path, Stim builds the Exact app on the machine that runs it and does not offload that build.

Clone this repository on its own (not as a worktree of another repository), run npm ci in expo/ and bare/, then bundle install and bundle exec pod install in their ios/ directories.

License

MIT, see LICENSE. exact/stim-bench-exact was created with exact new and keeps Exact's MIT license in its folder.

About

Fixture apps for the Stim agent benchmark

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages