browser-agents-benchmark

Reproducible head-to-head benchmark of browser-automation services across 100 real-world web tasks — scraping, multi-step navigation, login flows, form fills, pagination.

pre.dev Browser Agents and Browser Use Cloud both pass 100 / 100 this run — pre.dev wins ~2.6× on cost AND ~5.1× on speed (median).

---

Headline result

Provider	Pass rate	Median time / task	Avg time / task	$ / task	Total $ for 100 tasks
🏆 pre.dev Browser Agents	100 / 100	4.9 s	9.1 s	$0.0159	$1.59
Browser Use Cloud	100 / 100	25.0 s	35.7 s	$0.0415	$4.15

Same 100 tasks. Same JSON output schemas. Same uniform successCheck predicate. Headline speed is the median wall time per task: on the cheap tiers a handful of hard anti-bot sites hit a 20–80 s unlock/CAPTCHA-clear path that drags the mean well above the typical task, so the median is the honest representative number (the mean is shown alongside, and the full report page carries both). Browser agents are stochastic — individual runs on cheap-tier models vary by a few tasks per suite; the full per-task JSON + trace from this run is in results/2026-07-08-head-to-head/ so the data can be re-scored independently.

Cost per task

Median time per task

👉 Full interactive report — radar chart, leaderboard, per-task drilldown, raw JSON.

Also in this repo: results/2026-07-08-head-to-head/REPORT.md · results/2026-07-08-head-to-head/report.html (raw HTML — open locally to view).

---

Reproduce

git clone https://github.com/predotdev/browser-agents-benchmark
cd browser-agents-benchmark
bun install                  # or: npm install
cp .env.example .env         # then fill PREDEV_API_KEY + BROWSER_USE_API_KEY
bun run bench                # full 100-task suite, parallel, ~3-5 min
bun run report <runStamp>    # generate REPORT.md + report.html

API keys:

• pre.dev → https://pre.dev/projects/playground (suite runs ~$1 on an active Plus/Premium/Pro plan; free Trial only covers 1 task)
• Browser Use Cloud → https://cloud.browser-use.com/api-keys (suite runs ~$4 on bu-mini; check their current free credit allotment)

Narrow runs

tsx run.ts --config predev                 # one provider only
tsx run.ts --task 06-coingecko             # one task (prefix match)
tsx run.ts --limit 25 --parallel --concurrency 10

---

Methodology

1. 100 tasks in tasks.json — static scrapes, multi-page navigation, logins, form submissions, search → click → extract, dynamic JS pages, iframes, and CAPTCHA-protected sites.
2. Same input contract for every provider: url, instruction, optional input (form values), and output JSON Schema. pre.dev receives the schema via the output field on /browser-agent; browser-use receives it via the SDK's outputSchema option.
3. One attempt per task per provider. No adapter retries on empty output — the first run is the only run. (429 rate-limit backoff is allowed, since that's a throttling signal, not a task-level failure.)
4. Uniform successCheck predicate per task (tasks.json). It accepts any JSON shape containing the requested information — no provider's response format is privileged.
5. Cost is captured from each provider's response payload (or estimated at published rates). Wall time is measured by the harness.
6. Per-task JSON is written to results/<runStamp>/<configId>/<taskName>.json, a summary.json rolls them up, and report.ts generates a markdown + HTML report.

Models used

• pre.dev: gemini-2.5-flash-lite (the default low-cost tier on pre.dev Browser Agents)
• browser-use Cloud: bu-mini (browser-use's named budget tier on cloud.browser-use.com)

Both are each provider's default cheapest named tier, so this is a default-vs-default run. One caveat we disclose up front: despite the "mini" name, browser-use's dashboard shows bu-mini backed by a Claude-family model, not a Gemini/GPT-mini — so the cost gap partly reflects that pricier backend, not just efficiency. For a strict same-family cost comparison, browser-use's gemini-3-flash is the closest sibling to pre.dev's Gemini Flash Lite (set cfg.model in run.ts and rerun); that narrows the cost multiple but the speed and accuracy results hold regardless of tier.

Note on the SDK model parameter. The official browser-use-sdk v3 client expects model, not llm, in the run-options. Our adapter passed llm through v0.1.0, which browser-use's backend silently dropped — falling back to the server default (Claude Opus 4.7 at the time), NOT the bu-mini we thought we were selecting. We've since fixed the adapter so the model param actually reaches their router; the numbers above are all from post-fix runs.

Headline speed is the median wall time across all 100 tasks (mean shown alongside). The median is reported because the mean on both providers is skewed by a small tail of hard anti-bot sites that fall back to a slow unlock/CAPTCHA-clear path; the median reflects the typical task. Both the median and the mean are in the report.

The numbers in this README come from results/2026-07-08-head-to-head/. That directory is committed so anyone can audit the raw data without re-running.

---

Repo layout

browser-agents-benchmark/
├── README.md
├── .env.example
├── tasks.json                     # 100 benchmark tasks (self-contained)
├── tasks.ts                       # task loader + successCheck rehydration
├── types.ts
├── run.ts                         # bench runner
├── report.ts                      # markdown + HTML report generator
├── adapters/
│   ├── predev.ts
│   └── browser-use.ts
├── scripts/
│   └── generate-tasks.ts          # regenerate tasks.json from upstream
├── docs/
│   └── *.svg                      # charts embedded in this README
└── results/
    └── 2026-07-08-head-to-head/    # the run powering this README
        ├── summary.json
        ├── REPORT.md
        ├── report.html            # open in a browser for the interactive view
        ├── predev/<taskName>.json
        └── browser-use-cloud/<taskName>.json

---

Adding your own competitor

Drop a file in adapters/ implementing:

export async function runYours(task: BenchmarkTask, cfg: YourConfig): Promise<BenchmarkResult>;

Register it in run.ts's ALL_CONFIGS array. The harness will run it against every task and fold the results into the report.

---

License

MIT. Use this to validate any vendor's claims — including ours.