From 9cd670a9bc1bd94bb7b2e9fdae9061f5bbc39241 Mon Sep 17 00:00:00 2001 From: Ashley Zhang <69987606+ashleyzhang01@users.noreply.github.com> Date: Mon, 24 Aug 2026 16:45:43 -0700 Subject: [PATCH 1/6] update methodology with offline changes --- methodology/full.md | 17 ++++++++++------- methodology/summary.md | 2 +- 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/methodology/full.md b/methodology/full.md index 71f9c18..3ade032 100644 --- a/methodology/full.md +++ b/methodology/full.md @@ -145,13 +145,16 @@ We plot these statistics over time to track trends in tool performance and usage ### Offline benchmark -We build on the Augment/Greptile dataset. For each PR in the dataset, we: +We build on the Augment/Greptile dataset, with a curated gold set of **173 golden comments** across 50 PRs. Each golden comment has a severity level and an issue-type category. + +For each PR in the dataset, we: 1. Fork a copy of the repository for each code review tool being evaluated. 2. Open a PR that includes the description from the original human-authored PR. 3. Trigger the code review tool in the forked repo on GitHub. 4. Collect all issues identified by the tool, splitting multi-issue comments into individual issues. -5. Run a judge to evaluate which tool-identified issues correspond to issues in the gold set. +5. Deduplicate candidates, grouping those that express the same underlying concern so duplicate mentions are not penalized. +6. Run a judge to evaluate which tool-identified issues correspond to issues in the gold set. The judge uses the following prompt: @@ -168,19 +171,19 @@ The judge uses the following prompt: > > Respond with ONLY a JSON object: {"reasoning": "brief explanation", "match": true/false, "confidence": 0.0-1.0} -Matches against the gold set are counted as true positives. Precision and recall are computed from these matches. +Matches against the gold set are counted as true positives. Precision and recall are computed from these matches, with category-based scoring profiles and F-beta weighting available in the dashboard. Results are evaluated with three independent judge models (Claude Opus 4.5, GPT-5.2, Claude Sonnet 4.5); the top 5 tools are identical across all three judges. The code to reproduce this is available at: https://github.com/withmartian/code-review-benchmark ### Known limitations of the current implementation -The current implementation is deliberately minimal — a starting point we can measure and improve against. The main limitations, each addressed by a subsequent section of this document: +The current implementation is under active improvement but retains significant limitations, each addressed by a subsequent section of this document: -- Bug definitions are implicit in the gold set rather than explicit or conditioned on user preferences (§5). +- Bug definitions are semi-explicit through categories but not conditioned on user preferences (§5). - Recall is capped by the gold set, which is itself capped by human performance (§6). -- Precision treats all non-action as false positives (§7). +- Scoring profiles reduce false positive penalties for style/speculative findings, but the gold set has sparse coverage of style issues, so some correct style findings are still penalized (§7). - The dataset is static and based on older PRs, with contamination risk (§8). -- The judge is a single LLM with a single prompt, with no calibration against human annotations (§9). +- The judge uses three models with consistent rankings, but has not been formally calibrated against human annotations (§9). - There is no standardized harness — we're measuring product performance, not model performance (§9). --- diff --git a/methodology/summary.md b/methodology/summary.md index f2d5ef9..e39c1b3 100644 --- a/methodology/summary.md +++ b/methodology/summary.md @@ -18,7 +18,7 @@ The gap between measure and intent is the problem. To close it, you need two thi Our benchmark has two pieces: -**An offline benchmark** runs every tool on the same PRs with the same bug definitions and scores them against a curated gold set. This is the measure — it lets us make controlled comparisons. The v0 builds on Augment and Greptile's published dataset and improves from there. +**An offline benchmark** runs every tool on the same PRs with the same bug definitions and scores them against a curated gold set. This is the measure — it lets us make controlled comparisons. The current version builds on Augment and Greptile's published dataset with a curated gold set of 173 golden comments across 50 PRs, each categorized (bug, security, concurrency, etc.) and scored using category-based profiles and F-beta weighting. **An online benchmark** teaches us user intent based on how developers actually use these tools. When a developer fixes a problem flagged by a tool, they're voting that the flag was useful. These behavioral signals aren't controlled by us or by vendors — they're anchored to what actually happens. From 2aba31a25916ef8d6dc664473ecc44a03b71d7c2 Mon Sep 17 00:00:00 2001 From: Ashley Zhang <69987606+ashleyzhang01@users.noreply.github.com> Date: Mon, 24 Aug 2026 16:45:49 -0700 Subject: [PATCH 2/6] update readme --- README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index b111c57..f6ee724 100644 --- a/README.md +++ b/README.md @@ -42,9 +42,9 @@ Without shared evals for these tools, every company grades its own homework. You | [Discourse](https://github.com/discourse/discourse) | Ruby | Forum platform | | [Keycloak](https://github.com/keycloak/keycloak) | Java | Authentication | -Each PR has curated golden comments with severity labels (Low / Medium / High / Critical). An LLM judge matches each tool's review against the golden comments and computes precision and recall. +Each PR has curated golden comments (173 total) with severity labels (Low / Medium / High / Critical) and category tags (bug, security, concurrency, data, api, perf, test_gap, doc_defect, style, speculative). An LLM judge matches each tool's review against the golden comments using three judge models (Claude Opus 4.5, GPT-5.2, Claude Sonnet 4.5). Category-based scoring profiles (Strict / Core / All) control which issue types count toward the score, and F-beta weighting lets users prioritize recall over precision. -**Tools evaluated**: Augment, Claude Code, CodeRabbit, Codex, Cursor Bugbot, Gemini, GitHub Copilot, Graphite, Greptile, Propel, Qodo, and more. Adding a new tool takes an afternoon — fork the benchmark PRs, trigger the tool, run the pipeline. +**Tools evaluated**: Augment, Baz, Claude Code, CodeAnt, CodeRabbit, Cubic, Cursor Bugbot, Devin, Gemini, GitHub Copilot, GitLab Duo, Graphite, Greptile, KG, Kodus, Macroscope, Qodo, Sourcery, and more. Adding a new tool takes an afternoon — fork the benchmark PRs, trigger the tool, run the pipeline. > **Known limitation**: Static datasets risk training data leakage — tools may have seen these PRs during training. That's why we also run the online benchmark. @@ -84,14 +84,15 @@ Both benchmarks use an LLM-as-judge approach, but with different methodologies s | | Offline | Online | |---|---|---| -| **Ground truth** | Human-curated golden comments | Developer's post-review fixes | +| **Ground truth** | Human-curated golden comments (173, categorized) | Developer's post-review fixes | | **Precision** | Tool comments that match a golden comment / total tool comments | Bot suggestions matched to real fixes / total suggestions | -| **Recall** | Golden comments found by the tool / total golden comments | Real fixes caught by the bot / total fixes made | +| **Recall** | Golden comments in active profile found by the tool / total golden in profile | Real fixes caught by the bot / total fixes made | +| **Scoring** | Category-based profiles (Strict/Core/All) + F-beta (0.5–3.0) | F-beta with adjustable weighting | | **Judge input** | Golden comment + tool candidate | Full PR timeline: diff, bot comments, post-review commits | In both cases, the judge prompt asks "do these describe the same underlying issue?" — different wording is fine, only the substance matters. -**Judge model variance**: Different LLM judges can score differently. We mitigate this by storing results per judge model and reporting which model was used. The offline benchmark has been evaluated with Claude Opus 4.5, Claude Sonnet 4.5, and GPT-5.2. +**Judge model variance**: Different LLM judges can score differently. We mitigate this by storing results per judge model and reporting which model was used. The offline benchmark has been evaluated with Claude Opus 4.5, Claude Sonnet 4.5, and GPT-5.2 — the top 5 tools are identical across all three judges, with most tools varying by at most 2 rank positions. ## Repository structure @@ -135,7 +136,7 @@ uv run python -m code_review_benchmark.step2_5_dedup_candidates # Run the LLM judge (pass dedup groups to avoid penalising duplicate candidates) uv run python -m code_review_benchmark.step3_judge_comments \ - --dedup-groups results/$(MARTIAN_MODEL)/dedup_groups.json + --dedup-groups results/${MARTIAN_MODEL}/dedup_groups.json # View results open analysis/benchmark_dashboard.html From 00ace25d3941ecc50553f16d85161e5564ee3808 Mon Sep 17 00:00:00 2001 From: Ashley Zhang <69987606+ashleyzhang01@users.noreply.github.com> Date: Mon, 24 Aug 2026 17:31:44 -0700 Subject: [PATCH 3/6] update methodology and readmes with online changes --- README.md | 11 ++++++----- methodology/full.md | 12 +++++++----- online/README.md | 17 ++++++++++------- online/api_service/README.md | 21 +++++++++++++++++---- 4 files changed, 40 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index f6ee724..80e0e82 100644 --- a/README.md +++ b/README.md @@ -55,15 +55,16 @@ See [`offline/README.md`](offline/README.md) for setup and usage. The online benchmark continuously samples **fresh real-world PRs from GitHub** where code review bots left comments. Because the PRs are recent, tools can't have memorized them during training. ``` -GitHub Archive (BigQuery) +GitHub Search API │ ▼ ┌────────┐ ┌─────────┐ ┌─────────┐ ┌────┐ ┌───────────┐ │Discover│────▶│ Enrich │────▶│ Analyze │────▶│ DB │────▶│ Dashboard │ └────────┘ └─────────┘ └─────────┘ └────┘ └───────────┘ - BigQuery scan GitHub API LLM 3-step Postgres Interactive - finds bot PRs fetches full extraction & or SQLite filters & - PR context matching time series + Search API GitHub API LLM 3-step Postgres Interactive + finds merged fetches full extraction & or SQLite filters & + bot-reviewed PR context matching time series + PRs ``` **How analysis works**: @@ -74,7 +75,7 @@ GitHub Archive (BigQuery) **Bots tracked**: CodeRabbit, GitHub Copilot, Claude, Cursor, Augment, Codex, Gemini, Greptile, Graphite, Qodo, Propel, and others. -**Dashboard features**: Filter by language, project domain, PR type, issue severity, diff size. Track performance over time. Adjustable F-beta weighting. +**Dashboard features**: Filter by language, project domain, PR type, issue severity, diff size, engagement signals (human comments/commits after bot review), solo-bot PRs, and sample controls. Track performance over time. Adjustable F-beta weighting. See [`online/FILTERS.md`](online/FILTERS.md) for the full filter spec. See [`online/README.md`](online/README.md) for architecture and setup. diff --git a/methodology/full.md b/methodology/full.md index 3ade032..a58d20f 100644 --- a/methodology/full.md +++ b/methodology/full.md @@ -119,11 +119,9 @@ This section describes what we've built and deployed today. The methodology sect ### Online benchmark -Each day, we collect events from code review tools using the GHArchive dump, searching for events containing the agent ID associated with each tool. We group these events into PRs. +Each day, we discover merged PRs where tracked bots left reviews, using the GitHub Search API. We restrict to merged PRs because unmerged PRs rarely have enough post-review signal to score meaningfully. For each tool, we randomly sample PRs and cap per-repo to prevent any single project from dominating the sample. High-volume bots may produce more PRs than the API can return in a single query; we handle this transparently so the sample remains representative. -We filter to projects with more than 1,000 PRs, to avoid low-volume projects where behavioral signals are likely to be noisier. We plan to revisit this threshold — smaller projects may still provide useful data, but we need to examine where they do and don't before including them. - -For each tool, we randomly sample PRs each day. If the sample size required to compute a statistically significant mean would exceed 10% of the population, we compute the population mean directly by pulling all examples. +For each PR, we fetch the full context — commits, reviews, threaded discussions, and per-commit diffs — and assemble a unified chronological timeline. An LLM then performs a three-step analysis: extract what the bot suggested, extract what the developer actually changed after the review, and judge which suggestions correspond to real fixes. We compute the following metrics for each tool: @@ -139,10 +137,14 @@ We compute the following metrics for each tool: - Total comments acted on *Distributional controls:* -- Distribution of diff size, repo size, language, backend vs. frontend, and other repository and PR characteristics across tools, to identify whether tools are operating on comparable datasets +- Distribution of diff size, language, project domain, PR type, issue severity, and other repository and PR characteristics across tools, to identify whether tools are operating on comparable datasets We plot these statistics over time to track trends in tool performance and usage patterns. +Not all PRs are equally informative. A PR where no human ever looked at the review — no comments, no follow-up commits — tells us little about whether the bot's suggestions were useful. We compute engagement signals for each PR: whether humans commented or pushed commits after the bot's first review, how many distinct reviewers participated, and how many back-and-forth rounds occurred. These signals power a set of quality filters that let us progressively restrict to higher-signal subsets. By default, we exclude bot-authored PRs (where the "developer response" is another bot) and self-reviews (the bot reviewing its own PR). Users can also restrict to solo-bot PRs — where only one bot reviewed — to simplify attribution of human follow-up. + +Each of these filters is exposed individually in the dashboard, so users can combine them based on what they care about — requiring human engagement, capping per-author-per-repo contributions, setting minimum contributor diversity, and so on. Stricter filter combinations trade coverage for signal quality: fewer PRs, but more reliable scores. + ### Offline benchmark We build on the Augment/Greptile dataset, with a curated gold set of **173 golden comments** across 50 PRs. Each golden comment has a severity level and an issue-type category. diff --git a/online/README.md b/online/README.md index 8c45886..f83d124 100644 --- a/online/README.md +++ b/online/README.md @@ -7,20 +7,21 @@ The online benchmark solves this by **continuously sampling fresh PRs from GitHu ## How it works ``` -GitHub Archive (BigQuery) +GitHub Search API │ ▼ ┌────────┐ ┌─────────┐ ┌──────────┐ ┌─────────┐ ┌────┐ ┌───────────┐ │Discover│────▶│ Enrich │────▶│ Assemble │────▶│ Analyze │────▶│ DB │────▶│ Dashboard │ └────────┘ └─────────┘ └──────────┘ └─────────┘ └────┘ └───────────┘ - BigQuery scan GitHub API Build unified LLM 3-step Postgres Interactive - finds bot PRs fetches full PR timeline extraction & or SQLite filters & - PR context matching time series + Search API GitHub API Build unified LLM 3-step Postgres Interactive + finds merged fetches full PR timeline extraction & or SQLite filters & + bot-reviewed PR context matching time series + PRs ``` ### 1. Discover -A BigQuery scan of [GitHub Archive](https://www.gharchive.org/) finds PRs where tracked code review bots left comments. Sampling is deterministic (FARM_FINGERPRINT-based) so runs are reproducible. Up to 500 PRs per bot per day. +The GitHub Search API (`reviewed-by: is:merged`) finds merged PRs where tracked code review bots left reviews. For high-volume bots that exceed the 1,000-result API cap, adaptive time bisection splits the query window until each chunk fits. PRs are randomly sampled and capped per-repo to prevent any single project from dominating. Up to 500 PRs per bot per day. A BigQuery / [GitHub Archive](https://www.gharchive.org/) fallback is available via `--source bq`. ### 2. Enrich @@ -60,14 +61,16 @@ The dashboard shows tool performance over time with filters for: - **Severity**: low, medium, high, critical - **Diff size**: min/max lines changed - **F-beta**: adjustable weighting between precision and recall +- **Engagement**: require human engagement (comments/commits after the bot's review), solo-bot PRs, minimum reviewer count +- **Sample controls**: min scored PRs, min PRs per day, max PRs per (repo, author, bot) triple -Visualizations include time series of F-beta scores, precision/recall scatter plots, and a filterable leaderboard. +Visualizations include time series of F-beta scores, precision/recall scatter plots, and a filterable leaderboard. See [`FILTERS.md`](FILTERS.md) for the full filter spec. ## Components | Directory | What | Stack | |---|---|---| -| [`etl/`](etl/) | Data pipeline: discover, enrich, analyze, label | Python, asyncio, BigQuery, OpenAI API | +| [`etl/`](etl/) | Data pipeline: discover, enrich, analyze, label | Python, asyncio, GitHub Search API, OpenAI API | | [`api_service/`](api_service/) | Public dashboard server | Rust, Axum, Plotly.js | See each subdirectory's README for setup and usage details. diff --git a/online/api_service/README.md b/online/api_service/README.md index a157519..a11e40a 100644 --- a/online/api_service/README.md +++ b/online/api_service/README.md @@ -40,10 +40,12 @@ docker run -e DATABASE_URL=postgresql://... -p 3000:3000 pr-review-api | `GET` | `/api/options` | Available filter options (chatbots, languages, domains, etc.) | | `GET` | `/api/daily-metrics` | Daily time-series metrics with filtering | | `GET` | `/api/leaderboard` | Chatbot leaderboard with filtering | -| `GET` | `/api/volumes` | PR volume per tool per day (formal reviews from BigQuery) | +| `GET` | `/api/volumes` | PR volume per tool per day | ### Filter Parameters (for `/api/daily-metrics` and `/api/leaderboard`) +See [`../FILTERS.md`](../FILTERS.md) for the full spec with scopes, defaults, and composition examples. + | Parameter | Description | |---|---| | `start_date` | Start date (`YYYY-MM-DD`) | @@ -56,12 +58,23 @@ docker run -e DATABASE_URL=postgresql://... -p 3000:3000 pr-review-api | `diff_lines_min` | Minimum diff lines | | `diff_lines_max` | Maximum diff lines | | `beta` | F-beta score beta parameter (default: 1.0) | -| `min_total_prs` | Minimum total PRs to include a chatbot in leaderboard/scatter | -| `min_prs_per_day` | Minimum PRs per day to include in time series | +| `min_total_prs` | Minimum total PR volume (BigQuery count) to include a bot | +| `min_scored_prs` | Minimum scored PRs (after all PR-level filters) to include a bot | +| `min_prs_per_day` | Minimum scored PRs per day to include a day in time series | +| `exclude_bot_authored` | Exclude PRs authored by bots (`true`/`false`) | +| `exclude_self_authored` | Exclude PRs where the bot reviews its own PR (`true`/`false`) | +| `require_solo_bot` | Keep only PRs scored by exactly one bot (`true`/`false`) | +| `require_human_engagement` | Require human comments or commits after bot review (`true`/`false`) | +| `require_reviews` | Require a formal GitHub review event (`true`/`false`) | +| `min_human_reviewers` | Minimum distinct non-bot, non-author commenters after bot review | +| `min_commits_after_review` | Minimum commit events after bot's first review | +| `min_repo_contributors` | Minimum distinct PR authors in the repo | +| `max_author_repo_prs` | Cap PRs per (repo, author, bot) triple via deterministic random sample | +| `include_ignored` | Include bots flagged as ignored in the registry (`true`/`false`) | ### `/api/volumes` Parameters -Returns the number of unique PRs each tool first interacted with per day, from GitHub Archive. Each PR is counted exactly once on the earliest day the bot touched it, so summing daily counts gives the true unique total (no double-counting PRs that span multiple days). Only date range and chatbot filters apply — label filters are not relevant since this is raw BigQuery count data, not analyzed data. Days where a bot had no activity are zero-filled in the response so the chart shows gaps correctly. +Returns the number of unique PRs each tool first interacted with per day. Each PR is counted exactly once on the earliest day the bot touched it, so summing daily counts gives the true unique total (no double-counting PRs that span multiple days). Only date range and chatbot filters apply — label filters are not relevant since this is raw count data, not analyzed data. Days where a bot had no activity are zero-filled in the response so the chart shows gaps correctly. Volume data is populated by the ETL `volumes` command (GitHub Search API by default, BigQuery as fallback). | Parameter | Description | |---|---| From 2493e037bb0c4484fbe8b634bc16f991ec708c02 Mon Sep 17 00:00:00 2001 From: Ashley Zhang <69987606+ashleyzhang01@users.noreply.github.com> Date: Mon, 24 Aug 2026 23:55:55 -0700 Subject: [PATCH 4/6] update volumes description in readme --- online/api_service/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/online/api_service/README.md b/online/api_service/README.md index a11e40a..08db86e 100644 --- a/online/api_service/README.md +++ b/online/api_service/README.md @@ -74,7 +74,7 @@ See [`../FILTERS.md`](../FILTERS.md) for the full spec with scopes, defaults, an ### `/api/volumes` Parameters -Returns the number of unique PRs each tool first interacted with per day. Each PR is counted exactly once on the earliest day the bot touched it, so summing daily counts gives the true unique total (no double-counting PRs that span multiple days). Only date range and chatbot filters apply — label filters are not relevant since this is raw count data, not analyzed data. Days where a bot had no activity are zero-filled in the response so the chart shows gaps correctly. Volume data is populated by the ETL `volumes` command (GitHub Search API by default, BigQuery as fallback). +Returns PR volume counts per tool per day. When populated via the GitHub Search API (default), counts are attributed to the PR's **creation date**. When populated via BigQuery (`--source bq`), counts are attributed to the first day the bot touched the PR. Only date range and chatbot filters apply — label filters are not relevant since this is raw count data, not analyzed data. Days where a bot had no activity are zero-filled in the response so the chart shows gaps correctly. | Parameter | Description | |---|---| From 63fd8e85799f508c1079ed1eb9a71179e1aad323 Mon Sep 17 00:00:00 2001 From: Ashley Zhang <69987606+ashleyzhang01@users.noreply.github.com> Date: Tue, 25 Aug 2026 00:15:39 -0700 Subject: [PATCH 5/6] update etl readme --- online/etl/README.md | 37 +++++++++++++++++++++++++------------ 1 file changed, 25 insertions(+), 12 deletions(-) diff --git a/online/etl/README.md b/online/etl/README.md index 4894934..1ee8be7 100644 --- a/online/etl/README.md +++ b/online/etl/README.md @@ -1,8 +1,8 @@ # PR Review Dataset — ETL Pipeline -Data pipeline for the [online code review benchmark](../README.md). Continuously discovers PRs that code review bots have commented on, fetches full PR data from GitHub, and uses an LLM to analyze whether the bot's suggestions identified real issues and whether developers acted on them. +Data pipeline for the [online code review benchmark](../README.md). Continuously discovers PRs that code review bots have reviewed, fetches full PR data from GitHub, and uses an LLM to analyze whether the bot's suggestions identified real issues and whether developers acted on them. -Discovers PRs reviewed by a chatbot (via BigQuery), enriches them with GitHub API data, assembles a unified timeline, and runs LLM analysis. Everything is stored in a database (SQLite or PostgreSQL). +Discovers merged PRs reviewed by a chatbot via the GitHub Search API (`reviewed-by: is:merged`), enriches them with GitHub API data, assembles a unified timeline, and runs LLM analysis. A BigQuery / GH Archive fallback is available via `--source bq`. Everything is stored in a database (SQLite or PostgreSQL). ## Setup @@ -13,9 +13,8 @@ cp .env.example .env # fill in values ``` You need: -- A GCP project with BigQuery access (for querying `githubarchive.day.*`) - A GitHub personal access token with `public_repo` read access -- `gcloud auth application-default login` (for BigQuery auth) +- (Optional) A GCP project with BigQuery access, only needed for `--source bq` fallback and `gcloud auth application-default login` ## Environment @@ -72,17 +71,20 @@ Both `asyncpg` (pipeline) and `psycopg` (dashboard) connect through the proxy tr All commands run from the `etl/` directory. -### Discover PRs from BigQuery +### Discover PRs ```bash -# Single chatbot +# All chatbots, last 7 days (default: GitHub Search API) +uv run python main.py discover --all --days-back 7 + +# Single chatbot, specific date range uv run python main.py discover \ --chatbot "coderabbitai[bot]" \ - --start-date 2024-01-01 \ - --end-date 2025-01-01 + --start-date 2026-06-01 \ + --end-date 2026-08-01 -# All chatbots in one BQ scan (uses DB chatbots, or built-in defaults) -uv run python main.py discover --all --days-back 7 +# Use BigQuery / GH Archive fallback +uv run python main.py discover --all --days-back 7 --source bq ``` ### Enrich PRs via GitHub API @@ -117,6 +119,12 @@ uv run python -m jobs.enrich_job \ ```bash uv run python main.py analyze --chatbot "coderabbitai[bot]" uv run python main.py analyze --all + +# Only PRs reviewed in the last 7 days +uv run python main.py analyze --all --since 7d + +# Specific date window +uv run python main.py analyze --all --since 2026-08-01 --until 2026-08-15 ``` ### Label PRs @@ -130,7 +138,7 @@ uv run python main.py label --chatbot "coderabbitai[bot]" --since 7d ### Fetch PR volumes from BigQuery ```bash -# All chatbots, last 7 days +# All chatbots, last 7 days (default: GitHub Search API) uv run python main.py volumes --all --days-back 7 # All chatbots, specific date range @@ -142,9 +150,14 @@ uv run python main.py volumes --all \ uv run python main.py volumes \ --chatbot "coderabbitai[bot]" \ --days-back 30 + +# Use BigQuery / GH Archive instead of Search API +uv run python main.py volumes --all --days-back 7 --source bq ``` -Queries `githubarchive.day.*` for all PR-related events (reviews, review comments, issue comments on PRs), assigns each unique PR to the first day the bot touched it, and upserts daily counts into the `pr_volumes` table. Each PR is counted exactly once so summing daily counts gives the true unique total — this ensures total PRs >= sampled PRs holds in the leaderboard. Days with no activity for a bot are zero-filled by the API service. +Default source is the GitHub Search API (`reviewed-by: is:merged`), which counts PRs where the bot left a formal review. Counts are attributed to the **PR creation date** (the Search API's `created:` qualifier), so a PR created on June 1 but reviewed on June 15 appears under June 1. Use `--source bq` for the legacy BigQuery/GH Archive path, which instead attributes each PR to the first day the bot touched it. The `--weekly` flag (Search API only) queries in weekly chunks for fewer API calls at the cost of less granular per-day attribution. + +Note: because the Search API attributes by creation date while the scored dataset uses `bot_reviewed_at`, the volume totals and scored-PR counts may not be directly comparable on a per-day basis — but aggregate totals remain meaningful for the `min_total_prs` leaderboard filter. Days with no activity for a bot are zero-filled by the API service. ### Import legacy filesystem data From 65b28ea47e0ca258fcbbe4754ca1f4c83cfeee59 Mon Sep 17 00:00:00 2001 From: Ashley Zhang <69987606+ashleyzhang01@users.noreply.github.com> Date: Tue, 25 Aug 2026 15:16:42 -0700 Subject: [PATCH 6/6] add inclusion criteria --- README.md | 20 +++++++++++++++++--- offline/README.md | 2 +- 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 80e0e82..333ab57 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ Without shared evals for these tools, every company grades its own homework. You Each PR has curated golden comments (173 total) with severity labels (Low / Medium / High / Critical) and category tags (bug, security, concurrency, data, api, perf, test_gap, doc_defect, style, speculative). An LLM judge matches each tool's review against the golden comments using three judge models (Claude Opus 4.5, GPT-5.2, Claude Sonnet 4.5). Category-based scoring profiles (Strict / Core / All) control which issue types count toward the score, and F-beta weighting lets users prioritize recall over precision. -**Tools evaluated**: Augment, Baz, Claude Code, CodeAnt, CodeRabbit, Cubic, Cursor Bugbot, Devin, Gemini, GitHub Copilot, GitLab Duo, Graphite, Greptile, KG, Kodus, Macroscope, Qodo, Sourcery, and more. Adding a new tool takes an afternoon — fork the benchmark PRs, trigger the tool, run the pipeline. +**Tools evaluated**: Augment, Baz, Claude Code, CodeAnt, CodeRabbit, Cubic, Cursor Bugbot, Devin, Gemini, GitHub Copilot, GitLab Duo, Graphite, Greptile, KG, Kodus, Macroscope, Qodo, Sourcery, and more. Running a tool that isn't on the leaderboard takes an afternoon — fork the benchmark PRs, trigger the tool, run the pipeline. Publishing it alongside the others additionally requires meeting the [inclusion criteria](#inclusion-criteria). > **Known limitation**: Static datasets risk training data leakage — tools may have seen these PRs during training. That's why we also run the online benchmark. @@ -163,15 +163,29 @@ uv run python main.py analyze --all uv run python main.py dashboard ``` -## Adding a new tool to the offline benchmark +## Adding a new tool + +Running a tool that isn't on the leaderboard is open to anyone: 1. Fork the 50 benchmark PRs into a GitHub org where your tool is installed 2. Let the tool review each PR 3. Add the tool name to the download config and run the pipeline -4. Results appear alongside existing tools in the dashboard +4. Compare the results against the existing tools in the dashboard See [`offline/README.md`](offline/README.md) for detailed instructions. +### Inclusion criteria + +Publishing a tool on the leaderboard alongside the others has two further requirements. + +**Public usage.** Roughly at least 600–1,000 reviewed public PRs, across a spread of orgs, repos, and authors. The offline benchmark is a fixed set of 50 PRs, so on its own it can't tell us whether a score reflects how a tool behaves in practice. We validate it against the online benchmark, which measures how developers respond to a tool's reviews in the wild, and that cross-check needs enough public review activity to be meaningful. Private installs aren't visible to us and can't be counted, so this is a measurement constraint rather than a judgement about a tool's overall adoption. + +**Attributable reviews.** We need to be able to tell from the GitHub API that a review came from the tool rather than from a person — a bot account, a dedicated machine account, or a consistent marker in the comment body all work. Without one of those we can't separate a tool's findings from a human reviewer's comments. + +For any tool we publish we also run the pipeline ourselves rather than take submitted results, so every number on the leaderboard is produced the same way. + +If a tool doesn't meet these yet, the harness is public and you're welcome to run it and publish your own results. + ## Contributing We welcome contributions — new tools, better golden comments, improved judge prompts, additional datasets. Open an issue or PR. diff --git a/offline/README.md b/offline/README.md index 78b2815..bf0efd3 100644 --- a/offline/README.md +++ b/offline/README.md @@ -26,7 +26,7 @@ Open replication of the code review benchmark used by companies like [Augment](h | [Qodo](https://www.qodo.ai/) | AI code review | | [Sourcery](https://sourcery.ai/) | AI code review | -Adding a new tool requires forking the benchmark PRs and collecting the tool's reviews — see Steps 0 and 1 below. +Running a tool that isn't listed here requires forking the benchmark PRs and collecting the tool's reviews — see Steps 0 and 1 below. Publishing a tool on the leaderboard alongside the others additionally requires meeting the [inclusion criteria](../README.md#inclusion-criteria). ## Methodology