|
| 1 | +# Data Wishlist |
| 2 | + |
| 3 | +This document tracks data we'd like access to so the website can ship the |
| 4 | +features users (and reviewers) actually want. Each item lists what's needed, |
| 5 | +which page benefits, and what we ship without it. |
| 6 | + |
| 7 | +--- |
| 8 | + |
| 9 | +## 1. The full benchmark dataset (957 tasks, not 3) |
| 10 | + |
| 11 | +`src/data/website_data_lite.csv` currently contains workloads from **3 |
| 12 | +task_ids** across 3 repositories (astropy, pandas, scikit-learn). The paper |
| 13 | +claims 957 performance bottlenecks across 70+ repos. This is the single |
| 14 | +biggest wishlist item. |
| 15 | + |
| 16 | +- **Used by:** `/explorer/`, scrollytelling chart, leaderboard recompute. |
| 17 | +- **Ideal shape:** the existing CSV schema (`id, level, agent_id, agent/nop, |
| 18 | + oracle/nop, task_id, agent_recording, repo_name, benchmark_name`) with all |
| 19 | + ~1.4M workloads spanning all 957 tasks. |
| 20 | +- **Acceptable interim:** a representative slice of one task per repo (≈70 |
| 21 | + rows of `(repo, task_id)` pairs), enough that filter chips on the explorer |
| 22 | + feel populated. |
| 23 | +- **Without it:** the explorer reads as a "tiny preview slice." The `Browse |
| 24 | + N performance workloads` headline is honest but underwhelming. |
| 25 | + |
| 26 | +## 2. `merged_at` dates per task |
| 27 | + |
| 28 | +Already noted in `CLAUDE.md` — the time-travel slider on `/leaderboard/` |
| 29 | +stays dormant until each row carries a `merged_at` ISO 8601 string. The |
| 30 | +ranking engine (`src/utils/rankingEngine.js`) auto-activates once present. |
| 31 | + |
| 32 | +- **Used by:** `/leaderboard/` time-travel slider. |
| 33 | +- **Ideal shape:** add a `merged_at` column to `website_data.csv` and |
| 34 | + propagate via `process_remote_data.py` → `LITE_COLUMNS`. |
| 35 | +- **Without it:** slider hidden, leaderboard is static. |
| 36 | + |
| 37 | +## 3. Per-task gold patch and best-agent patch (diffs) |
| 38 | + |
| 39 | +MathNet's explorer shows the problem statement. FormulaCode's explorer |
| 40 | +currently shows the *benchmark harness* (the `time_*` function being |
| 41 | +measured). What the audience actually wants to see is the **optimization |
| 42 | +patch** — what the human expert did, and what the agent did, side by side. |
| 43 | + |
| 44 | +- **Used by:** `/explorer/` workload drawer (new "Patch" section), eventually |
| 45 | + a per-task page. |
| 46 | +- **Ideal shape:** for each `task_id`, two unified diffs: |
| 47 | + - `oracle_patch` — the merged human PR's patch |
| 48 | + - `agent_patches[<agent_id>]` — the patch the agent produced (per agent we |
| 49 | + ran) |
| 50 | + Plus optionally the full pre/post file contents for syntax-highlighted |
| 51 | + side-by-side rendering. |
| 52 | +- **Acceptable interim:** even just `oracle_patch` per task gets us 80% of |
| 53 | + the way there. |
| 54 | +- **Without it:** the drawer's value is capped at "here's the harness and |
| 55 | + here are the speedups." No insight into *why* the agent won/lost. |
| 56 | + |
| 57 | +## 4. Task descriptions and PR URLs |
| 58 | + |
| 59 | +Each `task_id` should link back to its origin so users can verify and dig |
| 60 | +deeper. |
| 61 | + |
| 62 | +- **Used by:** `/explorer/` workload drawer, future per-task page. |
| 63 | +- **Ideal shape:** a `tasks.json` keyed by `task_id`: |
| 64 | + ```json |
| 65 | + { |
| 66 | + "astropy_astropy_21": { |
| 67 | + "pr_url": "https://github.com/astropy/astropy/pull/12345", |
| 68 | + "pr_title": "Speed up Time initialization with units", |
| 69 | + "merged_at": "2024-03-15T12:34:56Z", |
| 70 | + "summary": "Caches unit conversion in Time constructor to avoid …", |
| 71 | + "optimization_strategy": "caching", |
| 72 | + "files_changed": 3, |
| 73 | + "lines_added": 42, |
| 74 | + "lines_removed": 18 |
| 75 | + } |
| 76 | + } |
| 77 | + ``` |
| 78 | +- **Without it:** the explorer can identify a task by ID but can't say what |
| 79 | + the task *is* in plain English. |
| 80 | + |
| 81 | +## 5. Repository metadata |
| 82 | + |
| 83 | +To support the "browse by repo" and any "performance by repo popularity" |
| 84 | +analysis (the paper mentions a 4th-quintile finding), each repo needs basic |
| 85 | +metadata. |
| 86 | + |
| 87 | +- **Used by:** `/explorer/` repo chips, future repo-detail page. |
| 88 | +- **Ideal shape:** `repos.json`: |
| 89 | + ```json |
| 90 | + { |
| 91 | + "astropy_astropy": { |
| 92 | + "owner": "astropy", |
| 93 | + "repo": "astropy", |
| 94 | + "stars": 4400, |
| 95 | + "language": "Python", |
| 96 | + "description": "Astronomy and astrophysics core library", |
| 97 | + "n_tasks": 21, |
| 98 | + "topic": "scientific-computing" |
| 99 | + } |
| 100 | + } |
| 101 | + ``` |
| 102 | +- **Without it:** repos render as `astropy/astropy` slug-only — fine, but the |
| 103 | + page is less informative than MathNet's "Browse by competition" panel. |
| 104 | + |
| 105 | +## 6. Per-agent cost & token usage per task |
| 106 | + |
| 107 | +The paper's cost-efficiency finding ("frontier LLMs are overall more cost |
| 108 | +effective than open weights models") is a key result that has no home on the |
| 109 | +site yet. |
| 110 | + |
| 111 | +- **Used by:** new "Cost vs. advantage" chart on `/leaderboard/`, agent |
| 112 | + cards. |
| 113 | +- **Ideal shape:** add to the per-row CSV: `agent_cost_usd`, |
| 114 | + `agent_input_tokens`, `agent_output_tokens`, `agent_wallclock_seconds`. |
| 115 | +- **Without it:** we can't surface the cost story visually. |
| 116 | + |
| 117 | +## 7. Optimization strategy labels |
| 118 | + |
| 119 | +The paper finds agents excel at parallelization/batching and struggle with |
| 120 | +vectorization. Surfacing these tags would let users filter the explorer by |
| 121 | +strategy. |
| 122 | + |
| 123 | +- **Used by:** `/explorer/` optional "Strategy" filter chip. |
| 124 | +- **Ideal shape:** per-task labels (`["caching", "vectorization", "io"]`) on |
| 125 | + the human PR. |
| 126 | +- **Without it:** strategy taxonomy lives only in the paper, not the site. |
| 127 | + |
| 128 | +--- |
| 129 | + |
| 130 | +## Out of scope (intentionally) |
| 131 | + |
| 132 | +- **Submission flow / new agent uploads.** Following MathNet's lead — we |
| 133 | + don't need a backend for this. Users submit via the documented `fceval` |
| 134 | + CLI; we ingest results into the static dataset. |
| 135 | +- **User accounts, comments, voting.** Not relevant. |
| 136 | +- **Real-time leaderboard.** Cadence is paper/release, not minutes. |
0 commit comments