You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
| F7 | Temporal generalization | Table 4 (Temporal analysis) | ⚠️ | Schema now matches paper Table 4's 6-bin layout (`6+ mo before` … `6+ mo after`) and Qwen 3 Coder is excluded — 3 models × 6 bins. **11 of 18 cells populated** vs paper's 18/18; the 7 nulls reflect the upstream task set being sparser per bin than the paper's run. Will fill in as more tasks ingest. |
19
+
| F7 | Temporal generalization | Table 4 (Temporal analysis) | ⚠️ | Schema now matches paper Table 4's 6-bin layout (`6+ mo before` … `6+ mo after`) and Qwen 3 Coder is excluded — 3 models x 6 bins. **11 of 18 cells populated** vs paper's 18/18; the 7 nulls reflect the upstream task set being sparser per bin than the paper's run. Will fill in as more tasks ingest. |
20
20
21
21
Open upstream issue tracking the divergences: [`formula-code/fc-eval#19`](https://github.com/formula-code/fc-eval/issues/19).
22
22
@@ -32,9 +32,9 @@ Open upstream issue tracking the divergences: [`formula-code/fc-eval#19`](https:
**Parity check:** Baseline row's `speedup_geomean` now shows **1.1193×** (was previously `null`), so the diverging color scale on the Speedup column has a center to read against. The RP ordering is what `analysis.task.compute_leaderboard` returns — that's the paper's methodology (Ranked Pairs voting), not advantage-sorted. The earlier-flagged "Claude at rp_rank=1" expectation came from a stale scaffold that had been sorted on advantage rather than computed via RP.
37
+
**Parity check:** Baseline row's `speedup_geomean` now shows **1.1193x** (was previously `null`), so the diverging color scale on the Speedup column has a center to read against. The RP ordering is what `analysis.task.compute_leaderboard` returns — that's the paper's methodology (Ranked Pairs voting), not advantage-sorted. The earlier-flagged "Claude at rp_rank=1" expectation came from a stale scaffold that had been sorted on advantage rather than computed via RP.
38
38
39
39
### F2 — Figure 3 (Stratified advantage)
40
40
@@ -62,7 +62,7 @@ Open upstream issue tracking the divergences: [`formula-code/fc-eval#19`](https:
| Render style | Heatmap table — agent x tag matrix, diverging RdBu @ 0 |
66
66
67
67
**Parity check:** The exporter now reads from `task.metadata["classification"]` (the same per-PR enum `analysis/table_8_tags.ipynb` uses), so the website table carries 9 tag keys — `parallelization, batching, caching, algorithmic, data_structure, reduce_work, higher_level, micro, lower_level`. The five paper tags missing from the API (`approximation, scale, db, io, uncategorized`) have zero tasks classified into them in `filtered_formulacode-verified.parquet`; that matches Table 2's actual sparsity. The website renders those columns as `—` and otherwise mirrors the paper.
68
68
@@ -76,7 +76,7 @@ Open upstream issue tracking the divergences: [`formula-code/fc-eval#19`](https:
| Render style | Heatmap table — model × 6 paper-Table-4 bins (`6+ mo before` / `3–6 mo before` / `0–3 mo before` / `0–3 mo after` / `3–6 mo after` / `6+ mo after`), sequential blues on speedup |
121
+
| Render style | Heatmap table — model x 6 paper-Table-4 bins (`6+ mo before` / `3–6 mo before` / `0–3 mo before` / `0–3 mo after` / `3–6 mo after` / `6+ mo after`), sequential blues on speedup |
122
122
123
123
**Parity check:** Upstream re-shipped `findings_temporal_generalization` with the paper Table 4 binning — six 3-month-wide windows (`pre6plus`, `pre3to6`, `pre0to3`, `post0to3`, `post3to6`, `post6plus`), three models (Claude / GPT-5 / Gemini, Qwen excluded per `figure_1_temporal.ipynb` cell 5 popping its cutoff). Cell values are mean `agent/nop` within each bin.
averaged across groups:</p> <div class="formula svelte-s2t5c8">Adv<sup>(ℓ)</sup> = (1/|G<sup>ℓ</sup>|) Σ Adv<sub>g</sub></div> <p class="svelte-s2t5c8">This captures whether improvements are spread evenly or concentrated
123
123
in specific subsystems.</p></div> <div class="metric-def svelte-s2t5c8"><h2 class="svelte-s2t5c8">Normalized Advantage</h2> <p class="svelte-s2t5c8">Signal-to-noise ratio of the advantage, penalizing erratic performance
124
124
across workloads:</p> <div class="formula svelte-s2t5c8">Ã = Adv / √(σ²<sub>agent</sub> + σ²<sub>expert</sub>)</div> <p class="svelte-s2t5c8">Rewards agents that deliver steady improvements across many workloads
125
-
rather than volatile, concentrated gains.</p></div> <div class="metric-def svelte-s2t5c8"><h2 class="svelte-s2t5c8">Cost-Weighted Advantage</h2> <p class="svelte-s2t5c8">Advantage divided by total inference cost (input + output tokens ×
125
+
rather than volatile, concentrated gains.</p></div> <div class="metric-def svelte-s2t5c8"><h2 class="svelte-s2t5c8">Cost-Weighted Advantage</h2> <p class="svelte-s2t5c8">Advantage divided by total inference cost (input + output tokens x
126
126
per-token price):</p> <div class="formula svelte-s2t5c8">Cost(Adv) = Adv / C<sub>agent</sub></div> <p class="svelte-s2t5c8">Captures the human-relative improvement obtained per unit of inference
127
127
budget.</p></div></article> <article id="submitting" class="doc-section svelte-s2t5c8"><h1 class="svelte-s2t5c8">Submitting Results</h1> <p class="svelte-s2t5c8">After running your evaluation, you can submit results to be included on
128
128
the <a href="/leaderboard/" class="svelte-s2t5c8">public leaderboard</a>:</p> <ol class="steps-list svelte-s2t5c8"><li class="svelte-s2t5c8"><strong class="svelte-s2t5c8">Run the full evaluation</strong> on the FormulaCode dataset with
Copy file name to clipboardExpand all lines: docs/docs/index.html
+9-9Lines changed: 9 additions & 9 deletions
Original file line number
Diff line number
Diff line change
@@ -64,7 +64,7 @@
64
64
averaged across groups:</p><divclass="formula svelte-s2t5c8">Adv<sup>(ℓ)</sup> = (1/|G<sup>ℓ</sup>|) Σ Adv<sub>g</sub></div><pclass="svelte-s2t5c8">This captures whether improvements are spread evenly or concentrated
65
65
in specific subsystems.</p></div><divclass="metric-def svelte-s2t5c8"><h2class="svelte-s2t5c8">Normalized Advantage</h2><pclass="svelte-s2t5c8">Signal-to-noise ratio of the advantage, penalizing erratic performance
66
66
across workloads:</p><divclass="formula svelte-s2t5c8">Ã = Adv / √(σ²<sub>agent</sub> + σ²<sub>expert</sub>)</div><pclass="svelte-s2t5c8">Rewards agents that deliver steady improvements across many workloads
67
-
rather than volatile, concentrated gains.</p></div><divclass="metric-def svelte-s2t5c8"><h2class="svelte-s2t5c8">Cost-Weighted Advantage</h2><pclass="svelte-s2t5c8">Advantage divided by total inference cost (input + output tokens ×
67
+
rather than volatile, concentrated gains.</p></div><divclass="metric-def svelte-s2t5c8"><h2class="svelte-s2t5c8">Cost-Weighted Advantage</h2><pclass="svelte-s2t5c8">Advantage divided by total inference cost (input + output tokens x
68
68
per-token price):</p><divclass="formula svelte-s2t5c8">Cost(Adv) = Adv / C<sub>agent</sub></div><pclass="svelte-s2t5c8">Captures the human-relative improvement obtained per unit of inference
69
69
budget.</p></div></article><articleid="submitting" class="doc-section svelte-s2t5c8"><h1class="svelte-s2t5c8">Submitting Results</h1><pclass="svelte-s2t5c8">After running your evaluation, you can submit results to be included on
70
70
the <ahref="/leaderboard/" class="svelte-s2t5c8">public leaderboard</a>:</p><olclass="steps-list svelte-s2t5c8"><liclass="svelte-s2t5c8"><strongclass="svelte-s2t5c8">Run the full evaluation</strong> on the FormulaCode dataset with
@@ -74,14 +74,14 @@
74
74
configuration details.</li><liclass="svelte-s2t5c8"><strongclass="svelte-s2t5c8">We'll review and publish</strong> your results on the leaderboard
75
75
within one update cycle (monthly).</li></ol><divclass="info-box svelte-s2t5c8"><strongclass="svelte-s2t5c8">Questions?</strong> Open an issue on
or reach out to the authors listed on the <ahref="/" class="svelte-s2t5c8">overview page</a>.</div></article></main></section></main><footerclass="svelte-126cfes"><divclass="c svelte-126cfes"><divclass="paper-footer-sections svelte-126cfes"><sectionclass="paper-section citation-section svelte-126cfes"><h3class="paper-section-title svelte-126cfes">Citation</h3><divclass="bibtex-wrapper svelte-126cfes"><preclass="bibtex-code svelte-126cfes"><codeclass="svelte-126cfes">@misc{sehgal2025formulacode,
78
-
title={FormulaCode: Evaluating Agentic Optimization on Large Codebases},
79
-
author={Atharva Sehgal and James Hou and Akanksha Sarkar and Ishaan Mantripragada and Swarat Chaudhuri and Jennifer J. Sun and Yisong Yue},
80
-
year={2025},
81
-
eprint={2409.XXXXX},
82
-
archivePrefix={arXiv},
83
-
primaryClass={cs.LG},
84
-
url={https://arxiv.org/abs/2409.XXXXX},
77
+
or reach out to the authors listed on the <ahref="/" class="svelte-s2t5c8">overview page</a>.</div></article></main></section></main><footerclass="svelte-126cfes"><divclass="c svelte-126cfes"><divclass="paper-footer-sections svelte-126cfes"><sectionclass="paper-section citation-section svelte-126cfes"><h3class="paper-section-title svelte-126cfes">Citation</h3><divclass="bibtex-wrapper svelte-126cfes"><preclass="bibtex-code svelte-126cfes"><codeclass="svelte-126cfes">@misc{sehgal2025formulacode,
78
+
title={FormulaCode: Evaluating Agentic Optimization on Large Codebases},
79
+
author={Atharva Sehgal and James Hou and Akanksha Sarkar and Ishaan Mantripragada and Swarat Chaudhuri and Jennifer J. Sun and Yisong Yue},
80
+
year={2025},
81
+
eprint={2409.XXXXX},
82
+
archivePrefix={arXiv},
83
+
primaryClass={cs.LG},
84
+
url={https://arxiv.org/abs/2409.XXXXX},
85
85
}</code></pre></div></section><sectionclass="paper-section funding-section svelte-126cfes"><h3class="paper-section-title svelte-126cfes">Acknowledgements</h3><pclass="paper-section-text svelte-126cfes"><!-- HTML_TAG_START -->This project was made possible by a Slingshot Award from the <ahref=https://www.laude.org/>Laude Institute</a> and an <ahref=https://www.neurosymbolic.org/>NSF Expeditions</a> grant.<!-- HTML_TAG_END --></p></section></div><divclass="footer-acknowledgements svelte-126cfes"><pclass="svelte-126cfes"><!-- HTML_TAG_START -->The website design heavily uses the template developed by <ahref=https://pudding.cool/author/fox-meyer/>Fox Meyer</a> and <ahref=https://pudding.cool/author/jan-diehm/>Jan Diehm</a> for their interactive article in <ahref='https://pudding.cool/'>Pudding.cool</a> on <ahref='https://github.com/the-pudding/wine-animals'>The Pour-ing of species</a> that is distributed under an MIT license. The code itself is based on <ahref='https://github.com/the-pudding/svelte-starter' target='_blank'>The Pudding's SvelteKit starter template</a>. The Visualizations use <ahref='https://layercake.graphics/' target='_blank'>LayerCake</a> and <ahref='https://d3js.org/' target='_blank'>D3.js</a>. The source code for this website is available <ahref='https://github.com/formula-code/formula-code.github.io/'>here</a>, also under an MIT license.<!-- HTML_TAG_END --></p></div></div></footer>
0 commit comments