Skip to content

feat(supabase): add findings_* tables for paper-figure summaries (formula-code/fc-eval#19) - #25

Merged
atharvas merged 1 commit into
mainfrom
feat/issue-19-findings-tables
May 11, 2026
Merged

feat(supabase): add findings_* tables for paper-figure summaries (formula-code/fc-eval#19)#25
atharvas merged 1 commit into
mainfrom
feat/issue-19-findings-tables

Conversation

@atharvas

Copy link
Copy Markdown
Member

Summary

Adds the database half of formula-code/fc-eval#19: eight public-read tables backing the seven paper-figure summaries the website (formula-code/formula-code.github.io) currently ships as static JSON under `src/data/findings/f{1..7}_*.json`.

Once this lands and the exporter populates rows, the website's `findings.js` shim can swap from static JSON to live PostgREST fetches at `https://api.formulacode.org/rest/v1/findings_*\` — no API gateway changes required (same anon-key surface as `repositories` / `pull_requests` / `harbor_runs`).

Tables added

Table Paper artifact
`findings_global_leaderboard` Table 1
`findings_stratified_advantage` Figure 3
`findings_tag_advantage` Table 2
`findings_repo_quintiles` Table 3
`findings_cost_pareto` Figure 4 / Table 10
`findings_workload_tradeoff` Figure 5
`findings_temporal_generalization` Table 4 / Figure 1
`findings_metadata` sidecar (`_source`, `_generated_at`, `axis_metadata`, `row_count`)

Schema choices

  • Column names match the website scaffolds verbatim — `rp_rank` (not `rp_rank_adv`), `advantage_norm` (not `normalized_advantage`), `speedup_geomean` (not `mean_speedup`), `worst_workload_speedup`, `cost_usd_per_task` + `advantage_weighted`. This means the website's `findings.js` can swap fetch sources with no field renaming.
  • Per-task rows use `(owner, repo, issue_number)` — aligns with `pull_requests` PK and `harbor_runs` columns. `findings_workload_tradeoff` carries an FK to `pull_requests` so the website can join back for PR metadata in one PostgREST call.
  • RLS + anon grants follow the existing pattern — enable RLS, `public_read` SELECT policy for `anon`, then explicit `GRANT SELECT` (since `00015_revoke_anon_select.sql` revoked the schema-wide default).
  • Migration sequencing — branches off `main` (currently at 00015). #23 lands 00016/00017/00019/00020; this PR's 00021 sits cleanly on top after that merges.

Test plan

Local apply + RLS sanity:

```bash
make supabase-up
docker exec supabase_db_datasmith_new psql -U postgres -d postgres \
-f /dev/stdin < supabase/migrations/00021_findings_tables.sql

RLS gating is in place:

docker exec supabase_db_datasmith_new psql -U postgres -d postgres -c \
"SELECT tablename, policyname FROM pg_policies WHERE tablename LIKE 'findings_%';"

→ 8 rows, all 'public_read'

Anon SELECT returns 200 OK (empty before exporter runs):

curl -s -H "apikey: $SUPABASE_ANON_KEY" \
http://127.0.0.1:54321/rest/v1/findings_global_leaderboard?select=*
```

Schema verification against canonical contract: each table's columns are a 1:1 match for the website scaffolds at `formula-code/formula-code.github.io:src/data/findings/f{1..7}_*.json` (verified during plan phase).

Follow-up

  • Exporter PR — `analysis/export_website_findings.py` exists locally in fc-eval and populates these tables via `--upsert-supabase`, but the `analysis/` tree there is currently uncommitted; it'll ship in a separate PR once the dependency tree (`nb_utils.py` / `task.py` / `ranking.py` / configs) is consolidated.
  • Prod deploy — once this PR lands, the migration needs to be applied to the prod Supabase instance, then the exporter run with `--upsert-supabase` to populate initial rows.
  • Refresh cadence — manual for v1; a scheduled refresh (GitHub Action / cron) is a follow-up.

🤖 Generated with Claude Code

…mula-code/fc-eval#19)

Adds eight public-read tables backing the seven paper-figure summaries on
api.formulacode.org/rest/v1/findings_*:

  findings_global_leaderboard       Table 1
  findings_stratified_advantage     Figure 3
  findings_tag_advantage            Table 2
  findings_repo_quintiles           Table 3
  findings_cost_pareto              Figure 4 / Table 10
  findings_workload_tradeoff        Figure 5
  findings_temporal_generalization  Table 4 / Figure 1
  findings_metadata                 sidecar: _source / _generated_at /
                                    axis_metadata / row_count per finding

Row column names align with the canonical scaffolds at
formula-code/formula-code.github.io:src/data/findings/f{1..7}_*.json
(rp_rank, advantage_norm, speedup_geomean, worst_workload_speedup,
cost_usd_per_task + advantage_weighted). Per-task rows in
findings_workload_tradeoff use (owner, repo, issue_number) to match
pull_requests / harbor_runs.

RLS + anon grants mirror 00012_public_read_rls.sql + 00015_revoke_anon_select.sql:
enable RLS, public_read SELECT policy for anon, then explicit GRANT SELECT
(since 00015 revoked the schema-wide anon default).

Rows are refreshed by analysis/export_website_findings.py in fc-eval
(separate; not yet upstreamed). Pending follow-up after merge:
  1. supabase migration up on prod
  2. uv run analysis/export_website_findings.py --upsert-supabase

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@atharvas
atharvas merged commit fa2aaba into main May 11, 2026
3 checks passed
@atharvas
atharvas deleted the feat/issue-19-findings-tables branch May 11, 2026 22:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant