Resolve #23: expose benchmark_codes and standardize task identity on (owner, repo, issue_number)

CLAUDE.md

@@ -62,6 +62,7 @@ fc-data --stage 7 --harbor-limit 10                                    # Smoke t
 6. **synthesize_images** — Agent-based Docker build context synthesis (uses env_payload/python_version from stage 4)
 7. **harbor_healthcheck** — Run every synthesized container through Harbor's oracle agent, record per-benchmark speedups to `harbor_runs`. Supports local Docker and Daytona via `--harbor-environment`; the row records which one in `harbor_runs.environment`. Local runs are useful for iteration; only Daytona runs gate stage 8.
 8. **publish** — Build, verify, and publish Docker images to DockerHub. Only publishes PRs with at least one successful **Daytona** `harbor_runs` row whose `max_speedup >= 1.05`.
+9. **scrape_benchmark_source** — For each `(owner, repo)` in `candidate_containers`, check out the repo at its container SHA, AST-parse every ASV-style benchmark function under the repo's `benchmark_dir`, and upsert one row per `(owner, repo, benchmark_without_params)` into `benchmark_codes` for the FormulaCode website's data sync.
 
 ### Dataset verification (`dataset/`)
 
@@ -112,7 +113,8 @@ import time (`src/datasmith/__init__.py` → `dotenv.load_dotenv`), so reading
 - **Existing uses** (non-exhaustive, grep `DATASMITH_` for the full list):
   `DATASMITH_RL_DEFAULT_PAUSE_S`, `DATASMITH_RL_PAUSE_JITTER_S`,
   `DATASMITH_RL_MAX_RETRIES`, `DATASMITH_NEIGHBOR_WINDOW_DAYS`,
-  `DATASMITH_NEIGHBOR_CAP`.
+  `DATASMITH_NEIGHBOR_CAP`, `DATASMITH_BENCH_SCRAPE_MAX_FILE_BYTES`,
+  `DATASMITH_BENCH_SCRAPE_DIRS`.
 
 ## Supabase
 
@@ -138,7 +140,7 @@ Local vLLM servers (8123, 8124) are exposed via a LiteLLM proxy on `https://mode
 
 ### Public read-only access (RLS)
 
-Four tables are readable by the `anon` role: `repositories`, `pull_requests`, `candidate_containers`, `harbor_runs`. Migration `00012_public_read_rls.sql` enables RLS with a `public_read` SELECT policy on those tables; migration `00015_revoke_anon_select.sql` revokes Supabase's default broad `anon` `SELECT` grant and re-grants it only on the four, so every other table returns `permission denied`. The service-role key bypasses both layers, so pipeline processes are unaffected. Public anon access is served on `https://api.formulacode.org` (no Cloudflare Access gate); pipeline operators continue to use `https://db.formulacode.org` with CF Access + service-role key.
+Six tables are readable by the `anon` role: `repositories`, `pull_requests`, `candidate_containers`, `harbor_runs`, `benchmark_information`, `benchmark_codes`. Migration `00012_public_read_rls.sql` enables RLS with a `public_read` SELECT policy on the original four; migration `00016_benchmark_information.sql` adds the same policy to `benchmark_information`; migration `00017_benchmark_codes.sql` adds it to `benchmark_codes`. Migration `00015_revoke_anon_select.sql` revokes Supabase's default broad `anon` `SELECT` grant and re-grants it only on the public set, so every other table returns `permission denied`. The service-role key bypasses both layers, so pipeline processes are unaffected. Public anon access is served on `https://api.formulacode.org` (no Cloudflare Access gate); pipeline operators continue to use `https://db.formulacode.org` with CF Access + service-role key.
 
 ### Key tables
 
@@ -148,6 +150,8 @@ Four tables are readable by the `anon` role: `repositories`, `pull_requests`, `c
 | `packages` | Resolved `env_payload` (pinned deps) and `python_version` per commit | Stage 4 |
 | `candidate_containers` | Successful agent-generated `build_pkg_sh` / `build_run_sh` per SHA | Stage 6 (on success) |
 | `harbor_runs` | One row per Harbor oracle trial for a synthesized container: `max_speedup`, `geomean_speedup`, `n_benchmarks`, `wallclock_sec`, `reward_payload`, `status`. One-to-many FK on `candidate_containers(owner, repo, sha)`. | Stage 7 |
+| `benchmark_information` | Per-benchmark speedup measurements from terminal-bench eval runs: one row per (run, owner/repo/issue, benchmark, agent, model). `speedup` is `(agent/nop)/(oracle/nop)` so 1.0 = parity with the human expert. `benchmark_type` (`time`/`mem`/`peakmem`/`track`) is a generated column derived from the ASV naming convention. Loaded out-of-band via `scripts/load_benchmark_information.py`. | (manual) |
+| `benchmark_codes` | One row per `(owner, repo, benchmark_without_params)` carrying the Python source of each ASV benchmark function plus its setup. Joined to `benchmark_information` on `(owner, repo, benchmark_name)` by the FormulaCode website. | Stage 9 |
 | `error_logs` | Per-attempt synthesis results: agent output, failure stage/return code, error messages | Stage 6 (`Synthesizer._log_attempt`) |
 | `runner_progress` | Live progress counters (total/completed/failed) per pipeline run | `BaseRunner` (all stages) |
 | `runner_failures` | One row per item failure with error message + traceback | `BaseRunner._log_failure` |
@@ -170,6 +174,21 @@ conn.cursor().execute(open('supabase/migrations/00007_error_logs.sql').read())
 
 Use `datasmith.utils.db.fetch_all(table, select=..., filters=..., gte_filters=..., ...)` for paginated reads, or `get_client()` for direct Supabase client access.
 
+### Level aggregation
+
+Per-benchmark speedups are rolled up into four levels via **geometric mean** in `src/datasmith/harbor_adapter/template/parser.py` (`geometric_mean()` + `aggregate_by_hierarchy()`):
+
+- **level1** — identity, one entry per benchmark (`module.Class.method`)
+- **level2** — grouped by `module.Class` (drop the last dotted segment), geomean within each group
+- **level3** — grouped by `module` (top dotted segment), geomean within each group
+- **level4** — a single overall geomean across every benchmark
+
+`benchmark_information.benchmark_name` is already param-stripped (e.g. `benchmarks.ConstructorsSuite.time_point`), so consumers can replicate the rollup locally without re-parsing. The FormulaCode website CSV's `1-Params` row is equivalent to datasmith's `level1`; the four upper website rows map onto `level2`/`level3`/`level4` modulo column naming.
+
+### Task identity
+
+The canonical identifier for one PR / one task is the tuple `(owner, repo, issue_number)`. Tables that need a single-column join key expose a `task_id` field whose value equals `issue_number`; never construct it as a derived string. `candidate_containers.task_id` is a STORED generated column (see migration `00019_candidate_containers_task_id.sql`).
+
 ## Environment setup
 
 Requires a `tokens.env` file in the repo root with `GH_TOKEN`, `CACHE_LOCATION`, `SUPABASE_URL`, `SUPABASE_KEY`, and optionally `DSPY_*` vars for LLM backends and `DOCKERHUB_*` vars for publishing. See README.md for the full template.

scripts/harbor_tasks_migration.sql

@@ -0,0 +1,110 @@
+-- ============================================================================
+-- Harbor `tasks` table migration: composite (owner, repo, issue_number) PK
+-- ============================================================================
+--
+-- Target: Harbor's Supabase project (see HARBOR_SUPABASE_URL in tokens.env),
+-- NOT datasmith's local Supabase. Apply via:
+--
+--   psql "$HARBOR_DATABASE_URL" -f scripts/harbor_tasks_migration.sql
+--
+-- This script realigns Harbor's tasks table with the canonical FormulaCode
+-- identity tuple `(owner, repo, issue_number)`. The legacy `task_id` column
+-- was constructed inconsistently across the codebase (`owner__repo-N` in
+-- the publish path, `owner_repo_N` in harbor_adapter), so we extract its
+-- components and key the table on the tuple instead. The `task_id` column
+-- is retained for one release as an integer mirror of `issue_number`.
+--
+-- Companion change: scripts/migrate_snapshot_keys.py walks Supabase Storage
+-- and renames `snapshots/{old_task_id}/oracle.tar.gz` →
+-- `snapshots/{owner}/{repo}/{issue_number}/oracle.tar.gz` once this script
+-- has run successfully (so the new tasks rows already carry the triple).
+--
+-- Re-runnable: every statement is idempotent.
+
+BEGIN;
+
+-- 1. Add the three target columns (nullable initially so the backfill can
+--    run before we add NOT NULL constraints).
+ALTER TABLE tasks ADD COLUMN IF NOT EXISTS owner        TEXT;
+ALTER TABLE tasks ADD COLUMN IF NOT EXISTS repo         TEXT;
+ALTER TABLE tasks ADD COLUMN IF NOT EXISTS issue_number INT;
+
+-- 2. Backfill from the legacy task_id string.
+--    Two patterns observed in the wild:
+--      Format A:  owner__repo-N    (publish/HuggingFace path)
+--      Format B:  owner_repo_N     (harbor_adapter path)
+--    Format A uses a double-underscore between owner and repo and a dash
+--    before the PR number; Format B uses single underscores throughout.
+--
+--    We try Format A first (more specific), then fall back to Format B.
+--    Rows that match neither stay NULL and will be flagged below.
+UPDATE tasks
+   SET owner        = split_part(task_id, '__', 1),
+       repo         = split_part(split_part(task_id, '__', 2), '-', 1),
+       issue_number = NULLIF(
+                        regexp_replace(
+                          split_part(task_id, '__', 2),
+                          '^[^-]*-(\d+)$',
+                          '\1'
+                        ),
+                        split_part(task_id, '__', 2)
+                      )::INT
+ WHERE owner IS NULL
+   AND task_id ~ '^[^_]+__[^_-]+-\d+$';
+
+-- Fallback: Format B (owner_repo_N) — but only for rows still unbackfilled.
+-- This is brittle because owner and repo can both contain underscores, so
+-- we assume the last underscore-separated segment is the integer issue.
+UPDATE tasks
+   SET owner        = regexp_replace(task_id, '^(.+)_([^_]+)_(\d+)$', '\1'),
+       repo         = regexp_replace(task_id, '^(.+)_([^_]+)_(\d+)$', '\2'),
+       issue_number = regexp_replace(task_id, '^(.+)_([^_]+)_(\d+)$', '\3')::INT
+ WHERE owner IS NULL
+   AND task_id ~ '^.+_[^_]+_\d+$';
+
+-- Surface anything that didn't match either pattern.
+DO $$
+DECLARE n_orphaned INT;
+BEGIN
+    SELECT count(*) INTO n_orphaned FROM tasks WHERE owner IS NULL;
+    IF n_orphaned > 0 THEN
+        RAISE WARNING 'tasks: % rows have task_id values that do not parse — leaving them with NULL owner/repo/issue_number; review manually before enforcing NOT NULL', n_orphaned;
+    END IF;
+END $$;
+
+-- 3. Once the backfill is verified clean, enforce NOT NULL + swap the PK.
+--    Skip this block if any orphans remain — operator handles them first.
+DO $$
+DECLARE n_orphaned INT;
+BEGIN
+    SELECT count(*) INTO n_orphaned FROM tasks WHERE owner IS NULL OR repo IS NULL OR issue_number IS NULL;
+    IF n_orphaned = 0 THEN
+        ALTER TABLE tasks ALTER COLUMN owner        SET NOT NULL;
+        ALTER TABLE tasks ALTER COLUMN repo         SET NOT NULL;
+        ALTER TABLE tasks ALTER COLUMN issue_number SET NOT NULL;
+
+        -- Replace the PK only if it isn't already on the triple.
+        IF NOT EXISTS (
+            SELECT 1
+              FROM pg_constraint
+             WHERE conrelid = 'tasks'::regclass
+               AND contype = 'p'
+               AND pg_get_constraintdef(oid) ILIKE '%(owner, repo, issue_number)%'
+        ) THEN
+            ALTER TABLE tasks DROP CONSTRAINT IF EXISTS tasks_pkey;
+            ALTER TABLE tasks ADD CONSTRAINT tasks_pkey PRIMARY KEY (owner, repo, issue_number);
+        END IF;
+    ELSE
+        RAISE WARNING 'tasks: skipping PK swap — % rows still have NULL owner/repo/issue_number', n_orphaned;
+    END IF;
+END $$;
+
+-- 4. Reduce the legacy task_id column to an integer mirror of issue_number.
+--    Keep the column for one release so any code we missed still reads
+--    something sensible. The column is no longer a unique key.
+ALTER TABLE tasks ALTER COLUMN task_id DROP NOT NULL;
+UPDATE tasks SET task_id = issue_number::TEXT WHERE issue_number IS NOT NULL;
+COMMENT ON COLUMN tasks.task_id IS
+  'deprecated: equal to issue_number::text — use (owner, repo, issue_number) instead';
+
+COMMIT;