Refactor benchmarking to use LSV with cache pulled from supabase by ArjunS07 · Pull Request #27 · formula-code/datasmith

ArjunS07 · 2026-05-24T21:04:40Z

Adds schema for LSV cache in table
Moves the complete Harbor adapter for FormulaCode into this repository
Adds instructions on how to use LSV benchmarking to the agent context

ArjunS07 · 2026-05-24T21:04:52Z

@copilot resolve the merge conflicts in this pull request

ArjunS07 · 2026-05-24T21:30:53Z

The merge strategy folded in snapshot downloads into the new LSV-based code (which contained a separate fix, downloading all oracle artifacts in one go). This required a few serious changes to the main branch's code.

The primary divergence was in the adapter. Main still had the previous two-step architecture (parser computes reward, upload.py uploads to Supabase separately, snapshots downloaded at trial-time). The branch collapses this into parser.py, which owns all post-trial persistence. Snapshots are pre-baked by the runner before the trial starts.

Specifically,

adapter.py: Main had a one-line docstring on generate_task_dir; branch had an expanded docstring explaining oracle_artifacts, auto-populated env keys, and the rec.task_id rename (from
rec.task_dir_name). Kept the branch's version in full.
parser.py: Main retained the old fetch_oracle_benchmarks(base_url, owner, repo, issue_number) signature. The branch refactored it to take a single task_id: str, adding two helpers:
_datasmith_headers() (auth header builder) and _parse_task_id() (splits owner_repo_ → tuple).
Kept the branch's version throughout.

test.sh had two conflicts:

Main added a runtime download_snapshots block (pulling oracle snapshots from Supabase inside the container using SUPABASE_ANON_KEY). Branch removed it, so snapshots are now pre-baked into the image by lsv_init.py at setup time. Kept branch's removal.
Main called parser.py --owner --repo --issue-number plus a separate upload.py invocation. Branch replaced this with an artifact-staging block (tars /logs/ into run_artifacts.tar.gz) followed by parser.py --task-id --agent-key — parser now owns all Supabase persistence. Kept branch's version.

upload.py had a modify/delete conflict. Branch deleted the file (its upload responsibilities were absorbed into parser.py and lsv_cache_writeback.py). Main added a download_snapshots function to it. Kept the deletion, since download_snapshots was only called from the test.sh block we also removed.

Copilot

Pull request overview

This PR refactors the Harbor/LSV benchmarking flow to pull and write back cached LSV artifacts from datasmith’s Supabase (deps DB, baselines keyed by resource attrs + detected CPU), and vendors a standalone Harbor adapter for FormulaCode into this repo.

Changes:

Add/adjust Supabase schema for LSV cache + Harbor run metadata (including new CPU-specific cache keying).
Update the Harbor stage-7 runner + adapter/templates to stage cache into images and support in-container cache fetch/writeback.
Extend task templates/instructions to document the benchmark environment and add new orchestration steps (cache writeback, artifact staging, snapshot handling).

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
supabase/migrations/00016_oracle_runs_compat.sql	Introduces legacy compatibility columns and initial LSV cache tables.
supabase/migrations/00017_lsv_cache_refactor.sql	Moves deps pointer onto `pull_requests`, expands cache key columns, adds `harbor_runs.artifacts_storage_url`.
supabase/migrations/00018_lsv_cache_drop_cpu_model.sql	Removes `cpu_model` from cache PK to improve hit rate.
supabase/migrations/00019_lsv_cache_add_detected_cpu.sql	Reintroduces CPU differentiation via `detected_cpu_model` in the cache PK.
src/datasmith/runners/harbor_healthcheck.py	Stages deps DB cache, threads resource identity, adds job wall-clock cap, uploads artifacts/snapshots, writes `harbor_runs`.
src/datasmith/harbor_adapter/utils.py	Adds shell quoting for exported env and TOML escaping for `[verifier.env]`.
src/datasmith/harbor_adapter/template/Dockerfile	Bakes `/opt/lsv` scripts + pre-staged cache directory into the image.
src/datasmith/harbor_adapter/template/setup.sh	Inlines rendered env exports and makes `HARBOR_AGENT_NAME` configurable.
src/datasmith/harbor_adapter/template/test.sh	Adds cache writeback invocation, snapshot exfil staging, artifact tar staging, and switches to parser-owned persistence.
src/datasmith/harbor_adapter/template/parser.py	Refactors reward parsing toward datasmith Supabase + in-container persistence.
src/datasmith/harbor_adapter/template/lsv_init.py	Adds cache layering (deps DB staged, baselines fetched, snapshots staged) and writes cache state/attrs sidecars.
src/datasmith/harbor_adapter/template/lsv_cache_writeback.py	New stdlib-only writeback script to upload deps DB + upsert baselines row.
src/datasmith/harbor_adapter/template/instruction.md	Adds concrete environment usage instructions and termination procedure.
src/datasmith/harbor_adapter/adapter.py	Vendors/extends adapter to support standalone usage + pre-staged oracle artifacts.
src/datasmith/harbor_adapter/init.py	Exposes oracle-artifact fetch helpers as public API.
src/datasmith/harbor_adapter/template/upload.py	Removes legacy upload script in favor of new persistence path.
.gitignore	Ignores `blacklist.txt`.

Comments suppressed due to low confidence (1)

src/datasmith/harbor_adapter/template/parser.py:956

fetch_oracle_benchmarks() was refactored to take (base_url, task_id), but main() still calls it with (base_url, args.owner, args.repo, args.issue_number). This is a TypeError and will break advantage computation for agent runs. Update the call site (or keep the old function signature) so they align.

        # Fetch oracle data from datasmith Supabase
        base_url = os.environ.get("DATASMITH_SUPABASE_URL", "")
        if base_url:
            oracle_benchmarks = fetch_oracle_benchmarks(
                base_url, args.owner, args.repo, args.issue_number
            )

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        identifier handy. SkyRL / direct ``harbor trials start`` callers
+        get the auto-populated values for free.
+        """
+        out_dir = self.out_root / rec.task_id


+# ── Tunable constants (overridable via tokens.env per CLAUDE.md) ────────────
+# Relative path inside each task's environment/ directory where cached LSV
+# artifacts get staged. The Dockerfile's `COPY cache/ /opt/lsv/cache/`
+# directive bakes this into the image so lsv_init.py can read from
+# /opt/lsv/cache/ at setup-time (Harbor's verifier mount of /tests/ doesn't
+# exist yet during setup.sh).
+DATASMITH_LSV_CACHE_DIRNAME: str = os.environ.get("DATASMITH_LSV_CACHE_DIRNAME", "cache")


+        # Per-task verifier_env: base creds + full cache identity. We thread
+        # cpu_count/mem_bytes through here too so lsv_init.py and
+        # lsv_cache_writeback.py read identical values via setup.sh / test.sh
+        # render_env (Harbor's [verifier.env] does not reach setup.sh's
+        # process env on daytona — see render_env workaround below).
+        verifier_env = dict(base_env)
+        verifier_env["LSV_TASK_ID"] = rec.task_id
+        verifier_env["LSV_DEPS_BUCKET"] = DATASMITH_LSV_DEPS_BUCKET
+        verifier_env["LSV_ENV"] = attrs["env"]
+        verifier_env["LSV_CONTAINER_NAME"] = attrs["container_name"]
+        verifier_env["LSV_IMAGE_DIGEST"] = attrs["image_digest"]
+        verifier_env["LSV_MACHINE_CLASS"] = attrs["machine_class"]
+        verifier_env["LSV_DOCKER_HOST_ID"] = attrs["docker_host_id"]
+        verifier_env["LSV_CPU_COUNT"] = str(attrs["cpu_count"])
+        verifier_env["LSV_MEM_BYTES"] = str(attrs["mem_bytes"])


+        # Stage cached files AFTER generate_task creates the directory tree.
+        # The adapter pre-creates an empty environment/cache/ (so the Dockerfile
+        # COPY directive always has something to copy); we drop the deps DB
+        # into that same directory. Baselines are NOT pre-staged — they're
+        # resource-keyed and the runner doesn't know the sandbox specs;
+        # lsv_init.py fetches them at runtime via PostgREST.
+        cache_dir = task_dir / rec.task_id / "environment" / DATASMITH_LSV_CACHE_DIRNAME
+        deps_staged = _stage_lsv_deps_db(client, cache_dir, deps_db_url=deps_db_url)
+        if deps_staged:
+            logger.info("LSV cache stage hit for %s: deps_db", rec.task_id)
+


@@ -451,6 +925,9 @@ async def run_harbor_healthcheck(
            continue
        row = _row_from_trial(trial, task_id_map, environment=environment)
        if row is not None:
+            artifacts_key = _upload_run_artifacts_for_row(artifacts_client, row, job_dir=job_dir)
+            if artifacts_key:
+                row["artifacts_storage_url"] = artifacts_key
            rows.append(row)

    n_success = sum(1 for r in rows if r["status"] == "success")
@@ -460,13 +937,25 @@ async def run_harbor_healthcheck(
        if r["status"] == "success" and r["max_speedup"] is not None and r["max_speedup"] >= MIN_SPEEDUP_GATE
    )
    logger.info(
-        "Harbor job '%s' done: %d/%d trials succeeded, %d >= %.2fx gate",
+        "Harbor job '%s' done: %d/%d trials succeeded, %d >= %.2fx gate%s",
        job_name,
        n_success,
        len(rows),
        n_fast,
        MIN_SPEEDUP_GATE,
+        " (job timed out)" if timed_out else "",
    )

    _insert_harbor_runs(rows)
+    if timed_out:
+        # Half-cancelled jobs can produce technically-successful trials that
+        # we don't trust enough to promote as the per-PR baseline pointer.
+        # Skip both the snapshot upload and the pull_requests pointer update
+        # — the harbor_runs rows are still recorded for triage.
+        logger.warning(
+            "Harbor job '%s' timed out; skipping baseline pointer + snapshot upload",
+            job_name,
+        )
+    else:
+        _update_baseline_pointers(rows, environment=environment, job_dir=job_dir)
    return rows


 TASK_ID="${ISSUE_NUMBER}"
 export OWNER REPO ISSUE_NUMBER TASK_ID


+# pointers. Writes /logs/verifier/run_id.txt as a sentinel so the runner's
+# post-trial sweep can tell which trials it must NOT re-write rows for.
+echo "[$(ts)] [test] Computing reward + persisting harbor_runs..."
+python /opt/lsv/parser.py --task-id "${TASK_ID}" --agent-key "${AGENT_KEY}"


 def main() -> None:
    parser = argparse.ArgumentParser(description="Compute reward from LSV results")
    parser.add_argument("--owner", required=True, help="Repository owner")
    parser.add_argument("--repo", required=True, help="Repository name")
    parser.add_argument("--issue-number", required=True, type=int, help="PR number")
    parser.add_argument(
        "--agent-key", default="agent", help="Agent key (e.g., oracle, terminus-2)"
    )
    args = parser.parse_args()

    # Load all sidecars written by setup.sh / test.sh. Each helper returns
    # an empty/default payload if its file is missing, so a partially-run
    # trial still produces a well-formed reward.json.
    patch_info = load_patch_info(LOG_DIR)
    timings = load_timings(LOG_DIR)
    snapshot_block = summarize_snapshots(LOG_DIR)
    setup_status = load_setup_status(LOG_DIR)

    # Load LSV results
    lsv_results = load_lsv_results(LSV_DIR)
    lsv_init_summary = summarize_lsv_init(lsv_results)
+
+    # Generate run_id ONCE here — persist_supabase uses it as the harbor_runs
+    # PK, the sentinel file at /logs/verifier/run_id.txt uses it to tell the
+    # runner's failure-path sweep "this trial already wrote a row."
+    run_id = str(uuid.uuid4())
+
+    is_oracle = args.agent_key == "oracle"
+
    if not lsv_results:
        print("[parser] No LSV results found. Writing zero reward.")
-        write_reward(
+        reward = write_reward(
            REWARD_DIR,
            None,
            None,
            {},
            {"level4": 0.0},
            False,
            snapshot_block,
            patch=patch_info,
            lsv_init_summary=lsv_init_summary,
            lsv_measure_raw=None,
            pytest_summary={},
            timings=timings,
            setup_status=setup_status,
+            is_oracle=is_oracle,
        )
+        # Even on the no-results path we still try to land a harbor_runs
+        # row (status will classify as setup_failed / lsv_init_empty / etc.)
+        # so the trial appears in dashboards. If persistence lands, write
+        # the sentinel before exiting non-zero.
+        if persist_supabase(run_id=run_id, task_id=args.task_id, reward=reward):
+            (REWARD_DIR / "run_id.txt").write_text(run_id)
        sys.exit(1)



+  * deps DB (per-PR, structural)         — Supabase Storage bucket ``lsv-deps``
+                                            + ``lsv_dep_cache`` row


+        ``oracle_artifacts``, when provided, pre-stages the oracle's deps DB,
+        baselines JSON, and snapshot tarball into ``environment/cache/`` so
+        the resulting image short-circuits the LSV survey + snapshot-capture
+        passes. Required for non-oracle agent runs; ignored (but harmless)
+        for oracle runs.


ArjunS07 added 8 commits May 2, 2026 21:56

Initial caching code, faulty

45ca0e1

Add working code to cache initial survey, use LSV, cache LSV benchmarks

64fb1b3

Refactor bucket name

c017b9c

Fix cache to be CPU specific

488c506

Refactor to make adapter work in standalone case

ca67175

Add container correct instruction

0228cc2

Enforce pytest and snapshot gate before speedup reward

be12625

add updated instruction

f0d746c

Copilot started work on behalf of ArjunS07 May 24, 2026 21:05 View session

Copilot stopped work on behalf of ArjunS07 due to an error May 24, 2026 21:05
Request session.create failed with message: Model "gpt-5.3-codex" is not available.

ArjunS07 added 2 commits May 24, 2026 21:12

Merge remote-tracking branch 'origin/main' into lsv-cache-integration

b409b2b

Merge conflict

1737101

ArjunS07 requested review from atharvas and Copilot May 24, 2026 21:30

Copilot started reviewing on behalf of ArjunS07 May 24, 2026 21:31 View session

Copilot AI reviewed May 24, 2026

View reviewed changes

ArjunS07 closed this Aug 26, 2026

Navigation Menu

Uh oh!

Refactor benchmarking to use LSV with cache pulled from supabase - #27

Refactor benchmarking to use LSV with cache pulled from supabase#27
ArjunS07 wants to merge 10 commits into
mainfrom
lsv-cache-integration

ArjunS07 commented May 24, 2026

Uh oh!

ArjunS07 commented May 24, 2026

Uh oh!

ArjunS07 commented May 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		TASK_ID="${ISSUE_NUMBER}"
		export OWNER REPO ISSUE_NUMBER TASK_ID

		* deps DB (per-PR, structural) — Supabase Storage bucket ``lsv-deps``
		+ ``lsv_dep_cache`` row

Uh oh!

Conversation

ArjunS07 commented May 24, 2026

Uh oh!

ArjunS07 commented May 24, 2026

Uh oh!

ArjunS07 commented May 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants