Cache initialize_diffcheck cost

CLAUDE.md

@@ -73,7 +73,7 @@ The dependency direction is roughly `utils/` → `github/` + `docker/` + `agents
 4. **resolve_packages** — Emit one dependency **seed** per commit, and the story of how it was reached. Six units compose it: `discover` picks the packaging root, `declare` reads only what the project states it needs, `interpreter` walks a declared ladder (`requires-python` → trove classifiers → `asv.conf.json` `pythons` → newest release at commit date) and records the rung in `interpreter_source`, `pin` runs one `uv pip compile` with the commit date as `--exclude-newer`, `probe` dry-runs the result, and the row is written. What it deliberately does not read: `requirements*.txt` globs, `environment.yml`, and import statements — so a project that declares nothing gets an empty seed and says so, rather than a list of invented PyPI names. Benchmark tooling (`asv`, `pytest`, `hypothesis`, `setuptools`, `wheel`, `pip`, `versioneer`) is stripped from both the declared set and the compiled one: the base image owns it, and a second owner only starts a version fight. **The stage gates nothing.** `can_install` is retained, nullable, and no longer read or written; `probe_status` (`installable` → `unresolved` → `failed` → `empty`) orders the stage 5 queue best-first and excludes nobody. Stage 6 is the sole arbiter of buildability, because it is the only stage that builds in the real container.
 5. **render_problems** — Scrape linked issues and render deconstructed problem contexts
 6. **synthesize_images** — Agent-based Docker build context synthesis (uses env_payload/python_version from stage 4)
-7. **harbor_healthcheck** — Run every synthesized container through Harbor's oracle agent, record per-benchmark speedups to `harbor_runs`. Supports local Docker and Daytona via `--harbor-environment`; the row records which one in `harbor_runs.environment`. Local runs are useful for iteration; only Daytona runs gate stage 8.
+7. **harbor_healthcheck** — Run every synthesized container through Harbor's oracle agent, record per-benchmark speedups to `harbor_runs`. Supports local Docker and Daytona via `--harbor-environment`; the row records which one in `harbor_runs.environment`. Local runs are useful for iteration; only Daytona runs gate stage 8. An **LSV cache** (`DATASMITH_LSV_CACHE_ENABLED`, default on) lets a repeat trial skip LSV's two expensive passes: the runner bakes the cached survey (`lsv_deps_cache`) into the image and injects datasmith Supabase creds + an 11-column resource key so `lsv_init.py` fetches the cached baselines (`lsv_baseline_cache`) and passes `force=False` only on a full hit; the oracle trial writes both back. It is a pure cost optimization — every miss or error degrades to today's `force=True`, changing no reward or gate — and requires `SUPABASE_URL` to be the `db.formulacode.org` tunnel so the trial container can reach it.
 8. **publish** — Build, verify, and publish Docker images to DockerHub. Two gates, and a PR must clear both: it needs at least one successful `harbor_runs` row whose `max_speedup >= 1.05` in an admitted environment (`DATASMITH_PUBLISH_ENVIRONMENTS`, default **daytona** only), *and* its `candidate_containers` row must be `verification_state = 'verified'`. The harbor row says the container is fast; `verification_state` says it is honest, and neither substitutes for the other — `harbor_runs` outlives the container generation that produced it, so a pre-honesty-gate row can carry a fast trial. That second gate is deliberately **not** a knob.
 9. **scrape_benchmark_source** — For each `(owner, repo)` in `candidate_containers`, check out the repo at its container SHA, AST-parse every ASV-style benchmark function under the repo's `benchmark_dir`, and upsert one row per `(owner, repo, benchmark_without_params)` into `benchmark_codes` for the FormulaCode website's data sync.
 
@@ -271,6 +271,8 @@ Note that `00016` and `00017` used `GRANT ALL ... TO anon` rather than `GRANT SE
 | `packages` | One seed per `(owner, repo, sha)`: `env_payload` (pinned deps) and `python_version`, plus `interpreter_source` (which ladder rung chose that interpreter), `primary_root`, `requires_python`, the advisory `probe_status` / `probe_log`, `dropped_requirements` (JSON-encoded text, like `env_payload` — every requirement that was refused, with its reason), and provenance: `resolver_version`, `uv_version`, `resolved_at`, `cutoff_used` (null when the commit-date cutoff had to be relaxed). `can_install` is deprecated — nullable, no longer read or written; `resolver_version = 'legacy'` marks the rows the predecessor wrote. | Stage 4 |
 | `candidate_containers` | Successful agent-generated `build_pkg_sh` / `build_run_sh` per SHA, plus `build_manifest` (sealed build facts merged with verify observations), `manifest_warnings` (non-fatal invariant ids), and `verification_state` (`unverified` / `verified`, with `verified_at`). `build_manifest IS NULL` identifies rows built before manifests existed; `verification_state = 'unverified'` identifies rows built before the honesty gate applied to them. | Stage 6 (on success) |
 | `harbor_runs` | One row per Harbor oracle trial for a synthesized container: `max_speedup`, `geomean_speedup`, `n_benchmarks`, `wallclock_sec`, `reward_payload`, `status`. One-to-many FK on `candidate_containers(owner, repo, sha)`. | Stage 7 |
+| `lsv_baseline_cache` | Resource-keyed cache of LSV base-commit baseline timings so stage 7 skips the timing pass on a repeat trial. 11-column PK pins every fact that moves a timing (task + env + image + host/machine_class + cgroup pins + in-sandbox `detected_cpu_model`); `baselines` JSONB is `session.export_baselines()`. Advisory, no FK. Oracle trials write it; every trial reads it. A miss falls back to `force=True`. | Stage 7 (oracle writeback) |
+| `lsv_deps_cache` | Task-keyed (`owner, repo, issue_number` PK) cache of the LSV coverage **survey** — the `lightspeed_deps.db` SQLite file, baselines stripped, as `deps_db` BYTEA — so stage 7 skips the survey pass. Resource-independent (survey depends only on code, not CPU), hence one row per task. Required for the baseline cache to load at all (`load_baselines` needs the surveyed DB on disk first). | Stage 7 (oracle writeback) |
 | `benchmark_information` | Per-benchmark speedup measurements from terminal-bench eval runs: one row per (run, owner/repo/issue, benchmark, agent, model). `speedup` is `(agent/nop)/(oracle/nop)` so 1.0 = parity with the human expert. `benchmark_type` (`time`/`mem`/`peakmem`/`track`) is a generated column derived from the ASV naming convention. Loaded out-of-band via `scripts/load_benchmark_information.py`. | (manual) |
 | `benchmark_codes` | One row per `(owner, repo, benchmark_without_params)` carrying the Python source of each ASV benchmark function plus its setup. Joined to `benchmark_information` on `(owner, repo, benchmark_name)` by the FormulaCode website. | Stage 9 |
 | `error_logs` | Per-attempt synthesis results: agent output, failure stage/return code, error messages | Stage 6 (`Synthesizer._log_attempt`) |
@@ -285,7 +287,7 @@ Note that `00016` and `00017` used `GRANT ALL ... TO anon` rather than `GRANT SE
 
 ### Migrations
 
-SQL migrations live in `supabase/migrations/`, numbered `00001_` upward (currently through `00029_`). The sequence has gaps because numbers get claimed on branches before they land: `00018_lsv_cache_drop_cpu_model.sql` lives on `origin/lsv-cache-integration`, and `00024` is authored in a separate working tree (per `00025`'s header) — `00026` re-lands that same table under a number that is free here. So check other branches before claiming a number, and record in the file header why you skipped one. `00027_pull_requests_window_indexes.sql` adds the two indexes the stage 2–5 window predicates need on `pull_requests` — `merged_at` for the stage-wide scan, `(owner, repo, merged_at)` for the per-repository skip set — and deliberately grants nothing to `anon`. `00028_packages_resolution_v2.sql` carries the stage 4 redesign's provenance columns, and `00029_candidate_containers_verification_state.sql` adds `verification_state` — every pre-existing row defaults to `unverified`, because the corpus predates the honesty gate and has not earned the label.
+SQL migrations live in `supabase/migrations/`, numbered `00001_` upward (currently through `00032_`). The sequence has gaps because numbers get claimed on branches before they land: `00018_lsv_cache_drop_cpu_model.sql` lives on `origin/lsv-cache-integration`, and `00024` is authored in a separate working tree (per `00025`'s header) — `00026` re-lands that same table under a number that is free here. So check other branches before claiming a number, and record in the file header why you skipped one. `00027_pull_requests_window_indexes.sql` adds the two indexes the stage 2–5 window predicates need on `pull_requests` — `merged_at` for the stage-wide scan, `(owner, repo, merged_at)` for the per-repository skip set — and deliberately grants nothing to `anon`. `00028_packages_resolution_v2.sql` carries the stage 4 redesign's provenance columns, and `00029_candidate_containers_verification_state.sql` adds `verification_state` — every pre-existing row defaults to `unverified`, because the corpus predates the honesty gate and has not earned the label. `00031_lsv_baseline_cache.sql` and `00032_lsv_deps_cache.sql` add the stage-7 LSV cache tables (baseline timings and the survey deps DB); both are private (`GRANT SELECT ... TO grafana_ro`, no anon) and squash the old `origin/lsv-cache-integration` cpu-model churn (`00016`→`00019`) into one clean pair.
 
 To apply a new migration against the local instance:
 

src/datasmith/harbor_adapter/adapter.py

@@ -98,13 +98,32 @@ def _copy_template_files(self, paths: HarborTaskPaths) -> None:
             "entrypoint.sh",
             "lsv_init.py",
             "lsv_measure.py",
+            "lsv_cache_writeback.py",
             "parser.py",
             "upload.py",
             "pytest_runner.py",
             "jinja_patch_plugin_pandas.py",
         ]:
             copy2(self.template_dir / name, paths.environment_dir / name)
 
+    def _write_cache_files(self, paths: HarborTaskPaths, deps_db: bytes | None) -> None:
+        """Populate ``environment/cache/`` with the pre-surveyed LSV deps DB.
+
+        The directory is created unconditionally -- even on a cache miss -- and
+        seeded with a ``.gitkeep`` so the Dockerfile's ``COPY cache/`` directive
+        always has a source and the build never fails. When ``deps_db`` is
+        present (a hit staged by the runner), it is written as
+        ``lightspeed_deps.db``; the Dockerfile bakes it to
+        ``/opt/lsv/cache/lightspeed_deps.db``, where lsv_init.py stages it before
+        ``load_baselines`` (which requires the surveyed DB on disk first). A miss
+        leaves only ``.gitkeep``, so lsv_init falls through to force=True.
+        """
+        cache_dir = paths.environment_dir / "cache"
+        cache_dir.mkdir(parents=True, exist_ok=True)
+        (cache_dir / ".gitkeep").write_text("")
+        if deps_db:
+            (cache_dir / "lightspeed_deps.db").write_bytes(deps_db)
+
     def _write_instruction_md(self, rec: FormulaCodeRecord, paths: HarborTaskPaths) -> None:
         """Generate instruction.md file."""
         instruction_content = render_instruction_md(rec.instructions)
@@ -146,6 +165,7 @@ def _write_test_files(
         paths: HarborTaskPaths,
         run_pytest: bool = True,
         rounds: int = 1,
+        render_env: dict[str, str] | None = None,
     ) -> None:
         """Generate test files (test.sh, config.json)."""
         # test.sh
@@ -156,6 +176,7 @@ def _write_test_files(
             issue_number=rec.issue_number,
             run_pytest=run_pytest,
             rounds=rounds,
+            render_env=render_env,
         )
         test_sh_path = paths.tests_dir / "test.sh"
         test_sh_path.write_text(test_sh_content)
@@ -179,6 +200,7 @@ def _write_solution_files(
         rec: FormulaCodeRecord,
         paths: HarborTaskPaths,
         rounds: int = 1,
+        render_env: dict[str, str] | None = None,
     ) -> None:
         """Generate solution files (solve.sh, setup.sh)."""
         # solve.sh
@@ -195,6 +217,7 @@ def _write_solution_files(
             issue_number=rec.issue_number,
             rounds=rounds,
             extra_setup_commands=extra_setup_commands,
+            render_env=render_env,
         )
         setup_sh_path = paths.tests_dir / "setup.sh"
         setup_sh_path.write_text(setup_sh_content)
@@ -211,9 +234,16 @@ def generate_task(
         rounds: int = DATASMITH_LSV_ROUNDS,
         verifier_env: dict[str, str] | None = None,
         expected_n: int | None = None,
+        render_env: dict[str, str] | None = None,
+        deps_db: bytes | None = None,
     ) -> Path:
         """Generate a complete Harbor task directory for the given FormulaCodeRecord.
 
+        ``render_env`` is baked as ``export K=V`` lines into setup.sh and test.sh
+        (see ``render_run_setup_sh``). Stage 7 uses it to hand the LSV baseline
+        cache its datasmith Supabase creds and resource key; ``None`` bakes
+        nothing, leaving both scripts byte-identical to the pre-cache output.
+
         ``expected_n`` is the operator-declared count of benchmarks this PR
         should impact, read from ``formulacode_task_overrides``. It is injected
         into the trial container as ``FORMULACODE_EXPECTED_N`` -- the producer
@@ -226,6 +256,11 @@ def generate_task(
         That is the common case (the column is hand-declared and usually
         NULL), and the invariant then skips. Emitting a key that is always
         empty would make the wiring look live when it is not.
+
+        ``deps_db`` is the pre-surveyed LSV deps DB the runner fetched from
+        ``lsv_deps_cache`` for this task; it is baked into the image so lsv_init
+        can skip the survey pass. ``None`` bakes only an empty ``cache/``
+        placeholder, leaving lsv_init on force=True.
         """
         out_dir = self.out_root / rec.task_dir_name
         out_dir.mkdir(parents=True, exist_ok=True)
@@ -235,14 +270,15 @@ def generate_task(
 
         # Copy static template files
         self._copy_template_files(paths)
+        self._write_cache_files(paths, deps_db)
 
         # Generate all task files
         self._write_instruction_md(rec, paths)
         if expected_n is not None:
             verifier_env = {**(verifier_env or {}), "FORMULACODE_EXPECTED_N": str(expected_n)}
         self._write_task_toml(rec, paths, timeout_sec, cpus, memory, storage, verifier_env=verifier_env)
         self._write_environment_files(rec, paths)
-        self._write_test_files(rec, paths, run_pytest, rounds)
-        self._write_solution_files(rec, paths, rounds)
+        self._write_test_files(rec, paths, run_pytest, rounds, render_env=render_env)
+        self._write_solution_files(rec, paths, rounds, render_env=render_env)
 
         return out_dir

src/datasmith/harbor_adapter/template/Dockerfile

@@ -3,19 +3,24 @@ FROM {{ base_image }}
 # Install required system packages
 RUN apt-get update && apt-get install -y git tmux asciinema
 RUN curl -LsSf https://astral.sh/uv/0.7.13/install.sh | sh
-RUN mkdir -p /logs /opt/lsv /tests
+RUN mkdir -p /logs /opt/lsv /opt/lsv/cache /tests
 
 # Copy environment setup scripts
 COPY entrypoint.sh /entrypoint.sh
 
 # Bake LSV helper scripts into the image so setup.sh can find them before
 # Harbor's verifier mounts /tests/ (verifier runs AFTER setup.sh).
-COPY lsv_init.py lsv_measure.py parser.py upload.py pytest_runner.py jinja_patch_plugin_pandas.py /opt/lsv/
+COPY lsv_init.py lsv_measure.py lsv_cache_writeback.py parser.py upload.py pytest_runner.py jinja_patch_plugin_pandas.py /opt/lsv/
 
 # lsv_init.py reads /tests/config.json during setup.sh — bake it in too.
 # Harbor's verifier later overwrites this with identical content.
 COPY config.json /tests/config.json
 
+# Pre-surveyed LSV deps DB (lsv_deps_cache), when the runner staged one. The
+# adapter always creates environment/cache/ (with a .gitkeep) so this COPY has a
+# source even on a cache miss; lsv_init reads lightspeed_deps.db from here.
+COPY cache/ /opt/lsv/cache/
+
 RUN chmod +x /entrypoint.sh
 ENV PYTHONPATH=/opt/lsv:${PYTHONPATH:-}