Ground up rewrite of datasmith.

.github/workflows/docs.yml

@@ -0,0 +1,48 @@
+name: Deploy Docs
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - "docs/**"
+      - "mkdocs.yml"
+      - "src/**"
+  workflow_dispatch:
+
+permissions:
+  pages: write
+  id-token: write
+
+concurrency:
+  group: pages
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out
+        uses: actions/checkout@v4
+
+      - name: Set up the environment
+        uses: ./.github/actions/setup-python-env
+
+      - name: Build docs
+        run: uv run mkdocs build --strict
+
+      - name: Upload Pages artifact
+        uses: actions/upload-pages-artifact@v3
+        with:
+          path: site
+
+  deploy:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v4

.gitignore

@@ -144,7 +144,6 @@ venv.bak/
 .ropeproject
 
 # mkdocs documentation
-/site
 
 # mypy
 .mypy_cache/
@@ -194,46 +193,27 @@ cython_debug/
 .cursorindexingignore
 
 tokens.env
-benchmark_results/
 scratch/artifacts/
-scratch/example-docker-task
-scratch/rsync-backups/
-scratch/artifacts/tarballs/
-asv-output/
-output
-*context_registry*.json
-# Database files
 *.db
-scratch/artifacts/git/
-scratch/artifacts/.git/
-scratch/notebooks/git/
-scratch/artifacts/**/*.parquet
-scratch/**/*.json
-scratch/pr_runs/arrow_36738_after.md
-scratch/pr_runs/arrow_36738_final.md
-scratch/pr_runs/arrow_36738.md
-scratch/pr_runs/arrow_adbc_2697_final.md
-scratch/pr_runs/arrow_adbc_2697.md
-scratch/pr_runs/astropy_13498_after.md
-scratch/pr_runs/astropy_13498_after2.md
-scratch/pr_runs/astropy_13498_final1.md
-scratch/pr_runs/astropy_13498_final2.md
-scratch/pr_runs/astropy_13498_run1_head.txt
-scratch/pr_runs/astropy_13498_run1.md
-scratch/pr_runs/astropy_13498_run2.md
-scratch/pr_runs/qiskit_10651_after.md
-scratch/pr_runs/qiskit_10651_after2.md
-scratch/pr_runs/qiskit_10651_final.md
-scratch/pr_runs/qiskit_10651_head.txt
-scratch/pr_runs/qiskit_10651.md
-scratch/pr_runs/qiskit_12869_final.md
-scratch/pr_runs/qiskit_12869.md
-.aider*
-
-# Dataset artifacts (kept locally, not tracked)
-dataset/
-
-# Scratch artifacts (kept locally, not tracked)
-scratch/example-repo/
-scratch/merged_context_registry_*.json
-scratch/artifacts/raw/online_dashboards.jsonl
+
+# Supabase
+supabase/.temp/
+supabase/.branches/
+
+# MkDocs build output
+site/
+
+# Node (Gemini CLI)
+node_modules/
+package-lock.json
+
+# Scratch / dev artifacts
+scratch/
+fc_record.pkl
+GEMINI.md
+PARITY_TODOs.md
+
+# One-off scripts
+backup
+rebuild
+run_interpreter.py

.pre-commit-config.yaml

@@ -6,12 +6,14 @@ repos:
       - id: check-merge-conflict
       - id: check-toml
       - id: check-yaml
+        args: [--unsafe]
       - id: check-json
         exclude: ^.devcontainer/devcontainer.json
       - id: pretty-format-json
         exclude: ^.devcontainer/devcontainer.json
         args: [--autofix, --no-sort-keys]
       - id: end-of-file-fixer
+        exclude: ^supabase/snippets/
       - id: trailing-whitespace
 
   - repo: https://github.com/astral-sh/ruff-pre-commit

AGENTS.md

@@ -3,40 +3,41 @@
 This guide helps contributors work effectively in the datasmith repository.
 
 ## Project Structure & Module Organization
-- Source: `src/datasmith/` — core modules: `agents/`, `docker/`, `scrape/`, `benchmark/`, `detection/`, `execution/`, `collation/`, `core/`.
-- Tests: `tests/` — pytest suites (e.g., `tests/test_docker_*`, `tests/agents/`).
-- Assets/Docs: `static/`, `docs/`.
-- Artifacts: `scratch/` (generated data), `dist/` (wheels). Do not commit contents.
+- Source: `src/datasmith/` — current modules: `agents/`, `docker/`, `github/`, `publish/`, `resolution/`, `runners/`, `update/`, `utils/`, plus helper modules like `filters.py` and `preflight.py`.
+- Tests: `tests/` — pytest suites in grouped folders (e.g., `tests/agents/`, `tests/docker/`, `tests/github/`, `tests/publish/`, `tests/runners/`, `tests/update/`).
+- Assets/Docs/Configs: `static/`, `docs/`, `dataset/`, `scripts/`, `supabase/`.
+- Generated artifacts: `dataset/` task outputs, `scratch/`, and `dist/` should be treated as build/runtime artifacts and not committed.
 
 ## Build, Test, and Development Commands
 - `make install` — create env with uv and install pre-commit.
-- `make check` — lock check, ruff lint/format, mypy, deptry.
+- `make check` — run lock check, pre-commit, mypy, deptry.
 - `make test` — run pytest with coverage (XML for CI/Codecov).
 - `make build` — build wheel into `dist/`.
-- `uv run <cmd>` — run tools inside the env (e.g., `uv run pytest`).
-- `uvx tox -q` — run the tox matrix (py39–py312) if tox is installed.
-- Optional: `make backup` uses `tokens.env` for `BACKUP_DIR` rsync.
+- `uv run <cmd>` — run tools inside the env (e.g., `uv run pytest`, `uv run mypy`, `uv run pre-commit run -a`).
+- `uv run python -m pytest --doctest-modules` — run doctests with the standard test suite.
+- `uvx tox -q` — run the tox matrix (py39–py312) as defined in `tox.ini`.
 - To run commands using the same environment variables as the user, use `uv run <command>`.
+- Pipeline entrypoint: `ds-update --help` and `ds-update --start-date ... --end-date ...`.
 
 ## Coding Style & Naming Conventions
 - Python 3.9–3.12. 4‑space indentation, type hints required (mypy strict; see `pyproject.toml`).
 - Lint/format via Ruff (line length 120). Run `make check` before pushing.
 - Naming: modules/functions `snake_case`, classes `CamelCase`, constants `UPPER_SNAKE_CASE`.
-- Prefer `logging` (see `src/datasmith/logging_config.py`) over prints.
+- Prefer package logging via `datasmith.utils.core` over `print`.
 
 ## Testing Guidelines
 - Framework: pytest + pytest‑cov. Place tests in `tests/` named `test_*.py`.
 - Run locally: `make test` or `uv run pytest`.
 - Coverage: Codecov target 90% (see `codecov.yaml`). Add tests for new code paths.
-- Tests must be deterministic and offline; use fakes for network calls.
+- Tests must be deterministic and offline; use fakes for network/AWS.
 
 ## Commit & Pull Request Guidelines
 - History is informal; please use clear, present‑tense summaries, optionally prefixing a subsystem tag: `docker: prune dangling layers`, `agents: improve build plan`.
-- PRs must include: description, rationale, test coverage notes, and any docs updates. Link issues. For CLI/UX changes, include sample output or screenshots.
+- PRs must include: description, rationale, test coverage notes, and any docs updates. Link issues. For CLI/UX changes, include sample output.
 - Ensure `make check` and `make test` pass; CI should be green.
 
 ## Security & Configuration Tips
-- Create `tokens.env` (ignored) for `GH_TOKEN`, `CODECOV_TOKEN`, `CACHE_LOCATION`, `BACKUP_DIR`. Never commit secrets.
+- Create `tokens.env` (git-ignored) for environment credentials (`GH_TOKENS` or `GH_TOKEN`, `SUPABASE_URL`, `SUPABASE_KEY`, `CACHE_LOCATION`) and any service tokens used by your workflow.
 - Docker tooling exists in `src/datasmith/docker/`; validate locally before pushing remote runs.
 
 ## Agent‑Specific Instructions

CLAUDE.md

@@ -0,0 +1,113 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What is this project?
+
+DataSmith is the Python toolchain for building the **FormulaCode** dataset — a benchmark of 67+ repositories with 964+ performance-improving commits, designed to evaluate LLMs' ability to optimize real-world codebases. It scores LLMs relative to the human-authored speedup using ASV (Airspeed Velocity) benchmarks rather than binary pass/fail tests.
+
+## Common commands
+
+```bash
+# Setup
+make install          # Create venv with uv, install pre-commit hooks
+make check            # Lint (ruff), format, type-check (mypy), dependency check (deptry)
+make test             # Run pytest with coverage
+
+# Development
+uv run pytest tests/                         # Run all tests
+uv run pytest tests/test_docker_context.py   # Run a single test file
+uv run pytest -xvs tests/test_scraper.py     # Verbose, stop on first failure
+uv run mypy                                  # Type checking
+uv run pre-commit run -a                     # Run all pre-commit hooks
+
+# Dataset verification (iterative Docker build debugging)
+python dataset/verify.py --task dataset/formulacode_verified/<owner_repo>/<sha>
+
+# Pipeline update (monthly, the primary entrypoint for DataSmith)
+ds-update --start-date YYYY-MM-DD --end-date YYYY-MM-DD          # Run all 6 stages
+ds-update --start-date 2026-01-01 --end-date 2026-01-31 --stage 4  # Run a single stage
+ds-update --start-date 2026-01-01 --end-date 2026-01-31 --resume   # Resume from last completed
+ds-update --help                                                    # See all options
+```
+
+## Architecture
+
+### Source modules (`src/datasmith/`)
+
+| Module | Purpose |
+|--------|---------|
+| `core/` | Infrastructure: SQLite storage (`storage.py`), GitHub/Codecov API clients (`api/`), caching (`cache/`), git operations (`git/`), data models (`models/`) |
+| `docker/` | Docker image lifecycle: build context creation (`context.py` — largest file), orchestration, DockerHub publishing (`dockerhub.py`), AWS Batch integration, multi-stage shell scripts |
+| `scrape/` | GitHub PR scraping, report generation with Jinja2 templates, issue extraction, code coverage integration. See `src/datasmith/scrape/CLAUDE.md` for report builder data flow |
+| `agents/` | DSPy-based LLM agents for build context synthesis and performance commit classification |
+| `resolution/` | Dependency resolution: parses pyproject.toml/setup.py/setup.cfg, resolves pinned deps via `uv pip compile`, validates installability. Pipeline stage 4. |
+| `execution/` | Commit collection/filtering from GitHub, Python environment management |
+| `detection/` | Performance breakpoint detection in benchmark results |
+| `benchmark/` | ASV benchmark collection |
+| `collation/` | Data aggregation |
+
+### Pipeline stages (`ds-update`)
+
+1. **scrape_repos** — Fetch repository metadata from GitHub
+2. **scrape_commits** — Scrape merged PR commits and patches
+3. **classify_prs** — LLM-based performance classification
+4. **resolve_packages** — Resolve Python dependencies via `uv pip compile`, persist to `packages` table
+5. **synthesize_images** — Agent-based Docker build context synthesis (uses env_payload/python_version from stage 4)
+6. **publish** — Build, verify, and publish Docker images to DockerHub
+
+### Dataset verification (`dataset/`)
+
+Each task lives in `dataset/formulacode_verified/<owner_repo>/<sha>/` with a multi-stage Dockerfile, shell build scripts, and validation scripts. The verification loop is:
+1. Run `verify.py` → check `failure.json` for errors → edit `docker_build_pkg.sh` and/or `docker_build_run.sh` → rerun until `verification_success.json` appears.
+
+**Only modify `docker_build_pkg.sh` and `docker_build_run.sh`** during verification fixes. See `dataset/CLAUDE.md` and `dataset/AGENTS.md` for detailed guidance.
+
+## Code quality standards
+
+- **Python**: 3.9–3.12, type hints required (mypy strict)
+- **Linting**: Ruff with 120-char line length
+- **Testing**: pytest + pytest-cov
+- **Build**: hatchling backend, uv for dependency management
+- **CI**: GitHub Actions runs `make check` + tests on Python 3.11 and 3.12
+
+## Supabase (local)
+
+DataSmith uses a **local Supabase** instance for all persistent state. Connection details live in `tokens.env`:
+
+- `SUPABASE_URL=http://127.0.0.1:54321` (PostgREST API)
+- `SUPABASE_KEY=sb_secret_...` (service-role key)
+- Direct Postgres: `host=127.0.0.1 port=54322 dbname=postgres user=postgres password=postgres`
+
+### Key tables
+
+| Table | Purpose | Populated by |
+|-------|---------|-------------|
+| `pull_requests` | All scraped PRs with classification, patches, rendered problems, container names | Stages 1-3, 5-6 |
+| `packages` | Resolved `env_payload` (pinned deps) and `python_version` per commit | Stage 4 |
+| `candidate_containers` | Successful agent-generated `build_pkg_sh` / `build_run_sh` per SHA | Stage 6 (on success) |
+| `error_logs` | Per-attempt synthesis results: agent output, failure stage/return code, error messages | Stage 6 (`Synthesizer._log_attempt`) |
+| `runner_progress` | Live progress counters (total/completed/failed) per pipeline run | `BaseRunner` (all stages) |
+| `runner_failures` | One row per item failure with error message + traceback | `BaseRunner._log_failure` |
+| `candidate_prs` | Deconstructed PR context components for re-rendering | Stage 5 |
+| `hook_cache` | Memoization cache for `@supabase_cached` decorated functions | Various |
+
+### Migrations
+
+SQL migrations live in `supabase/migrations/` (numbered `00001_` through `00007_`). To apply a new migration against the local instance:
+
+```python
+import psycopg2
+conn = psycopg2.connect(host='127.0.0.1', port=54322, dbname='postgres', user='postgres', password='postgres')
+conn.autocommit = True
+conn.cursor().execute(open('supabase/migrations/00007_error_logs.sql').read())
+# Don't forget: GRANT ALL ON <table> TO anon, authenticated;
+```
+
+### Querying
+
+Use `datasmith.utils.db.fetch_all(table, select=..., filters=..., gte_filters=..., ...)` for paginated reads, or `get_client()` for direct Supabase client access.
+
+## Environment setup
+
+Requires a `tokens.env` file in the repo root with `GH_TOKEN`, `CACHE_LOCATION`, `SUPABASE_URL`, `SUPABASE_KEY`, and optionally `DSPY_*` vars for LLM backends and `DOCKERHUB_*` vars for publishing. See README.md for the full template.

Makefile

@@ -5,22 +5,6 @@ install: ## Install the virtual environment and install the pre-commit hooks
 	@uv pip install -e .
 	@uv run pre-commit install
 
-.PHONY: backup
-backup: ## Create a backup of the datasets, results, and analysis directories
-	@echo "Syncing backup mirror with rsync"
-	@/usr/bin/env bash -euo pipefail -c '\
-		if [ ! -f tokens.env ]; then \
-			echo "❌ Error: tokens.env file not found"; exit 1; \
-		fi; \
-		BACKUP_DIR=$$(awk -F= '"'"'/^BACKUP_DIR=/{print $$2; exit}'"'"' tokens.env); \
-		if [ -z "$$BACKUP_DIR" ]; then \
-			echo "❌ Error: BACKUP_DIR not defined in tokens.env"; exit 1; \
-		fi; \
-		DEST="$$BACKUP_DIR/datasmith.mirror"; \
-		mkdir -p "$$DEST"; \
-		rsync -a --delete --human-readable --info=stats1 \
-			scratch/ "$$DEST/scratch/"; \
-	'
 .PHONY: check
 check: ## Run code quality tools.
 	@echo "Checking lock file consistency with 'pyproject.toml'"