Skip to content

Commit 980a697

Browse files
atharvasclaude
andauthored
Rename CLI from ds-update to fc-data (#14)
* Rename CLI entry point from ds-update to fc-data Aligns the CLI command name with the FormulaCode branding. Updated pyproject.toml entry point, argparse prog name, all docs, website pages, and tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Rename PyPI package from datasmith to fc-data The name "datasmith" is already taken on PyPI. Use "fc-data" to match the CLI command and FormulaCode branding. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add PyPI publish workflow via GitHub trusted publishers Triggers on GitHub release creation. Uses pypa/gh-action-pypi-publish with OIDC trusted publishing (no API token needed). Adds `build` to dev dependencies. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Rebrand DataSmith to fc-data across all docs and website Replaces all product-name references (DataSmith, Datasmith, `datasmith`) with fc-data in README, CLAUDE.md, AGENTS.md, mkdocs site, and all guide/getting-started pages. Python import paths (`from datasmith.xxx`) and GitHub repo URLs are intentionally preserved. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 81c9f89 commit 980a697

16 files changed

Lines changed: 139 additions & 173 deletions

File tree

.github/workflows/publish.yml

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: Publish to PyPI
2+
3+
on:
4+
release:
5+
types: [published]
6+
7+
jobs:
8+
publish:
9+
runs-on: ubuntu-latest
10+
environment: pypi
11+
permissions:
12+
id-token: write
13+
steps:
14+
- name: Check out
15+
uses: actions/checkout@v4
16+
17+
- name: Set up the environment
18+
uses: ./.github/actions/setup-python-env
19+
20+
- name: Build package
21+
run: uv run python -m build
22+
23+
- name: Publish to PyPI
24+
uses: pypa/gh-action-pypi-publish@release/v1

AGENTS.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Repository Guidelines
22

3-
This guide helps contributors work effectively in the datasmith repository.
3+
This guide helps contributors work effectively in the fc-data repository.
44

55
## Project Structure & Module Organization
66
- Source: `src/datasmith/` — current modules: `agents/`, `docker/`, `github/`, `publish/`, `resolution/`, `runners/`, `update/`, `utils/`, plus helper modules like `filters.py` and `preflight.py`.
@@ -17,7 +17,7 @@ This guide helps contributors work effectively in the datasmith repository.
1717
- `uv run python -m pytest --doctest-modules` — run doctests with the standard test suite.
1818
- `uvx tox -q` — run the tox matrix (py39–py312) as defined in `tox.ini`.
1919
- To run commands using the same environment variables as the user, use `uv run <command>`.
20-
- Pipeline entrypoint: `ds-update --help` and `ds-update --start-date ... --end-date ...`.
20+
- Pipeline entrypoint: `fc-data --help` and `fc-data --start-date ... --end-date ...`.
2121

2222
## Coding Style & Naming Conventions
2323
- Python 3.9–3.12. 4‑space indentation, type hints required (mypy strict; see `pyproject.toml`).

CLAUDE.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
44

55
## What is this project?
66

7-
DataSmith is the Python toolchain for building the **FormulaCode** dataset — a benchmark of 67+ repositories with 964+ performance-improving commits, designed to evaluate LLMs' ability to optimize real-world codebases. It scores LLMs relative to the human-authored speedup using ASV (Airspeed Velocity) benchmarks rather than binary pass/fail tests.
7+
fc-data is the Python toolchain for building the **FormulaCode** dataset — a benchmark of 67+ repositories with 964+ performance-improving commits, designed to evaluate LLMs' ability to optimize real-world codebases. It scores LLMs relative to the human-authored speedup using ASV (Airspeed Velocity) benchmarks rather than binary pass/fail tests.
88

99
## Common commands
1010

@@ -24,11 +24,11 @@ uv run pre-commit run -a # Run all pre-commit hooks
2424
# Dataset verification (iterative Docker build debugging)
2525
python dataset/verify.py --task dataset/formulacode_verified/<owner_repo>/<sha>
2626

27-
# Pipeline update (monthly, the primary entrypoint for DataSmith)
28-
ds-update --start-date YYYY-MM-DD --end-date YYYY-MM-DD # Run all 6 stages
29-
ds-update --start-date 2026-01-01 --end-date 2026-01-31 --stage 4 # Run a single stage
30-
ds-update --start-date 2026-01-01 --end-date 2026-01-31 --resume # Resume from last completed
31-
ds-update --help # See all options
27+
# Pipeline update (monthly, the primary entrypoint for fc-data)
28+
fc-data --start-date YYYY-MM-DD --end-date YYYY-MM-DD # Run all 6 stages
29+
fc-data --start-date 2026-01-01 --end-date 2026-01-31 --stage 4 # Run a single stage
30+
fc-data --start-date 2026-01-01 --end-date 2026-01-31 --resume # Resume from last completed
31+
fc-data --help # See all options
3232
```
3333

3434
## Architecture
@@ -47,7 +47,7 @@ ds-update --help # See all op
4747
| `benchmark/` | ASV benchmark collection |
4848
| `collation/` | Data aggregation |
4949

50-
### Pipeline stages (`ds-update`)
50+
### Pipeline stages (`fc-data`)
5151

5252
1. **scrape_repos** — Fetch repository metadata from GitHub
5353
2. **scrape_commits** — Scrape merged PR commits and patches
@@ -73,7 +73,7 @@ Each task lives in `dataset/formulacode_verified/<owner_repo>/<sha>/` with a mul
7373

7474
## Supabase (local)
7575

76-
DataSmith uses a **local Supabase** instance for all persistent state. Connection details live in `tokens.env`:
76+
fc-data uses a **local Supabase** instance for all persistent state. Connection details live in `tokens.env`:
7777

7878
- `SUPABASE_URL=http://127.0.0.1:54321` (PostgREST API)
7979
- `SUPABASE_KEY=sb_secret_...` (service-role key)

README.md

Lines changed: 46 additions & 111 deletions
Large diffs are not rendered by default.

docs/getting-started/installation.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ This creates a virtual environment with `uv`, installs all dependencies, and set
3333

3434
## 3. Configure `tokens.env`
3535

36-
DataSmith reads all configuration from a `tokens.env` file in the repo root. The `Settings` class (powered by `pydantic-settings`) loads it automatically — no manual `source` or `export` needed.
36+
fc-data reads all configuration from a `tokens.env` file in the repo root. The `Settings` class (powered by `pydantic-settings`) loads it automatically — no manual `source` or `export` needed.
3737

3838
Create the file:
3939

@@ -55,7 +55,7 @@ SUPABASE_KEY=<paste service-role key here>
5555

5656
# === GitHub (required) ===
5757
# One or more GitHub personal access tokens, comma-separated.
58-
# DataSmith rotates tokens automatically when one hits the rate limit.
58+
# fc-data rotates tokens automatically when one hits the rate limit.
5959
# Create tokens at https://github.com/settings/tokens with `repo` scope.
6060
GH_TOKENS=github_pat_xxx
6161
```
@@ -98,7 +98,7 @@ See [Configuration](../guide/configuration.md) for a complete reference of all e
9898

9999
## 4. Set up Supabase
100100

101-
DataSmith uses a local Supabase instance for all persistent state (no cloud account needed).
101+
fc-data uses a local Supabase instance for all persistent state (no cloud account needed).
102102

103103
### Start the instance
104104

@@ -129,7 +129,7 @@ This prints connection details. Copy the **service_role key** (not the anon key)
129129

130130
### Apply migrations
131131

132-
DataSmith's schema is defined in numbered SQL migrations:
132+
fc-data's schema is defined in numbered SQL migrations:
133133

134134
```bash
135135
npx supabase migration up --local
@@ -187,7 +187,7 @@ make test # pytest
187187
You're ready to run the pipeline:
188188

189189
```bash
190-
ds-update --start-date 2026-03-01 --end-date 2026-04-01
190+
fc-data --start-date 2026-03-01 --end-date 2026-04-01
191191
```
192192

193193
See the **[Pipeline guide](../guide/pipeline.md)** for the full CLI reference and stage descriptions.

docs/getting-started/quickstart.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Quickstart
22

3-
This guide walks through common DataSmith operations with code examples.
3+
This guide walks through common fc-data operations with code examples.
44

55
## Working with pull requests
66

docs/guide/configuration.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Configuration
22

3-
DataSmith is configured primarily through a `tokens.env` file in the repository root. The `Settings` class (powered by `pydantic-settings`) loads these automatically.
3+
fc-data is configured primarily through a `tokens.env` file in the repository root. The `Settings` class (powered by `pydantic-settings`) loads these automatically.
44

55
## Environment variables
66

docs/guide/docker-images.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Docker Images
22

3-
DataSmith uses a three-tier Docker image hierarchy to build reproducible environments for each pull request.
3+
fc-data uses a three-tier Docker image hierarchy to build reproducible environments for each pull request.
44

55
## Image hierarchy
66

docs/guide/pipeline.md

Lines changed: 25 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,22 @@
1-
# Pipeline (`ds-update`)
1+
# Pipeline (`fc-data`)
22

3-
`ds-update` is the primary command for running DataSmith. It discovers performance-improving commits from GitHub, classifies them with LLM agents, resolves dependencies, synthesizes Docker build contexts, and publishes verified images.
3+
`fc-data` is the primary command for running fc-data. It discovers performance-improving commits from GitHub, classifies them with LLM agents, resolves dependencies, synthesizes Docker build contexts, and publishes verified images.
44

55
## Quick reference
66

77
```bash
88
# Run all 7 stages for a date range
9-
ds-update --start-date 2026-02-01 --end-date 2026-03-01
9+
fc-data --start-date 2026-02-01 --end-date 2026-03-01
1010

1111
# Resume from where you left off (skips completed stages)
12-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --resume
12+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --resume
1313

1414
# Run only specific stages
15-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 3
16-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 5 --stage 6
15+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 3
16+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 5 --stage 6
1717

1818
# Preview what would run without executing
19-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --dry-run
19+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --dry-run
2020
```
2121

2222
## CLI flags
@@ -53,10 +53,10 @@ For each repository found, the runner fetches and stores metadata including desc
5353

5454
```bash
5555
# Discover repos with at least 1000 stars
56-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 1 --min-stars 1000
56+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 1 --min-stars 1000
5757

5858
# Include repos from an offline dataset
59-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 1 --offline-source data.parquet
59+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 1 --offline-source data.parquet
6060
```
6161

6262
**Writes to:** `repositories` table
@@ -76,7 +76,7 @@ For every repository in the `repositories` table, scrapes all merged pull reques
7676
If `--offline-source` is provided, also bulk-imports PR records from the Parquet file (useful for seeding the database with historical data).
7777

7878
```bash
79-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 2
79+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 2
8080
```
8181

8282
**Writes to:** `pull_requests` table (one row per PR)
@@ -95,10 +95,10 @@ Only PRs that pass a symbolic pre-filter (`is_performance_commit_symbolic = True
9595

9696
```bash
9797
# Classify all unclassified PRs
98-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 3
98+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 3
9999

100100
# Re-classify all PRs (including already-classified ones)
101-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 3 --force
101+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 3 --force
102102
```
103103

104104
**Updates:** `pull_requests` table (sets `is_performance_commit`, `classification`)
@@ -118,7 +118,7 @@ For each performance-classified PR, checks out the repository at the merge commi
118118
The resolved dependency set (`env_payload`) and the Python version used are stored in the `packages` table. These are consumed by stage 6 to build `docker_build_env.sh` — the shell script that installs dependencies inside the Docker image.
119119

120120
```bash
121-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 4
121+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 4
122122
```
123123

124124
**Writes to:** `packages` table (`env_payload`, `python_version`, `can_install`)
@@ -139,13 +139,13 @@ Builds a rich, deconstructed problem context for each PR. This context is what t
139139

140140
```bash
141141
# Render problem contexts
142-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 5
142+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 5
143143

144144
# Limit to 3 PRs per repo (useful for testing)
145-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 5 --tasks-per-repo 3
145+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 5 --tasks-per-repo 3
146146

147147
# Re-render already-processed PRs
148-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 5 --force
148+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 5 --force
149149
```
150150

151151
**Writes to:** `candidate_prs` table
@@ -169,19 +169,19 @@ Each attempt (success or failure) is logged to the `error_logs` table with the a
169169

170170
```bash
171171
# Use the default auto-detected agent
172-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 6
172+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 6
173173

174174
# Force a specific agent
175-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 6 --agent claude
175+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 6 --agent claude
176176

177177
# Skip LLM generation entirely — only use cached/similar scripts
178-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 6 --agent none
178+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 6 --agent none
179179

180180
# Limit concurrency and tasks per repo (controls cost)
181-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 6 --n-concurrent 2 --tasks-per-repo 5
181+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 6 --n-concurrent 2 --tasks-per-repo 5
182182

183183
# Re-synthesize already-completed PRs
184-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 6 --force
184+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 6 --force
185185
```
186186

187187
**Writes to:** `candidate_containers` table (on success), `error_logs` table (every attempt)
@@ -207,7 +207,7 @@ The publish pipeline:
207207
6. **Mark published** — Sets `published_at` timestamp on each published PR row
208208

209209
```bash
210-
ds-update --start-date 2026-02-01 --end-date 2026-03-01 --stage 7
210+
fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 7
211211
```
212212

213213
**Reads from:** `pull_requests`, `packages`, `candidate_containers`
@@ -221,13 +221,13 @@ A monthly update usually looks like this:
221221

222222
```bash
223223
# 1. Run the full pipeline
224-
ds-update --start-date 2026-03-01 --end-date 2026-04-01
224+
fc-data --start-date 2026-03-01 --end-date 2026-04-01
225225

226226
# 2. If it gets interrupted, resume where it left off
227-
ds-update --start-date 2026-03-01 --end-date 2026-04-01 --resume
227+
fc-data --start-date 2026-03-01 --end-date 2026-04-01 --resume
228228

229229
# 3. After fixing a synthesis issue, re-run just stages 6-7
230-
ds-update --start-date 2026-03-01 --end-date 2026-04-01 --stage 6 --stage 7 --force
230+
fc-data --start-date 2026-03-01 --end-date 2026-04-01 --stage 6 --stage 7 --force
231231
```
232232

233233
## Monitoring progress

docs/guide/publishing.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Publishing
22

3-
DataSmith publishes verified tasks to DockerHub (Docker images) and HuggingFace (Parquet datasets).
3+
fc-data publishes verified tasks to DockerHub (Docker images) and HuggingFace (Parquet datasets).
44

55
## Publishing workflow
66

@@ -49,7 +49,7 @@ rows = sb.table("pull_requests") \
4949

5050
## Pipeline integration
5151

52-
Publishing is the final stage of the `ds-update` pipeline (stage 7). The `publish_pipeline()` function orchestrates:
52+
Publishing is the final stage of the `fc-data` pipeline (stage 7). The `publish_pipeline()` function orchestrates:
5353

5454
1. Query DB for unpublished, verified PRs
5555
2. Push Docker images to DockerHub

0 commit comments

Comments
 (0)