Harden synthesis pipeline: rate limits, Qwen agent, neighbor cascade, remote access

CLAUDE.md

@@ -78,6 +78,40 @@ Each task lives in `dataset/formulacode_verified/<owner_repo>/<sha>/` with a mul
 - **Build**: hatchling backend, uv for dependency management
 - **CI**: GitHub Actions runs `make check` + tests on Python 3.11 and 3.12
 
+### Documentation
+
+After making a feature change, decide whether the change is significant enough to warrant updating the documentation in `docs/`. Changes that affect user-facing behavior, CLI flags, configuration knobs, pipeline stages, agent backends, or architectural decisions should be reflected in the relevant guide or design doc. Internal refactors, bug fixes, and implementation details generally do not need doc updates unless they change observable behavior.
+
+### Tunable constants
+
+Any module-level constant that is a knob — timeouts, retries, caps, windows,
+concurrency limits, thresholds — **must** be overridable from `tokens.env`
+without a code change. The `datasmith` package auto-loads `tokens.env` at
+import time (`src/datasmith/__init__.py` → `dotenv.load_dotenv`), so reading
+`os.environ.get(...)` at module scope picks up `tokens.env` values.
+
+- **Naming**: prefix every overridable constant and its env variable with
+  `DATASMITH_` so it is globally greppable in both Python and shell env.
+- **Pattern**: read the env var at module top, coerce to the target type,
+  and fall back to a literal default:
+
+  ```python
+  import os
+
+  DATASMITH_RL_MAX_RETRIES: int = int(os.environ.get("DATASMITH_RL_MAX_RETRIES", "3"))
+  DATASMITH_NEIGHBOR_WINDOW_DAYS: int = int(
+      os.environ.get("DATASMITH_NEIGHBOR_WINDOW_DAYS", "60")
+  )
+  ```
+
+- **Scope**: this rule applies to *tunable* knobs. Magic strings that
+  identify protocol fields, schema columns, or on-disk paths are not
+  constants in this sense and should stay as literals.
+- **Existing uses** (non-exhaustive, grep `DATASMITH_` for the full list):
+  `DATASMITH_RL_DEFAULT_PAUSE_S`, `DATASMITH_RL_PAUSE_JITTER_S`,
+  `DATASMITH_RL_MAX_RETRIES`, `DATASMITH_NEIGHBOR_WINDOW_DAYS`,
+  `DATASMITH_NEIGHBOR_CAP`.
+
 ## Supabase (local)
 
 fc-data uses a **local Supabase** instance for all persistent state. Connection details live in `tokens.env`:

Makefile

@@ -70,6 +70,10 @@ grafana-logs: ## Tail Grafana container logs
 grafana-tunnel: ## Expose Grafana publicly via Cloudflare Tunnel
 	@cloudflared tunnel run datasmith-grafana
 
+.PHONY: db-tunnel
+db-tunnel: ## Expose Supabase PostgREST API via Cloudflare Tunnel (db.formulacode.org)
+	@cloudflared tunnel run datasmith-db
+
 
 .PHONY: help
 help:

docs/design/components/datasmith.runners.md

@@ -34,7 +34,7 @@ graph LR
 * `ds.runners.scrape_commits`: For a given repository, scrapes all commits and runs compliance checks (`.exists`, `.attribute_compliance`, `.llm_compliance`) on each PR.
 * `ds.runners.classify_prs`: Runs a classifier agent across a set of PRs concurrently.
 * `ds.runners.resolve_packages`: For each classified PR, runs `ds.resolution.analyze_commit()` to resolve Python dependencies via `uv`, then persists results (pinned deps, Python version) to the `packages` table. Deduplicates by `(owner, repo, sha)`. See `datasmith.resolution.md`.
-* `ds.runners.synthesize_images`: For a given set of PRs, runs `ds.agents.synthesizer` for each. Reads `env_payload` and `python_version` from the `packages` table (populated by `resolve_packages`). Returns `list[str | None]`. This is expensive and must scale to ~20k PRs.
+* `ds.runners.synthesize_images`: For a given set of PRs, runs `ds.agents.synthesizer` for each. Reads `env_payload` and `python_version` from the `packages` table (populated by `resolve_packages`). Returns `list[str | None]`. This is expensive and must scale to ~20k PRs. Unlike the other runners, this one overrides `BaseRunner.run` with a queue-based worker pool so `_do_process_item` can enqueue additional items mid-flight: after every successful synthesis, PRs in the same repo whose `created_at` is within `±DATASMITH_NEIGHBOR_WINDOW_DAYS` are pushed onto the queue (capped at `DATASMITH_NEIGHBOR_CAP` per success, deduped against an in-memory set). Those neighbor items re-enter the synthesizer at `CHECK_CACHE → FIND_SIMILAR` so `TRY_SIMILAR` can reuse the freshly-cached context for free, only falling through to `LLM_GENERATE` on genuine environment drift. This collapses the old two-pass hydration workflow (`--agent codex` pass + `--agent none` pass) into a single run. Rate-limit errors from the CLI agent (detected via `ds.agents.rate_limit`) raise `RateLimitError`, which installs a shared pause across all workers until the budget resets — see the Synthesis user guide for tunable knobs.
 
 ## Async Concurrency Model
 

docs/getting-started/installation.md

@@ -182,6 +182,28 @@ make check    # Ruff lint + mypy type check
 make test     # pytest
 ```
 
+## Makefile reference
+
+Run `make help` to list all targets. The complete reference:
+
+| Target | Description |
+|--------|-------------|
+| `make install` | Create virtual environment with uv, install pre-commit hooks |
+| `make check` | Run ruff lint, mypy type check, and deptry dependency check |
+| `make test` | Run pytest with coverage |
+| `make build` | Build wheel file |
+| `make clean-build` | Remove build artifacts |
+| `make docker-clean` | Prune dangling Docker images and containers |
+| `make supabase-up` | Start local Supabase instance |
+| `make supabase-down` | Stop local Supabase instance |
+| `make supabase-status` | Show Supabase service status and URLs |
+| `make grafana-migrate` | Apply the `grafana_ro` read-only database role |
+| `make grafana-up` | Start Grafana dashboard (`http://localhost:3001`) |
+| `make grafana-down` | Stop Grafana dashboard |
+| `make grafana-logs` | Tail Grafana container logs |
+| `make grafana-tunnel` | Expose Grafana publicly via Cloudflare Tunnel |
+| `make db-tunnel` | Expose Supabase PostgREST API via Cloudflare Tunnel |
+
 ## Next steps
 
 You're ready to run the pipeline:

docs/guide/configuration.md

@@ -32,6 +32,36 @@ fc-data is configured primarily through a `tokens.env` file in the repository ro
 | `DOCKERHUB_TOKEN` | DockerHub access token |
 | `HF_TOKEN_PATH` | Path to HuggingFace token file |
 
+### Tunable constants
+
+Any module-level constant that is a knob — timeouts, retries, caps,
+windows, concurrency limits — is overridable from `tokens.env` without a
+code change. Every such variable is prefixed `DATASMITH_` so it is
+globally greppable in both Python and shell env. `tokens.env` is loaded
+at import time by `datasmith/__init__.py`, so setting one of these in the
+file is enough; no export needed.
+
+#### Stage 6: synthesize_images
+
+Rate-limit pause behavior (see [Synthesis → Rate-limit handling](synthesis.md#rate-limit-handling)):
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `DATASMITH_RL_DEFAULT_PAUSE_S` | Fallback pause (seconds) when the agent signals a rate limit but no reset time could be parsed | `3600` |
+| `DATASMITH_RL_PAUSE_JITTER_S` | Grace seconds added to the parsed reset time before workers resume, to ride out clock skew | `30` |
+| `DATASMITH_RL_MAX_RETRIES` | Maximum consecutive rate-limit pauses for a single item before it is marked failed | `3` |
+
+Chronological neighborhood cascade (see [Synthesis → Chronological neighborhood cascade](synthesis.md#chronological-neighborhood-cascade)):
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `DATASMITH_NEIGHBOR_WINDOW_DAYS` | ± window, in days of PR `created_at`, for enqueuing neighbor PRs after a successful synthesis | `60` |
+| `DATASMITH_NEIGHBOR_CAP` | Hard ceiling on neighbor PRs enqueued per successful item | `40` |
+
+Setting `DATASMITH_NEIGHBOR_CAP=0` disables the cascade entirely — the
+runner then behaves like a pre-cascade fixed-item pool, useful for tight
+unit-style reruns where extra enqueues would confuse progress tracking.
+
 ## Agent backend resolution
 
 The agent configuration (`agents/config.py`) checks environment variables in priority order:

docs/guide/monitoring.md

@@ -97,4 +97,5 @@ make grafana-up       # Start Grafana
 make grafana-down     # Stop Grafana
 make grafana-logs     # Tail container logs
 make grafana-migrate  # Apply the grafana_ro database role
+make grafana-tunnel   # Expose Grafana publicly via Cloudflare Tunnel
 ```

docs/guide/pipeline.md

@@ -30,7 +30,7 @@ fc-data --start-date 2026-02-01 --end-date 2026-03-01 --dry-run
 | `--dry-run` | Log what each stage would do without executing | `false` |
 | `--n-concurrent N` | Max concurrent items per runner stage | auto |
 | `--tasks-per-repo N` | Cap tasks per repo for stages 5–6 (useful for large repos) | unlimited |
-| `--agent AGENT` | Agent for stage 6 synthesis: `claude`, `codex`, `gemini`, or `none` | auto-detect |
+| `--agent AGENT` | Agent for stage 6 synthesis: `claude`, `codex`, `gemini`, `qwen`, or `none` | auto-detect |
 | `--force` | Re-process already-completed tasks in stages 5–6 | `false` |
 | `--offline-source PATH` | Import PR data from a Parquet file instead of scraping GitHub (stages 1–2) | — |
 | `--min-stars N` | Minimum GitHub stars for repo discovery in stage 1 | `500` |
@@ -186,7 +186,7 @@ fc-data --start-date 2026-02-01 --end-date 2026-03-01 --stage 6 --force
 
 **Writes to:** `candidate_containers` table (on success), `error_logs` table (every attempt)
 **Runner:** `SynthesizeImagesRunner`
-**Requires:** Docker daemon running, resolved packages (stage 4), rendered problems (stage 5). LLM agent CLI (`claude`, `codex`, or `gemini`) must be on `$PATH` unless `--agent none`.
+**Requires:** Docker daemon running, resolved packages (stage 4), rendered problems (stage 5). LLM agent CLI (`claude`, `codex`, `gemini`, or `qwen`) must be on `$PATH` unless `--agent none`.
 
 !!! warning
     Synthesis can be expensive — each LLM attempt may consume significant tokens and each Docker build takes minutes. Use `--n-concurrent` and `--tasks-per-repo` to control cost. Start with `--agent none` to exhaust cached/similar scripts before using LLM generation.

docs/guide/remote-access.md

@@ -0,0 +1,185 @@
+# Remote Access via Cloudflare Tunnel
+
+fc-data stores all persistent state in a local Supabase instance. By
+default this is only reachable from the host machine (`127.0.0.1:54321`).
+A **Cloudflare Tunnel** lets a remote machine run the same fc-data
+pipeline against the same database — no VPN, no open ports, no firewall
+rules.
+
+## Architecture
+
+```
+Remote machine                       Host machine
+┌──────────────┐                     ┌──────────────────────┐
+│  fc-data     │── HTTPS ──▶ Cloudflare Edge ──▶ cloudflared │──▶ Supabase
+│  tokens.env: │            (db.formulacode.org)  (tunnel)   │    :54321
+│  SUPABASE_URL│                                             │
+│  CF headers  │  Cloudflare Access                          │
+└──────────────┘  (service-token auth)                       └──────────────────────┘
+```
+
+**Two layers of auth protect the database:**
+
+1. **Cloudflare Access** — a service token (`CF-Access-Client-Id` /
+   `CF-Access-Client-Secret` headers) must be present on every request
+   or Cloudflare rejects it at the edge before it ever reaches the tunnel.
+2. **Supabase service-role key** — the standard `apikey` header required
+   by PostgREST, unchanged from local usage.
+
+## Prerequisites
+
+- A **Cloudflare account** (free plan is sufficient)
+- A **domain managed by Cloudflare** (e.g., `formulacode.org`)
+- `cloudflared` CLI installed on the **host machine** (the one running Supabase)
+
+## Host machine setup
+
+### 1. Install cloudflared
+
+```bash
+# Debian / Ubuntu
+curl -L https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb \
+  -o cloudflared.deb
+sudo dpkg -i cloudflared.deb
+
+# Verify
+cloudflared --version
+```
+
+### 2. Authenticate
+
+```bash
+cloudflared login
+```
+
+This opens a browser to authorize `cloudflared` with your Cloudflare
+account. Select the domain you want to use (e.g., `formulacode.org`).
+
+### 3. Create a tunnel
+
+```bash
+cloudflared tunnel create datasmith-db
+```
+
+Note the **Tunnel ID** printed (e.g., `a1b2c3d4-...`). A credentials
+file is saved to `~/.cloudflared/<TUNNEL_ID>.json`.
+
+### 4. Configure the tunnel
+
+Create `~/.cloudflared/config.yml`:
+
+```yaml
+tunnel: <TUNNEL_ID>
+credentials-file: /home/<user>/.cloudflared/<TUNNEL_ID>.json
+
+ingress:
+  - hostname: db.formulacode.org
+    service: http://localhost:54321
+  - service: http_status:404
+```
+
+### 5. Create the DNS record
+
+```bash
+cloudflared tunnel route dns datasmith-db db.formulacode.org
+```
+
+This creates a CNAME record pointing `db.formulacode.org` to the tunnel.
+
+### 6. Run the tunnel
+
+```bash
+# Using the Makefile target (recommended)
+make db-tunnel
+
+# Or directly
+cloudflared tunnel run datasmith-db
+
+# As a systemd service (persistent, survives reboots)
+sudo cloudflared service install
+sudo systemctl enable cloudflared
+sudo systemctl start cloudflared
+```
+
+At this point, `https://db.formulacode.org` proxies to your local
+Supabase PostgREST API — but **Cloudflare Access blocks all requests**
+until you create an access policy.
+
+## Cloudflare Access setup
+
+### 1. Create an application
+
+1. Go to [Cloudflare Zero Trust](https://one.dash.cloudflare.com/) →
+   **Access** → **Applications**
+2. Click **Add an application** → **Self-hosted**
+3. Set:
+   - **Application name**: `datasmith-db`
+   - **Session duration**: `24 hours`
+   - **Application domain**: `db.formulacode.org`
+4. Under **Policies**, create a policy:
+   - **Policy name**: `Service Token`
+   - **Action**: Service Auth
+   - **Include**: Service Token (select the token you'll create next)
+5. Save the application
+
+### 2. Create a service token
+
+1. Go to **Access** → **Service Auth** → **Service Tokens**
+2. Click **Create Service Token**
+3. Name it (e.g., `datasmith-remote`)
+4. **Copy both values immediately** — the Client Secret is only shown once:
+   - `CF-Access-Client-Id` (e.g., `abc123.access`)
+   - `CF-Access-Client-Secret` (e.g., `long-secret-string`)
+
+## Remote machine setup
+
+On the remote machine, edit `tokens.env`:
+
+```bash
+# Point at the tunnel instead of localhost
+SUPABASE_URL=https://db.formulacode.org
+SUPABASE_KEY=your-service-role-key          # Same key as the host machine
+
+# Cloudflare Access service token
+DATASMITH_CF_ACCESS_CLIENT_ID=abc123.access
+DATASMITH_CF_ACCESS_CLIENT_SECRET=long-secret-string
+```
+
+The `SUPABASE_KEY` is the same service-role key used on the host — it is
+not a Cloudflare credential.
+
+### Verify connectivity
+
+```bash
+fc-data --preflight
+```
+
+The Supabase connection check should show `[OK]`. If it fails:
+
+- **403 Forbidden** — the CF Access headers are missing or the service
+  token is invalid. Double-check `DATASMITH_CF_ACCESS_CLIENT_ID` and
+  `DATASMITH_CF_ACCESS_CLIENT_SECRET`.
+- **502 Bad Gateway** — `cloudflared` is not running on the host machine
+  or Supabase is down. SSH into the host and check
+  `systemctl status cloudflared` and `supabase status`.
+- **Connection refused** — DNS is not resolving. Verify
+  `cloudflared tunnel route dns` was run and the CNAME exists in your
+  Cloudflare DNS dashboard.
+
+## How it works in the code
+
+When `DATASMITH_CF_ACCESS_CLIENT_ID` and `DATASMITH_CF_ACCESS_CLIENT_SECRET`
+are both set, `datasmith.utils.db` automatically injects the
+`CF-Access-Client-Id` and `CF-Access-Client-Secret` headers into every
+Supabase client request via `ClientOptions`. No other code changes are
+needed — every call to `get_client()` or `get_async_client()` picks up
+the headers transparently.
+
+When neither variable is set (the default for local development), no
+extra headers are added and behavior is identical to before.
+
+## Makefile Targets
+
+```bash
+make db-tunnel        # Expose Supabase PostgREST API via Cloudflare Tunnel
+```