Refining improves the material without replacing it. The weights never move; only the harness around them does. The pipeline proposes, validates, and opens a PR — it never merges.
The measurement half of the self-evolving-harness project: runs a fixed task
suite (runner/tasks/) against a live Gemma agent driven by the
carbon harness, repeatedly per task
(held-out gets more samples than held-in), aggregates pass fractions
(averaged, never majority-voted), and computes Δ_in/Δ_ho between two harness
states for the acceptance rule Δ_in ≥ 0, Δ_ho ≥ 0, max(Δ_in, Δ_ho) > 0.
Why this is its own repo: the task definitions, verifier code, pinned commands, and oracle hashes must never share a home with the editable surface the external editor acts on — otherwise the editor can "pass" a task by rewriting its verifier. The boundary is a repo boundary, not a directory convention.
Layout it expects: refinery/ and carbon/ as sibling checkouts under one
root. pyproject.toml's ../carbon and runner/carbon_env.py's CARBON_ROOT
both assume that — change them together if you nest things differently.
LM Studio must be serving the model named in carbon/.env (real models, no
mocks). refinery has no .env of its own — it reads carbon's, so the suite
and the harness under test always agree on the endpoint.
uv sync
uv run python -m runner.cli run --label baseline-main # full suite, resumable
uv run python -m runner.cli run --label x --only D1 --attempts 1 # spot-check
uv run python -m runner.cli delta results/baseline-main.json results/candidate.json
Results stream to results/.jsonl per attempt (a killed run resumes,
skipping finished attempts; records are pinned to the carbon SHA + config
version they measured, and a resume refuses records from a different harness
state); aggregates land in results/.json. Partial runs (--only) are
stamped with a "filter" field and refuse to overwrite a full run's JSON.
delta refuses filtered inputs, mismatched per-task attempt counts, and
mismatched models by design — a Δ is only meaningful like-for-like.
Results are also stamped with runner_sha (the verifier's own version);
deltas across different runner versions are refused, so re-measure the
baseline after changing runner code.
To measure a different harness state: check out the branch in carbon
(the editable dependency points at that working tree), run with a new label,
then delta the two JSONs.
loop/ is the pipeline half: the mining and proposal steps are done by the
proposer model directly as reasoning (Fable, in-session, for iteration 1)
and land as fixed JSON artifacts in iterations/<iter>/ (clusters.json,
candidates.json); only validation and the branch+PR step are code. A
candidate is applied to the carbon WORKING TREE (never committed — a
rejected candidate leaves no trace there), the suite runs in a fresh
subprocess (config values bind at import), the edit is reverted, and the
acceptance rule Δ_in ≥ 0, Δ_ho ≥ 0, max > 0 decides. Accepted edits each get
their own branch off self-improvement in carbon and a PR targeting it
(explicit base — never main), with the evidence (cluster, knobs, per-task Δ,
provenance) in the body. The pipeline never merges.
uv run python -m loop.cli dry-run --iteration iter-01 --candidate clamp-12k --tasks A2 D1
uv run python -m loop.cli validate --iteration iter-01 [--candidate clamp-12k]
uv run python -m loop.cli pr --iteration iter-01 --candidate clamp-12k
- runner/tasks/ — the task specs (mechanical verifiers only; sentinels, pinned commands, and seed-file sha256s authored at import time)
- runner/{run,suite}.py — attempt/suite drivers; runner/delta.py — Δ + rule
- runner/helpers.py — approve-and-log approver, environ guard, transcript/hash utilities
- results/ — committed measurement artifacts
- loop/ — the validate→branch→PR pipeline (imports runner.suite / runner.delta)
- iterations/ — per-iteration artifacts: mining notes, clusters, candidates, validation records (rejected candidates included — they are the honest bulk)
uv run pytest # verifier helpers, Δ math, registry shape — no model calls