feat: sync score averaging, adapter scale and recent cleanup

.env.example

@@ -1,11 +1,6 @@
-# Benchmarking against the hosted Jev API.
 TYPESAFE_API_KEY=
-
-# The LLM baseline `levbench compare` measures against.
 ANTHROPIC_API_KEY=
 
-# Only if you front a local /v1/systemone server with auth. Your real
-# TYPESAFE_API_KEY is deliberately never sent to a --base-url host.
-# Left commented on purpose: `VAR=` sets the variable to the empty string,
-# which is not the same as leaving it unset.
+# Optional local-server auth; TYPESAFE_API_KEY is never sent to --base-url hosts.
+# Leave commented to keep unset; VAR= assigns an empty string.
 # LEVBENCH_LOCAL_API_KEY=

CLAUDE.md

@@ -1,34 +1,57 @@
-## Comments
+# lev
+
+lev is an open System One decision model: `f(state, questions) → typed, calibrated distributions`, served over the same `/v1/systemone` wire format as TypeSafe's Jev, with no token generation. It is `Qwen/Qwen3.5-4B` with a rank-32 LoRA adapter, a set-attention candidate head, and per-bucket temperature calibration, released on the Hub as [`interfaze-ai/lev`](https://huggingface.co/interfaze-ai/lev). `levbench` is the harness that measures it, and Jev, on S1Bench.
+
+@CONTRIBUTING.md
+
+## Where the facts live
 
-Keep comments rare and information-dense.
+Read the relevant doc before changing anything it covers; most "obvious" improvements here were already tried and are recorded with their evidence.
 
-- Do not add comments by default.
-- Do not explain code that is already obvious from the implementation.
-- Never narrate the implementation line-by-line.
-- Do not add comments for simple assignments, conditionals, loops, function calls, or standard library usage.
-- Prefer better naming and structure over explanatory comments.
-- Comments should explain information that cannot be expressed clearly in the code.
+| Doc | Holds |
+| --- | --- |
+| [`docs/STATUS.md`](docs/STATUS.md) | What is built, how far each part is exercised, the current headline numbers |
+| [`docs/DECISIONS.md`](docs/DECISIONS.md) | ADR-001 to ADR-030; the next decision is ADR-031 |
+| [`docs/FINDINGS.md`](docs/FINDINGS.md) | Every measurement and head-to-head, in order, with provenance |
+| [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) | The design, and the trade-offs refused |
+| [`docs/TRAINING.md`](docs/TRAINING.md) | The training order, presets, budgets, resume |
+| [`docs/SETUP.md`](docs/SETUP.md) | Local and Modal setup, volumes, deploy knobs |
+| [`hf/README.md`](hf/README.md) | The Hub model card; keep it consistent with `README.md` |
 
-Add a comment only when it explains one of:
+## The model
 
-- non-obvious intent or rationale
-- an important business rule
-- an external API/protocol assumption
-- a performance or security constraint
-- an edge case that is easy to miss
-- a workaround for a known bug or limitation
+- **Readout.** Mode A reads label-token logits at the answer boundary whenever every label code is a single token — 68 options for this tokenizer, since the 69th code, `BQ`, is two. Above that, Mode B scores candidate text with the trained head. Training and serving route the same way (ADR-025, ADR-026); serving also skips split codes (ADR-028). Routing is in `router.py`.
+- **Calibration** is per bucket, keyed `{type}:{mode}[:{band}]`, selected by leave-one-family-out transfer ECE. `calibrate.fit` refuses `test`, `eval` and `holdout` splits — fit only on the dedicated `calibration` split.
+- **The release** (`lev-4b`) is preset `4b-instruct`, step 18750, chat prompt style. Its `lev_release.json` manifest sets the base model, prompt style and optional `adapter_scale`, and `lev.load` honours all three.
+- **The released weights were trained on 2026-09-23 code, before this repository's first commit.** Changes to data, sources, prompts or the training loop since are not in the weights until a retrain; do not attribute a benchmark move to them.
 
-Keep implementation comments to one or two concise lines whenever possible.
+## Running things
 
-Do not write multi-paragraph comments or verbose explanatory blocks.
+- **Anything that needs a GPU runs on Modal**, through `modal/app.py` and the `make` targets. Do not download checkpoints or run models locally to check something — probe the deployed endpoint or add a `modal run` function.
+- **Pass `--preset 4b-instruct` to every Modal function you call directly.** Their defaults are preset `4b`, the plain base-model run, not the release; the `make` targets pass `PRESET=4b-instruct` for you. `modal deploy` likewise serves `4b` unless `LEV_SERVE_PRESET` is set, which `make deploy` does.
+- `make deploy` replaces the live endpoint. `modal/app.py` must import inside the container, where `python-dotenv` is not installed — keep its import guarded, or every container crash-loops.
+- Modal volumes: `lev-models` (base weights), `lev-checkpoints` (runs and `releases/`), `lev-data` (mixture and splits).
 
-For public APIs, use docstrings/documentation when the API contract, parameters, return behavior, exceptions, or usage constraints need to be documented.
+## Benchmarks
 
-When modifying existing code:
+- S1Bench is 13 subsets pinned by Bespoke Nimble manifests at commit `62076b4`, in `packages/lev/src/lev/data/s1bench_subsets/`. All 13 are banned from training (ADR-009).
+- **Never select anything on S1Bench** — a hyperparameter, a scale, a prompt. Select with leave-one-family-out over the subsets, as FINDINGS §19 does, or on data outside both training and S1Bench. The `calibration` split cannot select out-of-distribution knobs: it is drawn from the training mixture.
+- Compare against Jev through identical task files, one complete run each.
+
+## Safety
+
+- **Ask before any paid call**: `levbench` against Jev, the Anthropic backend, or a sweep. They bill per token.
+- `levbench` never sends `TYPESAFE_API_KEY` to a `--base-url` host; a custom endpoint gets `LEVBENCH_LOCAL_API_KEY` or `local`. Keep it that way.
+- Do not read `.env`.
+- This is the public repository. Development happens in [`Abhinavexists/lev`](https://github.com/Abhinavexists/lev) and is ported here by hand; the two diverge, so a change made here must be ported back, not assumed.
+
+## Comments
 
-- Preserve useful existing comments.
-- Remove comments that are made obsolete by the change.
-- Do not rewrite comments merely for stylistic reasons.
-- Do not perform unrelated comment cleanup.
+The code should explain itself; a comment exists only for what it cannot say.
 
-Default assumption: if the code is understandable without the comment, do not write the comment.
+- **One line is the unit.** A comment is one line, a docstring is one line. Multi-line is earned — a module's two or three lines of orientation, or tensor shapes and wire formats. A second paragraph on a function belongs in `docs/` with a pointer from the code.
+- **Do not restate the code.** If a reader would have to check the prose against the implementation to confirm they agree, the comment costs more than it gives.
+- **Compress before deleting.** A real fact buried in a block above a statement usually becomes a short trailing comment on the statement.
+- **Point, do not restate.** `(ADR-028)` beats a paragraph reproducing the decision. A contract the code states plainly — a guard that raises, a named check like `_refuse_held_out` — needs no prose.
+- **Public surface is the exception.** `lev.load`, the server, and the `/v1/systemone` types are consumed without reading the body; document their contracts.
+- Do not rewrite comments for style. When changing code, remove comments the change made obsolete, and do no unrelated cleanup.

CONTRIBUTING.md

@@ -6,7 +6,7 @@
 make check     # ruff + the full test suite
 ```
 
-Both must be clean. Tests run without a GPU, network or API keys. A basic `uv sync` skips tests that need optional model dependencies; use `uv sync --extra train` to include the CPU tensor, cache and training-loop tests.
+Both must be clean. Tests run without a GPU, network or API keys. use `uv sync --extra train` to include the CPU tensor, cache and training-loop tests.
 
 ## Finding the code
 

Makefile

@@ -1,4 +1,15 @@
-# `make help` lists every target. Variables are overridable: `make train PRESET=9b`.
+# setup: Install workspace  |  setup-train: + training deps
+# setup-serve: + serving deps  |  setup-modal: + Modal client
+
+# test: Run tests  |  lint: Check code  |  fmt: Format code
+# check: Lint + test  |  clean: Remove caches
+
+# smoke-local: CPU smoke test  |  smoke: Modal H100 smoke test
+# train: Full training run  |  calibrate: Fit temperatures
+
+# serve: Dev server  |  deploy: Stable endpoint
+# release: Package checkpoint  |  weights: Pull weights  |  publish: Push to HF
+
 .DEFAULT_GOAL := help
 
 PRESET  ?= 4b-instruct
@@ -15,111 +26,101 @@ URL     ?= http://localhost:8000
 FRESH   ?=
 RESUME  ?=
 
-help: ## Show this help
-	@awk 'BEGIN{FS=":.*?## "} \
-		/^##@/ {printf "\n\033[1m%s\033[0m\n", substr($$0, 5)} \
-		/^[a-zA-Z0-9_-]+:.*?## / {printf "  \033[36m%-12s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST)
-
-##@ Setup
+help:
+	@awk '/^#/ {sub(/^# ?/, ""); print; next} /^$$/ {print; next} {exit}' $(MAKEFILE_LIST)
 
 .PHONY: setup setup-train setup-serve setup-modal
-setup: ## Install the workspace
+setup:
 	uv sync
 
-setup-train: ## Install with the training extras (torch, transformers, peft)
+setup-train:
 	uv sync --extra train
 
-setup-serve: ## Install with the serving extras (adds fastapi, uvicorn)
+setup-serve:
 	uv sync --extra serve
 
-setup-modal: ## Install with the Modal client
+setup-modal:
 	uv sync --extra modal
 
-##@ Develop
-
 .PHONY: test lint fmt check clean
-test: ## Run every test
+test:
 	uv run pytest
 
-lint: ## Check formatting and lint rules
+lint:
 	uv run ruff check .
 	uv run ruff format --check .
 
-fmt: ## Apply formatting and autofixes
+fmt:
 	uv run ruff check --fix .
 	uv run ruff format .
 
-check: lint test ## Lint then test — run before pushing
+check: lint test
 
-clean: ## Remove caches and build artefacts
+clean:
 	find . -type d -name __pycache__ -prune -exec rm -rf {} +
 	rm -rf .pytest_cache .ruff_cache dist build
 
-##@ Data
 
 .PHONY: plan check-data data eval-set s1bench
-plan: ## Print the H100 training budget without spending it
+plan:
 	uv run lev plan --preset $(PRESET)
 
-check-data: ## Contamination guard over a source list (SOURCES=path)
+check-data:
 	uv run lev check-data $(SOURCES)
 
-data: ## Build the training mixture locally (LIMIT=rows per source)
+data:
 	uv run lev data build --out $(OUT) --limit-per-source $(LIMIT)
 
-eval-set: ## Export the held-out split as levbench task files
+eval-set:
 	uv run lev data eval --data $(OUT) --out $(EVAL)
 
-s1bench: ## Export the S1Bench eval subsets as levbench task files
+s1bench:
 	uv run lev s1bench export --out $(S1)
 
-##@ Train
 
 .PHONY: smoke-local smoke train calibrate
-smoke-local: ## Train 0.8B on CPU for a few steps — proves the path without Modal
+smoke-local:
 	uv run lev data build --out /tmp/lev-mix --limit-per-source 2000 --n-examples 4000
 	uv run lev train --preset smoke --data /tmp/lev-mix \
 		--output-dir /tmp/lev-smoke --max-steps $(STEPS)
 
-smoke: ## Modal: exercise the whole training path on 0.8B (~5 min of H100)
+smoke:
 	modal run modal/app.py::smoke
 
-train: ## Modal: the real run; resumes the newest checkpoint (FRESH=1 to start over, RESUME=path)
+train:
 	modal run modal/app.py::train --preset $(PRESET) $(if $(FRESH),--fresh,) $(if $(RESUME),--resume $(RESUME),)
 
-calibrate: ## Modal: fit per-bucket temperatures after training
+calibrate:
 	modal run modal/app.py::calibrate --preset $(PRESET)
 
-##@ Serve and release
 
 .PHONY: serve deploy release weights publish
-serve: ## Modal: dev server on an H100 (ephemeral; any `modal run` on this app steals its URL)
-	LEV_SERVE_PRESET=$(PRESET) modal serve modal/app.py
+serve:
+	LEV_SERVE_PRESET=$(PRESET) $(if $(NAME),LEV_SERVE_RELEASE=$(NAME)) modal serve modal/app.py
 
-deploy: ## Modal: deploy /v1/systemone with a stable URL (PRESET selects the checkpoint)
-	LEV_SERVE_PRESET=$(PRESET) modal deploy modal/app.py
+deploy:
+	LEV_SERVE_PRESET=$(PRESET) $(if $(NAME),LEV_SERVE_RELEASE=$(NAME)) modal deploy modal/app.py
 
-release: ## Modal: package the newest PRESET checkpoint into a release dir on the volume
+release:
 	modal run modal/app.py::export_checkpoint --preset $(PRESET) $(if $(NAME),--name $(NAME),)
 
-weights: ## Pull a packaged release to weights/RELEASE (RELEASE=name, default PRESET)
+weights:
 	mkdir -p weights
 	modal volume get --force lev-checkpoints releases/$(RELEASE) weights/
 
-publish: ## Upload weights/RELEASE to the Hub (REPO=org/name; needs HF_TOKEN)
+publish:
 	uv run lev release publish weights/$(RELEASE) --repo $(REPO)
 
-##@ Benchmark
 
 .PHONY: bench bench-local sweep snake
-bench: s1bench ## Benchmark Jev on the S1Bench subsets (needs TYPESAFE_API_KEY)
+bench: s1bench
 	uv run levbench eval --backend jev --tasks $(S1)
 
-bench-local: s1bench ## Benchmark a /v1/systemone server on the S1Bench subsets (URL=server)
+bench-local: s1bench
 	uv run levbench eval --backend lev --base-url $(URL) --tasks $(S1)
 
-sweep: ## Measure shared-state batching economics
+sweep:
 	uv run levbench sweep --backend jev
 
-snake: ## The decision model plays Snake over /v1/systemone (URL=server; --record in artifacts/)
+snake:
 	uv run levbench snake --backend lev --base-url $(URL) --record artifacts/snake/run.jsonl